Those good robots that respect it. In it is sometimes overlooked or unfortunately misunderstood because of its simplicity. In the file, we then encounter many errors or over-optimization, which is counterproductive. And who else could talk about the importance of boring robots.txt with passion, humor, and at the same time, in a well-founded way, than ex-Google Fili Wiese (now Searchbrothers)? However, the pace of the lecture was more murderous than on a German autobahn, so we only managed to note a few interesting insights. For example, did you know that: Robots.txt can't prevent web scraping in any way? Can you have robots. not only for subdomains, but also for different protocols (e.g. ftp) and individual URL path folders? Robot names in robots.txt cannot contain numbers or spaces and must always be lowercase?
Shouldn't you warn about specific folders with passwords or other sensitive data by banning browsing, for example, but rather hide them behind a login form? Does every crawler always follow the Bolivia Mobile Database least restrictive general rule? But at the same time, he only follows the set of rules that are specifically designed for him, and completely ignores the others? Directives for specific robots must be first in the file and only then are general directives listed? If there is no clear directive for a specific URL path or the directive is single-line, is it taken as allow? If 2 different commands apply to the same URL path, robots prefer allow? Is robots.txt “case-sensitive”?
A URL path with uppercase and lowercase letters will be 2 different URLs with different rules. Does the main sitemap.xml URL always have to be given in absolute form? It's a shame that Fili Wiese hasn't published his presentation yet, because there have been a lot of recommendations for robots.txt. 7. Tom Greenwood – Web design for people and planet (by Frantisek Bielik) Tom's presentation focused on the issue of sustainability in the world of the internet and web design. He started with some rather scary numbers that showed everyone in the audience the harsh reality of the impact the internet has on our planet. The carbon footprint of the internet is comparable to that of the aviation industry Personally, I've always thought of the internet as an intangible thing that has no impact on the world.
