I was shoulder-surfing a colleague today while they were trying to diagnose a webserver problem for a client. I noticed, certainly not for the first time, that the Apache error log message was filled with messages like “robots.txt not found” and “favicon.ico not found”. Surely these must be amongst the most frequently logged errors (if not the top two).
Multiplied by many hundreds of servers, with millions of hits per day, and you have a significant amount of disk space being taken up by these trivial messages. What’s more, any time you spend scrolling through the hordes of messages like these is time taken away from debugging the real problem, if it exists.
So please, be kind to your sysadmin and include a robots.txt and favicon.ico for your website. It makes sense from a search engine point of view and makes your website just a little bit prettier, so why not?