A robots.txt file is a simple text file that lets search engines know what NOT to crawl and index. If your intention is to allow all search engines to crawl and index all aspects of your site, then you do not need a robots.txt file at all. If, however, you want to block some or all of the search engines from indexing some or all of your site, then you will need to create a robots.txt file. My favorite robots.txt tutorial can be found at https://www.gnc-web-creations.com/creating_robotstxt_file.htm.
What is a robots.txt file?
