A Robots.txt file gives search engines bots the direction they need to index your pages. With the help of the Robots.txt file, search engines know what pages they can index and what pages they should not index.
One of the most critical SEO tasks is to control the search engine spiders (like Googlebot) that crawl and index your Web site. Mastery of these spiders is paramount to preventing duplicate content while ensuring that search engines focus mainly on your most important pages.
Although it may seem a bit technical, spider control is actually easier than most people think. It's simply a matter of deploying an essential tool called the robots.txt file.
Robots.txt gives spiders (aka, robots) the direction they need to find your most important pages and skip the ones you don't want indexed. Robots.txt is also called the Robots Exclusion Protocol.
In respect to SEO, the robots.txt file is a must-have! First, you should exclude folders without search value like cgi-bin, folders with scripts, pages that are available only to registered users; it is also a good SEO idea to exclude pages with duplicate content
(for example, your articles archive) to prevent them from outranking the original pages.
See an example of a robots.txt file:
This robots.txt file will disallow all spiders from scanning your "cgi-bin" and "tmp" directories (where most webmasters keep the server-side scripts and temporarily files) however the Googlebot will have access to it.
Once the robots.txt file is ready, be sure to place it in the root of your Web site hierarchy.
You can generate a robots.txt file using Google Search Console (formerly Google Webmaster Tools). To do this click the site you want, then under Site configuration,click Crawler access, click the Generate robots.txt tab and follow mentioned steps. To check that your robots.txt file is behaving as expected, use the Test robots.txttool in Search Console.
Tip: You should avoid making lots of changes to your site's robots.txt file all at once. Why? Because if you make a mistake it will be easy to see exactly what you changed so you can easily roll it back. It's important that you be very careful whenever you're making changes that impact which of your pages get listed in the search results. Small-steps are always a good idea.
For complete guidelines, visit Robots.txt Primer: Get Your Pages Indexed Faster by Controlling Google's Spider.