How to Setup robots.txt for SEO

How to Setup robots.txt for SEO


Google Indexing is very well known and important issue to the SEO professionals and Digital Marketers. That’s why they always keep themselves dedicated to index their target pages and Google crawl. They spend their time, money, resources, etc for doing their best indexing and best Google crawl their pages.
They do on-page as well as off-page optimization, image optimization, link building, social bookmarking, etc for their good ranking. But if they forget about the technical part which is very minor in volume but large in effect, then there arise huge bad impact for the terms of SEO as well as Google ranking.

The Local SEO Expert Guide (LSEG) is ready to give you the support for setting up the robots.txt. I am here going to discuss about robots.txt and its effect on SEO.

What is robots.txt?

A text file which is used to give instructions to the search engine bots (also known as crawlers, robots, or spiders) on how to crawl and index website pages is the robots.txt.

This file is kept into the root directory of the website so that the crawlers can access the instruction perfectly. This file treats as the part of the robots exclusion protocol (REP), which is a group of web standards that regulate on how robots crawl the web, access and index content as well as serve that content up to users. The REP also has the list of inclusion of directives like meta robots, pages, subdirectory or site-wide instructions on how search engines should treat links such as “follow” or “nofollow”.

Location of the robots.txt File

If everything is right according to the concepts of SEO, please check your robots.txt with Google’s robots.txt testing tool as this file will speed up the whole indexing process.

Basic Format of robots.txt

User-agent: [user-agent name]
Disallow: [URL string not to be crawled]

To be completed, a robots.txt file must have the above two lines. The file can contain multiple lines of user agents and directives like allows, disallows, crawl-delays, etc.
Each set of user-agent directives appear as a directive set, separated by a line break.

User Agent Directives Separated By Line Break
User Agent Directives Separated By Line Break

In order to access a robots.txt file, simply type the /robots.txt just after the domain like the below


Check robots.txt
Check robots.txt

Technical Syntax of robots.txt

The robots.txt file has to use the following technical terms according to the Google page speed test tool MOZ like below:

  • User-agent: The specific web crawler to which you’re giving crawl instructions (usually a search engine). A list of most user agents can be found here.
  • Disallow: The command used to tell a user-agent not to crawl particular URL. Only one “Disallow:” line is allowed for each URL.
  • Allow (Only applicable for Googlebot): The command to tell Googlebot it can access a page or subfolder even though its parent page or subfolder may be disallowed.
  • Crawl-delay: How many milliseconds a crawler should wait before loading and crawling page content. Note that Googlebot does not acknowledge this command, but crawl rate can be set in Google Search Console.
  • Sitemap: Used to call out the location of any XML sitemap(s) associated with this URL. Note this command is only supported by Google, Ask, Bing, and Yahoo.


Steps to Setup the robots.txt

The process to setup the robots are described by the SearchEngineJournal as below

  • Place your robots.txt file in the top-level directory of your website code to simplify crawling and indexing.
  • Structure your robots.txt properly, like this: User-agent → Disallow → Allow → Host → Sitemap. This way, search engine spiders access categories and web pages in the appropriate order.
  • Make sure that every URL you want to “Allow:” or “Disallow:” is placed on an individual line. If several URLs appear on one single line, crawlers will have a problem accessing them.
  • Use lowercase to name your robots.txt. Having “robots.txt” is always better than “Robots.TXT.” Also, file names are case sensitive.
  • Don’t separate query parameters with spacing. For instance, a line query like this “/cars/ /audi/” would cause mistakes in the robots.txt file.
  • Don’t use any special characters except * and $. Other characters aren’t recognized.
  • Create separate robots.txt files for different subdomains. For example, “” and “” have individual files with directory- and page-specific directives.
  • Use # to leave comments in your robots.txt file. Crawlers don’t honor lines with the # character.
  • Don’t rely on robots.txt for security purposes. Use passwords and other security mechanisms to protect your site from hacking, scraping, and data fraud.


Benefits of robots.txt

Robots.txt file can control crawler access to the specific areas of your site. For any reason, if Googlebot disallows whole site to crawl, there will be handy ways to be recover from this dangerous scenario.
Few common use cases include:

  • Able to keep private a whole section or department of your website like the admin section or the accounts section
  • It protects from appearing the duplicate contents in SERPs
  • It keeps the internal search results pages from showing up on a public SERP
  • It can species the sitemaps location
  • It prevents search engines to index certain files of your website (images, PDFs, html, php, etc)
  • It can define the delay to crawl the pages to make the server’s load balance

If there are no requirements of areas on your site to which you want to control user-agent access, you may not need a robots.txt file at all.

33 thoughts on “How to Setup robots.txt for SEO

  1. Hey there! I know this is somewhat off topic but I was wondering which blog platform are you using for this website? I’m getting fed up of WordPress because I’ve had issues with hackers and I’m looking at alternatives for another platform. I would be great if you could point me in the direction of a good platform.

  2. you’re in point of fact a good webmaster. The web site loading velocity is incredible. It sort of feels that you’re doing any distinctive trick. Also, The contents are masterpiece. you have done a excellent task in this subject!

  3. I was curious if you ever considered changing the page
    layout of your website? Its very well written; I love what youve got
    to say. But maybe you could a little more in the way of content so people could connect with it better.
    Youve got an awful lot of text for only having 1 or two pictures.
    Maybe you could space it out better?

  4. Hey there! Someone in my Myspace group shared this site
    with us so I came to take a look. I’m definitely enjoying the information. I’m bookmarking and will be tweeting this to my
    followers! Superb blog and brilliant design and style.

  5. I am really enjoying the theme/design of your website.
    Do you ever run into any internet browser compatibility problems?
    A small number of my blog visitors have complained about my site not operating
    correctly in Explorer but looks great in Safari.
    Do you have any advice to help fix this issue?

  6. Great beat ! I wish to apprentice while you amend your website,
    how could i subscribe for a blog website? The account aided me a acceptable deal.
    I had been a little bit acquainted of this your broadcast offered bright clear idea

  7. Hey I know this is off topic but I was wondering if you knew of any widgets I could
    add to my blog that automatically tweet my newest twitter updates.
    I’ve been looking for a plug-in like this for quite some time and was
    hoping maybe you would have some experience with something
    like this. Please let me know if you run into anything.

    I truly enjoy reading your blog and I look forward to
    your new updates.

  8. Its like you read my mind! You seem to know so much about this, like you
    wrote the book in it or something. I think that you can do with some pics to drive the message
    home a bit, but other than that, this is magnificent blog.

    A great read. I’ll definitely be back.

  9. Nice blog here! Also your web site so much up very fast!
    What web host are you the use of? Can I am getting your affiliate hyperlink to your host?

    I want my web site loaded up as fast as yours lol

  10. hey there and thank you for your information – I’ve certainly picked up anything new from right here.
    I did however expertise a few technical issues using this web site, since
    I experienced to reload the site lots of times previous to I could get it to load properly.
    I had been wondering if your hosting is OK?

    Not that I am complaining, but sluggish loading instances times will very frequently affect your placement in google and could damage your high quality score if advertising and marketing
    with Adwords. Well I am adding this RSS to my e-mail and could look out for much more of your respective exciting content.
    Ensure that you update this again very soon.

  11. You are so awesome! I don’t believe I have read through a single thing like this before.
    So nice to find someone with some genuine thoughts on this subject matter.
    Really.. thank you for starting this up. This site is something that’s needed on the internet, someone with some originality!

  12. I’m really enjoying the theme/design of your website. Do you ever run into any internet browser compatibility problems?
    A small number of my blog readers have complained about my site not operating correctly in Explorer
    but looks great in Chrome. Do you have any recommendations
    to help fix this issue?

  13. Unquestionably believe that which you said. Your favorite justification appeared to be on the net the
    simplest thing to be aware of. I say to you, I definitely get annoyed while people think
    about worries that they plainly don’t know about. You managed to hit the nail upon the top
    and also defined out the whole thing without having side
    effect , people can take a signal. Will probably be back to get more.

  14. Howdy! This is my first visit to your blog!

    We are a collection of volunteers and starting
    a new project in a community in the same niche.
    Your blog provided us useful information to work
    on. You have done a marvellous job!

  15. I used to be suggested this web site through my cousin. I’m not certain whether or not this post is written through him as nobody else recognize such
    exact approximately my difficulty. You are wonderful!
    Thank you!

  16. It’s the best time to make some plans for the future and it’s time to be happy.
    I’ve read this post and if I could I wish to suggest you
    some interesting things or advice. Perhaps you can write next articles referring to this article.
    I desire to read more things about it!

Leave a Reply

Your email address will not be published.