Robots.txt (Robots File) Explained – SEO Video Tutorial

Robots.txt (Robots File) Explained – SEO Video Tutorial
In this week’s web design video blog, Nick and I explain the purpose of a robots.txt file and also share the common Robot Exclusion Standards. Checkout the supporting code at
[ne_semantic_video video_id=”73RY142lb0E” title=”Robots.txt (Robots File) Explained – SEO Video Tutorial” upload_time=”2011-11-25T18:39:16.000Z” description=”In this week’s web design video blog, Nick and I explain the purpose of a robots.txt file and also share the common Robot Exclusion Standards. Checkout the” duration=”PT4M36S”]

24 thoughts on “Robots.txt (Robots File) Explained – SEO Video Tutorial”

  1. When i type this in my website
    User-agent *
    Disallow: /

    What does this mean… is this , no any google bot or search engine crawl my site… it is beneficial or not for my website.. we have another robots.txt file in my server which is
    User-agent *
    Allow: /

    How i can change this ….. Kindly tell me…

  2. Hello!I really have been making a living online now making use of the new PanCashful Secrets. It is a easy way to earn money online, and the beauty of it is that it is newbie friendly. It shows you some really good techniques. Pay bills, buy presents for family, and begin paying down that debt now! Just google PanCashful Secrets and learn how to begin making dollars at this moment.Just google for “PanCashful Secrets” to find out more.

  3. Hello, have you come across “HugeShortcuts”? Just head over to that website at On there you will see a valuable free video by an experienced web millionaire revealing how to make easy money online. It helped Aaron along with other everyday men and women to work straight from their home and consequently enjoy financial autonomy very fast. Hopefully it helps you also!

  4. Hey, have you heard of * MonEYEopener * or something like that (just do a Google search for it…)? At their site you will see practical information about how to make easy money online just by responding to simple and easy consumer surveys. It made it possible for Gary to work from the comfort of home and then benefit from monetary liberty very fast. Perhaps it will work for you as well.

  5. exactly. I don’t know if I agree with everything you said but you’ve got a point. By the way I am a marketer,and I utilize youtube to gain web traffic and this particular video is very helpful for me. thanks. you can visit my channel to learn more information about how i utilize the power of youtube to the benefit of my web marketing business _(how to get more traffic, boost sales, youtube views etc..)_. good luck

  6. I need to know how to optimize my robots.txt file, to know the top unwanted robots, the specified site and domain requirements for the major search engine bots, how promote reviewing of your website from the bots(their cronjobs) I mean the use of the noarchive attribute so google bot must visit you always when showing info about your site, also for the newbies: don’t worry if robots.txt isn’t working well yet with the search engines the most bots need time to revisit your site (some of them 2 times a month) also robots.txt tends to save itself in the cache memory of some browsers don’t worry if you upload it to the FTP and it doesn’t change in the browser (google webmaster tools are good help)

  7. crawl postpone because robot.txt was inaccessible I an getting this message in my webmaster tool what step to be take to remove this warning ……………………

  8. my photos are being shown in google search and i really need them to get deleted.
    and i do not know how to do this robotic thing.
    please i need solution

  9. in google webmaster, robots.txt file was showing red(unreachable) i re-corrected somehow and today it is showing yellow? Tell me it is temporary problem or do i need to copy exact command which you told in this video? Please reply

Comments are closed.


SDARR Studios is a digital marketing agency located in Chandler, Arizona. We specialize in content marketing, Search Engine Optimization, and link building. We have created strong business relationships with site owners around the world, making us a premier resource for link acquisition and featured guest posting.