Why use a robots.txt file?

Home
Community Discussion
SEO

spykaushik Level 1

November 8, 2013, 6:29 am

Write the reason you're deleting this FAQ

Why use a robots.txt file?

What’s the significance of Robots.txt file in a website?

Comments

Please login or sign up to leave a comment

Join

Layla

A robots txt file helps the search engine bot crawl your website to find all links. Sometimes, these links will be buried in obscure places on your site.

11 years ago

Are you sure you want to delete this post?

CountD

The Robots.txt file is a text document that informs the search engine spiders which parts of your website you would like to be crawled and which parts you do not want the search engine crawling .

11 years ago