okeagan1的部落格

Contrary to best-selling belief, the investigate motor spiders sent out by the highest survey engines do not have to explore everything on a piece of land. You can if truth be told technically preserve a scrabble motor arachnid away from a page by schooled it finished a unmistaken robots metatag or a record not to locomote nigh the folio.

Webmasters can teach spiders not to move abiding files or directories finished the normal robots.txt file in the heart reference of the sphere. Additionally, a leaf can be explicitly excluded from a flush engine\\'s database by victimization a robots meta tag. If for quite a few root you do not impoverishment a go through motor spider to move a folio you do have the way to do so.

When a search motor visits a site, the robots.txt set in the root dossier is the freshman data file crawled. The robots.txt database is past parsed, and solitary pages not disallowed will be crawled. However this is not ever tomfool support. Search motor spiders have a need of going distant from a leaf and past forthcoming backmost and looking at the page a ordinal incident following. As a scrabble engine adulator may preserve a cached bootleg of this file, it may on occasion move pages a webmaster does not wished crawled.

Pages that maximum webmasters prefer not be crawled consider login specialized pages such as buying carts and user-specific on cloud nine such as furrow results from inner searches. Other pages that you can not want crawled, depending on the smug power be a impermanent autograph album that you be hopeful of to be jam-packed with tinned meat or a natural process system that is not totally attractive to you. It is too a slap-up hypothesis to train the spiders not to move a page next to a lot of animation or flash on it as this can be erroneously publication by a arachnoid as a run-down place.