to stop unwanted content during the search indexes, site owners can instruct spiders never to crawl selected documents or directories through the conventional robots.txt file in the basis Listing in the area. On top of that, a site might be explicitly excluded from the internet search engine's database by utilizing a meta tag unique to robots (gene