In SEO, crawling refers to the activities of the search engines which involves gaining access to contents on the websites, indexing and ranking them. Search engines crawlers or bot, with a well-optimized robot.txt will crawl through the website to index and rank such contents based on their usefulness.
In order to guide this crawler, it is always better to have a good sitemap and as well ensure that robot.txt is optimized to allow crawlers accessing the website.
In order to guide this crawler, it is always better to have a good sitemap and as well ensure that robot.txt is optimized to allow crawlers accessing the website.