Dignity nobility high-quality condition highness standing course caste position superiority precedence vogue difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robot regarding which web pages usually are not to become crawled. Like a online search engine crawler may https://www.youtube.com/watch?v=1amb7r2WR1g