# Robots.txt file to exclude restricted pages from search engine crawlers # Search Engine crawler # Use 'Disallow: /folder/' to exclude directories Disallow: /_notes/ Disallow: /bin/ Disallow: /cgi-bin/ Disallow: /css/ Disallow: /files/ Disallow: /images/ Disallow: /Library/ Disallow: /scripts/ Disallow: /Templates/ User-agent: *