# wildcard charcter specifies all robots
User-agent: *

# disallows entire folder content
# each disallow must be on separate line
Disallow: /_database/
Disallow: /_derived/
Disallow: /_fpclass/
Disallow: /_private/
Disallow: /_scripts/
Disallow: /_stylesheets/
Disallow: /_vti_cnf/
Disallow: /_vti_pvt/
Disallow: /aspnet_client/
Disallow: /images/
Disallow: /moved/
Disallow: /valves/

# disallows individual files
Disallow: fprotate.class
Disallow: fprotatx.class
Disallow: global.asa

# Search engines will look in your root domain for a special file named robots.txt 
# http://www.mydomain.com/robots.txt The file tells the robot spider which files it may
# spider download. This system is called The Robots Exclusion Standard 
# The format for the robots.txt file is special. It consists of records. Each record 
# consists of two fields, a User-agent line and one or more Disallow lines. The format
# The robots.txt file should be created in Unix line ender mode Most good text editors
# will have a Unix mode or your FTP client should do the conversion for you. Do not 
# attempt to use an HTML editor that does not specifically have a text mode to create a 
# robots.txt file. 

# META NAME="ROBOTS" CONTENT="INDEX,FOLLOW"
# META NAME="ROBOTS" CONTENT="NOINDEX,FOLLOW"
# META NAME="ROBOTS" CONTENT="INDEX,NOFOLLOW"
# META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW"

# The Robots META tag is a tag to tell a robot if it is ok to index this page or not. It 
# also is used to invite a spider to walk down through all your pages. It is growing in # importance. 

# It is also useful if you don't have access to your servers root directory to control a 
# robots.txt file. 

# Some search engines, such as Inktomi now fully obey the Robots Meta Tag. Inktomi will 
# crawl down through a site if the Index,Follow syntax is used. 

# Robots Meta Tag Format 
# The Robots META tag is placed in the HEAD section of your HTML document: The format is
# quite simple: (case is not significant)


# HTML>
# HEAD>
# META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
# META NAME="DESCRIPTION" CONTENT="THIS PAGE ....">
# TITLE>...</TITLE>
# /HEAD>
# BODY>