12 lines
147 B
Plaintext
12 lines
147 B
Plaintext
User-agent: *
|
|
Crawl-delay: 1
|
|
Disallow: /magicat/
|
|
Disallow: /home/
|
|
Disallow: /doc/
|
|
|
|
User-agent: chklinks
|
|
Disallow:
|
|
|
|
User-agent: HTTrack
|
|
Disallow: /
|