This answer has moved
Read the combined answer:
How should I use robots.txt to control AI crawler access?
.