This answer has moved

Read the combined answer: How should I use robots.txt to control AI crawler access?.