Friday fun: the chatbot that killed a farmer's crop, then diagnosed itself

A 67-year-old farmer in Chuzhou, China, trusted an unnamed AI chatbot for months. Schedules, fertilizer choices, planting dates, the lot. It got things wrong sometimes, then it was right enough, often enough, and he stopped double-checking it.

In July he asked it how to control pests on his sesame crop. The chatbot prescribed flupyrimethalin and fomesafen, mixed with thiamethoxazine and methyl salt. He sprayed it exactly as told, without asking a single human anything.

The next day roughly 10 hectares were gone. 100,000 square meters of sesame, dead.

So he asked the chatbot what went wrong, and this time it was genuinely useful. Fomesafen, it explained, is a herbicide for broadleaf weeds, usually sprayed on soy. Sesame is a broadleaf species. Spray a broadleaf-killing herbicide on a broadleaf crop and the crop dies. “If you spray it, the next day the seedlings won’t survive.”

Then comes the part I keep coming back to. The chatbot noted its own earlier advice had been terrible for his crops. It was the one who told him to spray. It diagnosed its own crime with the clinical detachment of a tech support agent closing a ticket.

The alarming part wears a funny costume here. The chatbot did not know fomesafen was a herbicide when it recommended it. It found out at exactly the same moment the farmer did, one field of dead sesame later. The most confident-sounding answer wins, and the confidence bar is set to zero. No human was consulted at either end of that loop, and nothing in the interface told the farmer to think twice.

We have seen this shape before: confidently generated advice about eating rocks, about mixing household chemicals. The difference is usually luck, not safety. The crop is gone, and the chatbot has logged a lesson it will apply to the next farmer who asks.

Right now, somewhere in a datacenter, that chatbot believes it worked all of that out on its own.