A catalogue for people designing, running or reviewing studies with AI agents. These failures can arise when an agent designs the experiment, builds the analysis, optimises a method or writes up the result. People make them too. An agent can carry a mistaken assumption through enough code and polished prose to make it hard to spot.
The central check is whether the claim matches what the study measured. For each headline or conclusion, write down:
Which system and setup were tested, what inputs did each method receive, what comparison and scoring rule were used, and which data influenced the design?
Then compare that sentence with the published claim. A claim that performance generalises to new data needs an evaluation independent of the choices made during development. Check the measured sentence against the saved inputs, code and results. A well-written methods section cannot establish that the code followed the method.