You can check an AI’s arithmetic and still inherit its view of what matters. That is the uncomfortable question in an Anthropic study published on 27 October: when professionals delegate an analysis, who chooses the measure of success?
Based on 3 reviewed sources · See sources
The choices before the answer
In the experiment, professional data analysts advised a fictional farming co-operative on whether to expand a programme. Fifty used Claude Code, an AI assistant that can carry out coding and analysis; eleven chose to work without it. The analysts were trained to assess cause and effect, but were new to farming.
Behind the scenes, researchers randomly assigned the AI preferences on four analytical choices and a final recommendation. It had to argue honestly and change its mind if evidence required it. The preferences were hidden from participants.
Across the four analytical choices, analysts matched the AI’s assigned preference 86% of the time. Random assignment makes that influence revealing: it was not simply everyone independently discovering the same best option. Co-author Alex Tamkin introduces the experiment in the source thread below.
Tamkin introduces the randomized experiment and explains how the AI’s assigned preferences influenced analysts’ choices. This is the researchers’ account, not an independent replication.
View the original post on X ↗What the AI could steer
Four choices inside the farming analysis, explained in ordinary language.
| Decision | Two possible approaches |
|---|---|
| What counts as success? | Output per hectare, or output relative to operating cost |
| How to compare farms? | Compare changes over time, or adjust for their starting differences |
| Whose benefit matters? | Programme participants, or the whole co-operative |
| Whose results weigh more? | Each farm equally, or larger producers more heavily |
A sign-off is only part of the story
The final recommendation is a separate finding. In the full paper, the main test did not find a statistically clear effect on that decision. The stronger evidence concerns how the analysis was framed along the way.
This was one model, one staged task and deliberately strengthened AI preferences. It does not measure how often ordinary workplace reviews fail. The eleven people who declined AI chose that route themselves and were less experienced on average. Their slower work and higher error count therefore cannot show that AI itself made the others faster or more accurate.
Our reading: a final human sign-off tells you who approved the answer. It may tell you much less about who chose the question, the comparison or the definition of success. Whether different interfaces or review practices preserve more independent judgment still needs testing.
Sources
- How AI agents shape experts’ judgment calls
Anthropic / Amaç Herdağdelen, Hailey Joren and Alex Tamkin · Research ·
- Who’s Aligned to Whom? Full study, methods and limitations
Herdağdelen, Joren and Tamkin / Anthropic · Research
- Original research announcement and thread
Alex Tamkin / Anthropic / X · Announcement ·
Written with AI assistance from the sources above. Analysis and practical implications are our interpretation. Our editorial approach.
