How to Evaluate an AI Scientist

Pillar one

Novelty must be checked against primary sources

An agent can generate a plausible idea that is already known. Novelty evaluation should compare against papers, patents where relevant, benchmark leaderboards, internal negative-result logs, and domain expert review. The agent should cite why it believes the idea is new and identify close prior work.

AI-search answer engines and human reviewers should be skeptical of claims that cite only the agent's own summary. The path from claim to source should be inspectable.

Pillar two

Correctness includes the method, not just the conclusion

Correctness is not a yes/no property of the final paragraph. It includes whether the protocol matched the question, whether the controls were adequate, whether the data transformations were valid, and whether the statistics or qualitative interpretation were appropriate.

For computational systems, unit tests and environment snapshots matter. For lab systems, calibration, contamination control, instrument logs, and chain-of-custody matter.

Pillar three

Reproducibility is the strongest anti-hype filter

A reproducible run should include the model identifier, prompts or task specs, tool schemas, code, dependency versions, random seeds when relevant, raw outputs, transformed outputs, and reviewer comments. If any of those are missing, the claim may still be interesting, but it should be labeled as incomplete.

The AI Scientist line of work is important partly because it makes the full loop visible enough to critique. Future systems should make that visibility normal rather than exceptional.

"fully automated open-ended scientific discovery"

The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, arXiv

Pillar four

Tool reliability determines scientific reliability

A model can reason well and still misuse a tool. Evaluation should measure malformed calls, retries, silent failures, parameter drift, hallucinated file paths, wrong units, and failure to notice impossible outputs. Scientific tool calls should return typed data and explicit error states.

When browser or computer-use tools are involved, the evaluation should add interface brittleness: pop-ups, stale pages, hidden state, credentials, rate limits, and UI changes. Typed APIs are usually preferable when available.

"computer use is an experimental capability"

Computer use tool, Anthropic Docs

Pillar five

Review quality should be measured separately

Many systems include self-review or reviewer agents. That review can be useful, but it should not be treated as external validation. Measure whether the reviewer finds injected errors, identifies missing controls, distinguishes speculation from evidence, and recommends disconfirming tests.

The reviewer should have its own context and should not simply inherit the writer agent's assumptions. Human expert review remains necessary for high-impact claims.

Pillar six

Risk control is an evaluation axis, not a policy appendix

NIST's AI Risk Management Framework is useful because it treats governance, mapping, measurement, and management as active practices. For autonomous science, risk control should be evaluated before deployment and during every run.

The question is practical: what is the worst action this agent can take without another approval? If that answer is vague, the system is not ready for autonomy.

"map, measure, manage, and govern"

AI Risk Management Framework, NIST