Safety and Governance for Autonomous Science Agents

Scope

Write the autonomy charter before the first demo

An autonomy charter states what the agent can do without approval, what requires approval, and what is never delegated. It should cover data access, code execution, compute spend, external communication, laboratory actions, procurement, publication, and safety-sensitive domains.

The charter should be short enough that every researcher can read it and specific enough that engineers can enforce it. "Human in the loop" is not a policy unless it says which human, at what step, with which evidence.

Risk mapping

Map risks to actions, not abstractions

NIST's AI RMF vocabulary is useful, but autonomous science teams need action-level maps. Reading public papers is one action. Querying private patient data is another. Running a docking simulation is another. Triggering a synthesis robot is another.

Each action should have a risk class, required controls, logging requirements, and escalation path. The map should be versioned because tool capabilities change.

"map, measure, manage, and govern"

AI Risk Management Framework, NIST

Dual use

Screen domains before connecting tools

Some scientific domains carry dual-use or misuse risk. The safer architecture is to screen the domain before the agent sees operational tool access. If a task involves regulated data, hazardous materials, biological protocols, controlled substances, or security-relevant research, the agent should remain advisory until approved by a responsible owner.

The system should record refusals and escalations. A refusal is not a product failure when the requested action is outside the charter.

Publication

Do not let the agent publish its own authority

Autonomous systems can generate confident manuscripts and review responses. That makes publication controls important. The agent may draft, format, and assemble evidence, but a responsible human author must verify claims, authorship, conflicts, methods, data availability, and limitations.

The Messeri and Crockett warning about illusions of understanding is especially relevant here. A polished generated report can compress uncertainty into clean prose before the team has done the hard interpretive work.

"illusions of understanding"

Artificial intelligence and illusions of understanding in scientific research, Nature

Monitoring

Runtime monitoring should include scientific failures

Monitor more than crashes. Track protocol changes, repeated retries, anomalous outputs, unit conversions, unexpected tool costs, missing data, reviewer overrides, and divergence between planned and executed steps.

A good incident review for an AI scientist asks both engineering and scientific questions: Did the tool fail? Did the agent notice? Did the method still answer the question? Did the audit trail preserve enough evidence?