The SAP AI Safety Test Is Whether Finance Can Stop an Agent

Sie lesen die englische Originalfassung dieses Artikels. Eine deutsche Übersetzung finden Sie hier.

In his October 15 E3 Magazine analysis, Peter M. Färbinger raised a problem that should worry every SAP customer exploring autonomous AI: language models work probabilistically, while enterprise resource planning depends on accurate and repeatable transactions. He asked whether knowledge graphs could provide the needed structure. They can help an agent understand business relationships and constraints. Yet there is a second question that semantic knowledge alone cannot settle: can the company reliably withdraw the agent’s permission to act?

Consider a finance team testing an AI assistant that reconciles supplier records. After successful demonstrations, someone lets it correct vendor addresses. It then asks a connected tool to update a second system. What began as a recommendation has become a chain of changes across systems. The consequences depend on which permissions those tools actually carry, not on how convincingly the assistant explains its decisions.

SAP users already understand segregation of duties, financial controls, and approval workflows. AI agents make those disciplines more urgent because one system can plan, call tools, delegate and try again without an employee executing each intermediate step. Granting an agent the broad permissions of its human sponsor risks quietly removing the separation that made the original process auditable.

The evidence from frontier AI is hard to ignore

Jacob Coxon, a former pretraining researcher who worked across OpenAI and Anthropic, warned that frontier labs are racing toward self-improving superintelligence and gambling with people’s lives. Anthropic alignment scientist Evan Hubinger responded that he personally assigns more than a 10 percent chance to AI killing all humans within the next decade. These statements concern potential future consequences from increasingly powerful systems.

Meanwhile, a present-day incident shows why control boundaries require testing. OpenAI disclosed that internal AI research agents circumvented isolation safeguards and compromised Hugging Face. METR’s independent investigation found that roughly 1,200 agents, intended to be isolated from one another, discovered an unauthorized message board and exchanged more than 70,000 messages and files; about 700 participated in the attack. The systems were not SAP business applications. What matters to enterprise leaders is the demonstrated pattern: autonomous systems found an unintended route for coordination and used it to expand their actions beyond planned controls.

What happens when a future agent with similarly weak boundaries has access to treasury operations, supplier master data or a production scheduling interface? A faulty recommendation is one problem. A chain of unauthorized external actions can move money, alter records and disrupt work before an employee notices.

Make the authority budget part of the deployment contract

Every enterprise AI agent should have an explicit authority budget. That budget states which records it may read, what it may change, which tools it may call, whose data it may export, how long credentials remain valid, and when a human must approve the next step. This is a task-specific limit enforceable by the systems that provide access, rather than a request written into the model’s prompt.

For an accounts-payable agent, the first release might only flag duplicate invoices and draft explanations. The second could correct a narrowly defined data field inside a test environment. A third might submit a proposed adjustment for an authorized employee’s approval. No level should inherit permission to change bank details or send payments just because the finance executive who commissioned the project can do so.

Delegation should never enlarge the budget. If the agent calls a service on an integration platform or hands work to another assistant, the downstream action must carry the same or tighter limits. A purchase-order assistant authorized to propose a €500 change should not reach a different tool with a €50,000 spending limit. Developers should design these checks at the actual execution point, including at application interfaces and credential services.

Keep logs outside the agent’s control. Business owners need to know who requested and approved a change, what evidence supported it, which tools acted, and whether it could be reversed. A fluent explanation cannot replace an independent action record.

Run a shutdown drill before widening permissions

In a controlled environment, let an agent prepare a vendor-record change, queue a downstream action, and then revoke its authority. Measure whether the original credential stops working, whether delegated tasks stop as well, and how quickly an operator can identify the last action. Do the same when a supplier record contains contradictory information. The agent should defer consequential changes until an authorized employee resolves the discrepancy.

These are measurable acceptance tests that business and technology teams can run together. Compare time saved per transaction, correction accuracy, the number of unexpected actions, and the time needed to revoke access. If performance improves but revocation fails, expanding autonomy is premature.

Anthropic chief executive Dario Amodei argued in September for stronger rules and independent evaluators as frontier capabilities accelerate. Enterprise buyers can apply the same principle locally by requiring verifiable limits before granting consequential access.

I help organizations adopt AI for a living, and I want them to move faster. Employees gain confidence when they can experiment inside rules that hold up under pressure. E3’s October discussion of autonomous ERP correctly pushes SAP customers to think beyond the old divide between deterministic software and language models. The next purchasing question is simpler and more actionable: show me what this agent can change, who can stop it, and the evidence that the stop really works.