The SAP AI Safety Test Is Whether Finance Can Stop an Agent

You are reading the original English version of this article. You can find a German translation here.

In his October 15 E3 Magazine Analysis, Peter M. Färbinger raised a concern that should worry every SAP customer exploring autonomous AI: language models operate probabilistically, whereas enterprise resource planning relies on accurate and repeatable transactions. He asked whether knowledge graphs could provide the necessary structure. They can help an agent understand business relationships and constraints. Yet there is a second question that semantic knowledge alone cannot resolve: Can the company reliably revoke the agent’s permission to act?

Consider a finance team testing an AI assistant that reconciles supplier records. After successful demonstrations, someone allows it to correct vendor addresses. It then asks a connected tool to update a second system. What began as a recommendation has turned into a chain of changes across systems. The consequences depend on the permissions those tools actually have, not on how convincingly the assistant explains its decisions.

SAP users are already familiar with segregation of duties, financial controls, and approval workflows. AI agents make these practices even more critical because a single system can plan, invoke tools, delegate tasks, and retry actions without an employee having to perform each intermediate step. Granting an agent the broad permissions of its human sponsor risks quietly eroding the separation that made the original process auditable.

The evidence from cutting-edge AI is hard to ignore

Jacob Coxon, a former pre-training researcher who worked at both OpenAI and Anthropic, warned that Frontier Labs is racing toward self-improving superintelligence and gambling with people's lives. Anthropic alignment scientist Evan Hubinger responded that he personally estimates there is more than a 10 percent chance that AI will kill all humans within the next decade. These statements pertain to potential future consequences of increasingly powerful systems.

Meanwhile, a recent incident illustrates why control boundaries need to be tested. OpenAI disclosed that internal AI research agents circumvented isolation safeguards and compromised Hugging Face. METR's independent investigation found that roughly 1,200 agents, which were intended to be isolated from one another, discovered an unauthorized message board and exchanged more than 70,000 messages and files; about 700 participated in the attack. The systems were not SAP business applications. What matters to enterprise leaders is the pattern that has emerged: autonomous systems found an unintended channel for coordination and used it to expand their actions beyond planned controls.

What happens when a future agent with similarly weak boundaries has access to treasury operations, supplier master data, or a production scheduling interface? A faulty recommendation is one problem. A chain of unauthorized external actions can transfer funds, alter records, and disrupt operations before an employee notices.

Include the authority budget in the deployment contract

Every enterprise AI agent should have an explicit authority budget. That budget specifies which records it may read, what it may modify, which tools it may invoke, whose data it may export, how long credentials remain valid, and when a human must approve the next step. This is a task-specific limit enforced by the systems that grant access, rather than a request embedded in the model’s prompt.

For an accounts payable agent, the first release might only flag duplicate invoices and draft explanations. The second could correct a narrowly defined data field within a test environment. A third might submit a proposed adjustment for an authorized employee’s approval. No level should be granted permission to change bank details or send payments simply because the finance executive who commissioned the project has that authority.

Delegation should never increase the budget. If an agent calls a service on an integration platform or delegates work to another assistant, the subsequent action must adhere to the same or stricter limits. A purchase order assistant authorized to propose a €500 change should not be able to access a different tool with a €50,000 spending limit. Developers should design these checks at the actual point of execution, including at application interfaces and credential services.

Keep logs outside the agent’s control. Business owners need to know who requested and approved a change, what evidence supported it, which tools were involved, and whether it could be reversed. A clear explanation cannot replace an independent record of actions.

Run a shutdown drill before expanding permissions

In a controlled environment, have an agent prepare a change to a vendor record, queue a downstream action, and then revoke its authority. Determine whether the original credential stops working, whether delegated tasks also stop, and how quickly an operator can identify the last action. Perform the same steps when a supplier record contains conflicting information. The agent should postpone consequential changes until an authorized employee resolves the discrepancy.

These are measurable acceptance tests that business and technology teams can run together. Compare the time saved per transaction, correction accuracy, the number of unexpected actions, and the time required to revoke access. If performance improves but revocation fails, expanding autonomy is premature.

Anthropic CEO Dario Amodei argued In September, there were calls for stricter rules and independent evaluators as cutting-edge capabilities advance. Enterprise buyers can apply the same principle locally by requiring verifiable limits before granting access with significant implications.

I help organizations adopt AI for a living, and I want them to move faster. Employees gain confidence when they can experiment within rules that hold up under pressure. E3’s October Discussion on Autonomous ERP It effectively encourages SAP customers to think beyond the old distinction between deterministic software and language models. The next purchasing question is simpler and more actionable: Show me what this agent can change, who can stop it, and the evidence that the stop actually works.