Artificial Intelligence Without the Cloud Requirement: SAP and AI—Side by Side

For the past two or three years, SAP has been vigorously driving the integration of artificial intelligence into its solutions, while focusing on the latest S/4 HANA releases and the SAP Cloud. The Walldorf-based software company has certainly been able to win over new customers with this approach—at least, according to SAP CEO Christian Klein, two-thirds of new contracts now include AI components. For many existing customers, however, switching to S/4 HANA and the cloud is not an option, as they have complex on-premises environments with SAP ECC—some of which have evolved over decades. Migrating these environments would be extremely complex and costly and would likely take several years. Much of the careful customization done in the past would be lost.

In addition, more and more companies are now prioritizing data sovereignty and are particularly skeptical about moving business-critical data and systems to hyperscalers’ clouds. They fear a loss of control and dependencies that are difficult to resolve when necessary, making it nearly impossible to switch to other providers in the event of sudden cost increases, major functional changes, or restrictions on access to services. According to a study by Red Hat, for example, only 57 percent of German companies have a defined exit strategy for their primary AI provider, and even among these companies, 37 percent anticipate moderate to significant impacts on their business continuity should they actually have to switch providers.

Separation of SAP and AI

Regardless of this, depending on the industry, there are also regulatory requirements of varying stringency regarding data storage, which more or less bar many companies—including those in the financial and healthcare sectors—from moving to the cloud. When it comes to using AI in the SAP ecosystem, these companies are thus left out and cannot leverage the wealth of data from their SAP systems to gain new insights and improve processes and decision-making—at least not with the AI components provided by SAP, such as SAP Joule.

While it is possible to enhance existing on-premises systems with AI using third-party add-ons, many of these add-ons are also cloud-based and come with dependencies and, not infrequently, significant costs. A compelling approach to integrating AI into traditional SAP, however, is the side-by-side model. In this model, the AI systems remain separate from the core SAP systems and are operated in a separate, closely integrated architecture. The existing SAP landscape remains unchanged and does not need to be transformed or migrated. It continues to serve as the central system for business data, which AI systems can access via standardized interfaces such as OData or REST. The results of the AI analyses can then be fed back via the APIs—typically in JSON format—and displayed in SAP applications or other custom-built tools. These tools can be designed using SAP’s Fiori design system to match the look and feel of SAP applications, ensuring a consistent user experience and helping users navigate the system quickly.

The approach of running AI outside of SAP systems offers several advantages. For one thing, companies can develop AI systems independently of their core systems, which enables rapid implementation and deployment and simplifies ongoing maintenance. The SAP core remains untouched, ensuring there are no compatibility issues with future updates and upgrades. Additionally, companies are not locked into specific AI models but can choose freely. Thus, in addition to numerous Large Language Models (LLMs), a wide range of Small Language Models (SLMs) is also available. While these do not possess the broad general knowledge of LLMs with billions of parameters, they do offer detailed specialized knowledge in specific areas, which leads to better results in many use cases. Furthermore, the resource requirements of small models are significantly lower, meaning that less powerful hardware is needed and both the acquisition and operating costs of the systems are lower.

Free choice of model

The freedom to flexibly select AI models makes it possible to switch when necessary—for example, when better models or model versions come onto the market, or when the company’s requirements change. In addition, it is possible to specifically choose open-source models, which offer advantages in terms of the traceability and explainability of decisions. Unlike many proprietary models, they do not operate as a “black box” where it remains completely unclear how they arrive at their results. Their transparency is particularly important in highly regulated industries, but also wherever the goal is to achieve deterministic results. After all, in areas such as functional safety, medical diagnoses, credit decisions, and the evaluation of job applicants, AI must always produce the same output for the same input and must not behave unpredictably.

In addition to SAP-RPT-1, which is provided by SAP itself, other proven model families for analyzing business data from SAP systems include IBM Granite, LLaMA, and Mistral.

Unlike the models run in the cloud by AI market leaders and hyperscalers, these models—deployed within a company’s own infrastructure—can be flexibly expanded to include company-specific expertise without sensitive data leaving the organization. Thus, the models not only possess the knowledge pre-trained by the provider or the community, but also internal expertise drawn from databases, file shares, and SAP systems. The information stored there can be integrated directly into the model via fine-tuning or connected via Retrieval Augmented Generation (RAG).

Knowledge Base for AI Models

Finetuning improves not only the model’s knowledge but also its contextual understanding and generative capabilities with regard to domain-specific terminology; however, it is relatively resource-intensive. Companies require high-performance IT systems—typically expensive GPU clusters—which is why it often takes place in the cloud. In this case, data must be anonymized or synthetic data must be generated for training. The model is then run on-premises and processes the actual company data exclusively within the company’s own infrastructure. If new knowledge is to be incorporated into the model, retraining is required—so the effort involved is significant. The open-source project InstructLab, launched by IBM and Red Hat, therefore supports fine-tuning and retraining by enabling model adjustments using a specialized method that requires less data, providing high-quality synthetic data, and conserving computing resources. It significantly lowers the barriers to entry for model adjustments—especially for companies that lack AI experts and data scientists.

At RAG, on the other hand, the model accesses a vector database containing company information. This makes it easier to update the knowledge base, and the AI’s results can be readily supplemented with source citations, enabling employees to verify the results. Contextual understanding and generative capabilities remain unchanged—but for many use cases that involve only data analysis, this is entirely sufficient. Both approaches can also be combined as needed, for example, to teach a model subject-specific terminology and provide it with constantly updated data.

Efficient Inference

Running AI models on-premises offers the major advantage—beyond control over corporate data—that the AI runs close to the data, meaning the data does not first have to be transferred to the AI in a time-consuming process. Extensive data transfers to the cloud are therefore unnecessary, allowing companies to avoid high network costs. However, they do require local infrastructure for what is known as inference—the process of making decisions and generating responses based on data and queries. While the requirements are less demanding than those for training and fine-tuning, inference can quickly become resource-intensive as user numbers grow or additional AI projects are implemented. An efficient inference server that optimally utilizes GPUs and their memory helps reduce hardware costs and accelerate AI applications, so users don’t have to wait long for responses.

Ultimately, the main challenge for companies with the side-by-side model lies in consistently hosting, customizing, deploying, and monitoring the models for various use cases. After all, processes and framework conditions should remain as uniform as possible—not only for the sake of efficiency and to prevent the creation of AI silos. Security and data protection requirements can also be more easily met this way, including through centralized guardrails. These protective mechanisms check all inputs and outputs and prevent the AI from being manipulated or from producing erroneous or undesirable results. In this way, they also help ensure compliance with ethical standards and strengthen trust in the systems.

Consistent monitoring also ensures consistently high model performance. It detects biases that lead to unfair or discriminatory results, as well as model drifts—that is, a decline in the quality of the model’s outputs, perhaps because the data originally used to train the model is no longer up to date or because newly added data is influencing the model. Such drifts usually occur imperceptibly over time and are difficult to detect without monitoring tools.

Basis: Integrated Platform

Ideally, when building an AI architecture, companies should rely not only on SAP systems but also on an integrated platform that seamlessly connects to SAP and provides everything needed to consistently manage a wide variety of AI models throughout their entire lifecycle. Red Hat provides such a platform with Red Hat AI Enterprise. It is based on Red Hat OpenShift, a hybrid cloud application platform that supports the deployment and operation of models and applications in hybrid environments, enabling companies to use it not only for AI integration with SAP under the side-by-side model, but for all AI workloads —regardless of whether they run in their own data center, at the edge, or in the cloud, or whether they are physical, virtualized, or containerized environments. Red Hat AI Enterprise supports a broad ecosystem of open-source AI models and hardware accelerators, ensuring that companies remain flexible. The integrated inference server is based on the open-source vLLM inference engine, which scales seamlessly and combines efficient GPU utilization with stable, low latencies and an open, portable deployment. In addition, there are tools that automate the management of models and applications and promote GenAIOps and MLOps approaches. These practices—similar to DevOps for other applications—enable rapid, incremental, and iterative development and improve collaboration between data scientists, AI engineers, developers, and various IT operations teams.

The side-by-side model allows companies to enhance their SAP systems with advanced AI capabilities. For example, financial data can be used for AI-based invoice reconciliation to detect anomalies and prevent fraud. AI can optimize inventory levels based on weather and current events and, if necessary, trigger orders. It can use job descriptions from SAP to presort incoming applications. Or it can analyze camera images on the production floor to detect manufacturing defects and then enter the results into the SAP systems. The possible applications are extremely diverse, and there are essentially no limits to the imagination.

Of course, companies need a fair amount of expertise to customize and operate AI on their own. However, building this capability is worth the effort because it guarantees independence: It’s possible to adapt or switch models at any time—and in the fast-paced world of AI, this capability can be extremely valuable. Furthermore, AI expertise will help implement additional use cases in the future and can also save money in the long run. After all, cloud computing often results in surprisingly high costs, especially as user numbers grow or new AI applications are added. By then, however, the path back to an in-house data center is often blocked because the necessary infrastructure and expertise are no longer available.

However, when building expertise and implementing initial AI projects, the value of experienced partners should not be underestimated. They bring knowledge into the company and help prevent the company from losing focus and getting bogged down in never-ending projects that never make it into production. Ultimately, the side-by-side model thus offers a pragmatic solution not only for current challenges but also for future ones faced by companies with traditional SAP environments.

To the partner entry: