The time of the lighthouses is over

As part of a student job in 1998, I naively asked the SAP Basis admin how I could access this SAP from my Visual Basic application.

At the time, users had to copy and paste material data from the SAP GUI into the external warehouse application. Wordlessly, he pressed a gray, printed softcover book into my hand:

"Mastering SAP Remote Function Call in C/C++". In retrospect, that was probably his way of saying: Don't do it.

Almost 20 years have passed since then and a lot of water has flowed down the Leimbach past the SAP headquarters in Walldorf. The questions about the business benefits and the right technical approach are still relevant.

Data and process integration

When it comes to SAP interfaces, we basically distinguish between two major areas: Data integration and process integration.

Data integration generally involves the transportation of large volumes of structured data. The purpose is to be seen in areas related to data analysis.

This can range from simple charts for the management level to predictive analytics and data mining with the help of artificial intelligence. In any case, this area should be viewed downstream of the classic business transactions that are generally covered by SAP ERP.

If you were to remain completely in the SAP world, you would cover the data analysis requirements in the classic way with BW, Hana and the BO front-end tools.

The second major area is process integration. This is where the data transfer is broken down to the individual transaction. A process could start in SAP and be transferred to a subsystem.

A good example of this would be a customer quotation that is created in SAP SD and then enriched with additional data, images and drawings outside of SAP.

The opposite direction is just as conceivable. Information on a new material master data record to be created is collected from various departments via a non-SAP workflow. This involves product management, purchasing and, if necessary, colleagues from logistics.

Only when all information is complete and consistent is the data and the investment process transferred to SAP MM. Virtually all SAP interfaces can be divided into these two categories of process and data integration, each of which is technically completely different.

Predetermined breaking points

Before we look at the technical aspects, we should ask the why question. With its almost unmanageable range of products and modules, SAP theoretically covers all the requirements of almost every existing SAP customer.

Every interface between systems or manufacturers is often seen as a kind of predetermined breaking point that becomes particularly annoying when it doesn't work. And, of course, the other party is usually to blame. And yet there are arguments that drown out these concerns.

Users often call for higher performance. I would like to expressly defend myself against the blanket accusation that SAP is slow, but nevertheless it has almost been a tradition among the Walldorf-based company since its foundation to design its software in such a way that the available hardware always feels a tad too slow for the applications.

For a long time, this was especially true for SAP BW. The use of Hana has certainly put things into perspective somewhat, but it was the data analysis area in particular that stretched the user's patience to the pain threshold and beyond.

It therefore stands to reason that many existing SAP customers find a significantly better ratio between costs and performance in external subsystems.

Another important argument is mixing with non-SAP data - whether in data or process integration. Keeping company data completely within a manufacturer's software only works in glossy brochures, but never in real life.

And yes, of course SAP ERP and SAP BW offer the option of loading external data into SAP and processing it there, but this is by no means a realistic, affordable procedure.

If you need SAP and non-SAP data at the same time, you will have to look for a place outside SAP or you will have to bring a lot of time and a large budget with you.

Specifications of biblical proportions

Digitalization is on everyone's lips. The media, politicians and company bosses interpret everything possible and impossible into this term. It also provides our third argument, which is actually much older than the term digitalization itself: Agility.

It is certainly due to the age and long history of SAP that certain patterns are often found in project management: At the start of a project, the consultant enters the meeting room, sharpens his pencil and listens to the future user about what he would like.

The specifications - often thicker than the Bible - go into implementation and after agonizingly long months or even years, the go-live takes place.

Unfortunately, the world has already turned three times and even the business user from the initial meeting is not a clairvoyant. But this is how SAP-managed processes are generally set up, which is the exact opposite of agility.

However, if we transfer the data to a suitable subsystem via an elegant interface, we can technically and organizationally ensure that the iteration cycles in projects shrink from months to days and that adaptability skyrockets. That is agility.

Incidentally, a major pioneer of this way of thinking is Self Service BI, which means nothing other than officially admitting that when creating an analysis data source, it is not yet known exactly which questions are to be answered with the dataset.

Anyone who thinks that users are now sitting in front of their computers with wet eyes and hands shaking with excitement is mistaken. Users have always done it this way anyway, with Excel.

We have now learned the three most important arguments: performance, blending with non-SAP data and agility. There are many, many more, but over the past twenty years and across almost 2,500 interface projects, these three are the distillation of what drives existing SAP customers on this topic. Interestingly, almost nothing has changed over time.

Technology for data transfer

Let's first look at the technical aspects of data integration, a typical sub-discipline of business intelligence. Completely independent of the manufacturer, this part is handled by a layer that we call ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform).

The data often still needs to be prepared for analysis, e.g. through quality and consistency checks. Depending on whether this transformation is carried out during transport or after arrival in the target system, it is referred to as ETL or ELT.

In every type of analysis system or data warehouse, such a layer can be found in various forms.

ETL and SQL

One of the most common scenarios, for example, would be data storage in a Microsoft SQL Server. The ETL part would be handled by SQL Server Integration Services (SSIS).

A tool that is delivered together with the SQL Server. Here, the data flows are modeled graphically and then automated. This works so well and inexpensively that even customers without a Microsoft background occasionally use SSIS to transport data to external systems (e.g. an Oracle data warehouse).

In any case, the desired transformations are carried out in transit. In addition to other ETL providers such as Alteryx, an alternative would be to leave the original data from the upstream systems as it is, store it first and then refine it downstream from a staging layer - ELT in other words.

This approach also works very well with all common data targets: SQL Server, Oracle, Hadoop applications, Amazon Redshift. The list goes on and on and depends on the customer's taste.

Once the data has been prepared for analysis, a practically infinite pool of analysis options opens up. From classic Excel and Power BI to providers such as Board or Tableau, all of which set standards in terms of user experience.

Even if SAP is theoretically just one of many data suppliers in such a landscape, experience has shown that it occupies a special position. This is mainly due to the fact that connecting SAP is technically more complex than others and can quickly turn into a penny pit if the wrong tools are used.

If we consider SAP ERP as a data supplier, a number of source objects are suitable as a starting point. In the simplest case, tables. While it is true that the number of SAP tables is somewhere in the six-figure range depending on the release and module, experience has shown that the really relevant ones are easy to find and prove to be a manageable problem in practice.

Many customers also like to fall back on queries. Although this is an impressively old technology, existing SAP customers with a long history in particular can fall back on existing artifacts without having to reinvent the wheel.

If the data volumes become larger, incremental loading must of course be taken into account. The simplest option is to use classic OLTP data sources, just as they would supply a BW with data.

Incidentally, SAP is currently renovating this type of data transfer and replacing it with the more elegant and robust ODP. It is therefore not foreseeable that this type of data transfer will fall victim to a Hana roadmap in the foreseeable future.

SAP BW and the toll called hub license

One important aspect is the role of SAP BW. Experience shows that two thirds of all customers access data from SAP ERP and one third from BW. In the simplest case, classic BEx queries can be used to access data via BW.

When data volumes increase, export datasources can ensure that data is transferred incrementally from the respective BW object to a downstream system.

One very important point should be mentioned at this point: If the data is not transported directly from the ERP, but from BW to an external data warehouse, depending on the license conditions and contract, a toll with the name Open Hub License must be paid to Walldorf.

Politically, this is obviously intended to discourage existing SAP customers from taking this route. In my experience, however, this tends to have the opposite effect. It encourages them financially to remove the existing BW from the data flow altogether.

The reasons for first transporting the data via a BW and only then into a DataMart are rather tenuous and often have more to do with politics and history than with technical necessity.

Process integration

The technology behind process integration is fundamentally different to the data integration in the previous section. This is primarily about returning a single transaction either from SAP to the outside or from the outside to SAP.

Workflow systems are often used on the external side. This could be Nintex, K2 or Microsoft Flow, for example. In each case technically embedded in a SharePoint or without.

If it is not a workflow application, we often find Office applications (most likely Excel) on the non-SAP side through to machine consumers such as a conveyor system or other production machines, which are also dependent on SAP data.

In the case of Excel, a typical use case could be either transferring data from SAP to an Excel sheet (e.g. customer address to customer number) and/or importing it back from there (e.g. maintaining material BOMs). The applications are diverse and usually a great efficiency gain for the end user.

They do not have to leave their familiar environments (Excel, SharePoint or other third-party software). In the past, SAP and Microsoft have made several attempts with Duet to take this approach.

This was hardly crowned with success and in the end good ideas were ground to dust by the wrong technical implementation and too much politics. On the SAP side, the starting point is practically always a function module. This can either be an existing standard BAPI or a self-developed Z-module.

Other technologies such as transaction recorders or similar have proven to be less flexible. The use of an SAP gateway should also be carefully considered. The SAP gateway is ultimately just an additional layer that translates the function module into OData.

This has two serious disadvantages: The additional layer reduces performance and responsiveness; moreover, only some business transactions can be squeezed into this table-oriented way of thinking of OData.

Use cases in which the data is very hierarchical and not table-like (e.g. a parts list or complex material master data) very quickly become ugly and confusing in Gateway, which not only limits agility in the development process, but often brings it to a complete standstill.

Practice has shown that access with as few layers as possible works best, easiest and fastest: accessing the module or the BA directly from the external system via RFC.

Either through a few lines of code that are programmed in-house, or through intelligent tools that encapsulate the programming in a graphical editor. Even if this sounds a little shirt-sleeved, in practice it has almost always proved to be superior to the gateways and PIs of this world.

Conclusion

We have learned about the most important aspects of SAP interfaces here. The first question that plays a role is whether the customer wants to transport mass data (data integration) or connect processes and transactions within SAP with the outside world.

The most important reasons for this are good performance and bringing SAP and non-SAP data closer together. Another reason is the need to build system landscapes and information flows in such a way that they are already prepared for future changes, i.e. are agile.

From a technical point of view, many objects on the SAP side can be considered data providers: Tables, queries, OLTP sources, BW queries, etc. When integrating transactions, the existing customer is well advised to omit intermediate layers as far as possible.

Even if those who would like to sell the intermediate layers naturally see it differently. The advantages are outweighed by better performance and agility. In conclusion, it is generally advisable to be more pragmatic and to embrace short feedback and iteration cycles.

The time of big lighthouse projects that solve all integration problems is over. At the end of the day, things have to function stably in the real world and not on glossy paper.