Missing Data Link

Data integration: real-time access for real-time decisions

Fast decisions are required, for which real-time access to data is necessary. There are three main hurdles in the way, which we will look at in more detail in this article. The example of Pitney Bowes, a global provider of shipping and mail processing solutions, e-commerce logistics and financial services, shows how companies can overcome them.

99 percent of companies struggle with data integration issues. For 65 percent, access to ERP data alone is difficult. These are findings from a recent global survey of ERP professionals conducted by Dimensional Research and Fivetran. The impact of the lack of data access is massive: 82 percent of ERP professionals surveyed say their company is working with outdated data. 85 percent see this as the cause of incorrect decisions. Delays in data analysis, and therefore in important decision-making, are cited as a problem by around 70 percent. Not even a quarter of respondents currently have real-time access to ERP data. Why is this percentage so low here? Three hurdles make it difficult for companies to integrate SAP ERP data. 

The high burden of isolated SAP ERP data on companies.

Limited access

Several factors make real-time access difficult. The most important factors include: the immense amount of data stored in SAP systems; the speed at which new data is constantly being generated; and the variety of different data formats and sources.

But access is not the only problem. In addition, the data is not constantly available; on the contrary, it is very volatile. Moreover, access alone is not enough. Only by linking SAP data with data from other company systems and external sources - ideally in real time - can companies draw insights from it. This is another area where many fail. That's because the pace at which new data is generated continues to increase. And the continued growth in the use of smartphones and social media, as well as interactive business processes, is creating an often crippling level of complexity. This is especially true when it comes to integrating ERP data with other operational functions such as CRM, sales or unstructured sources. But this is the only way to give companies a unified view of their data - and a huge potential to leverage it.

For example, to gain useful insights from OLTP (Online Transactional Processing) data using OLAP (Online Analytical Processing), the data must be extracted, transformed, and loaded into a data warehouse for analysis. Until now, it was common practice not to synchronize OLTP and OLAP systems end-to-end. Instead, data was loaded into the analytics systems on a daily or monthly basis using in-house developed data pipelines and manual processes. This gives users access to data, but it is out of date.

No seamless integration

The issue presents companies with the task of finding the ideal balance between access to SAP data at the necessary speed and scale on the one hand, and the cost of installing, licensing or developing appropriate tools on the other. Previously, this was hardly a problem because ERP data was only exported for very specific purposes. But now it is increasingly mission-critical to make enormous volumes of corporate data usable quickly and at scale.

Nevertheless, to keep costs as low as possible, many companies are building their own data pipelines. However, their data specialists are faced with the almost insurmountable task of keeping up with the exploding volumes of data. New product lines and new data sets are constantly being added, so companies must continually adapt or even rebuild their data pipelines. This not only makes seamless data integration difficult, but also ties up a lot of valuable time of data scientists and data engineers.

Limited access leads to inaccurate decisions and poor results.

Not optimal use

The fact that many companies are not getting the full value from their SAP ERP data is due to several reasons: Complexity, data spread across different silos and sources, for example cloud and on-premises; the lack of skilled professionals slows down the development of DIY data pipelines; and security concerns or lack of robust security and data protection concepts keep many companies from leveraging their SAP ERP data.

How can companies leap over these hurdles? Only with scalable and automated data pipelines can companies use their SAP data in such a way that they derive the greatest possible benefit from it. After all, this is the only way to get a handle on the ever-growing volumes of data, integrate SAP ERP data with unstructured data systems, and gain insights in real time. The most cost-effective way to achieve this goal is to replicate large volumes of data using log-based change data capture (CDC) technology. Data replication involves continuously copying data residing on one physical or virtual server or cloud to a second server or cloud instance. 

When it comes to ERP data, such a data replication solution must be able to process the immense volumes of data present here. For access in (near) real time, it must also be able to capture changes on the primary server - the SAP system - and transmit them in real time to a target system - a cloud platform or a data warehouse. 

There are several CDC methods to replicate the changes between databases. For large amounts of data, log-based CDC technology is recommended. It is considered a superior replication method because it can process large amounts of data quickly and with minimal impact on transaction sources.

Best Practice

Pitney Bowes provides shipping and mail processing solutions, e-commerce logistics, and financial services used by some 750,000 customers worldwide. Its enterprise information management (EIM) team was extracting data from both SAP and Oracle every night for sales reporting. The volume of data integration from SAP to Snowflake had grown significantly, with the batch ETL process taking up to 31 hours. Also, the increasing number of companies requesting SAP data for dashboards and reporting could no longer be handled with the old ETL processes.

Because the existing system was not scalable to meet these growing demands, Pitney Bowes chose a CDC tool from Fivetran. In addition to its high flexibility, speed and scalability, seamless integration with the Snowflake database also played a role in the selection. Pitney Bowes also had difficulty finding a solution that was compatible with SAP. Fivetran scored with a feature set that allowed Pitney Bowes to easily replicate from SAP.

Today, Pitney Bowes synchronizes, validates and replicates data from Oracle and SAP into the cloud-based data warehouse four times a day. This gives users access to up-to-date SAP and Oracle lease data as a "single source of truth" in an enterprise database without any impact on source applications. Most of the replication management is handled through the interface. This means that the manual monitoring required for batch processing is eliminated.

Pitney Bowes has seen significantly faster processing times for both its Oracle and SAP systems. ETL jobs that used to take days are now completed in less than an hour. As a result, the EIM team can meet increasing demands for Oracle lease and SAP data. In addition, the "technical footprint" of the data collection process has been minimized, meaning redundant software licenses could be terminated and hardware and data storage reduced.