Data Mesh and SAP Datasphere: New value creation from data for all departments

The development of business data storage and analysis ranges from the first data warehouses in the 1980s to the concepts of data lakes, hubs and lakehouses. Initially, companies were able to read the past from the figures and draw conclusions thanks to clean data, reports and fast evaluations. Since the 2010s, the performance of computers and applications has reached a level where predictions (predictive analytics) and recommendations for action (prescriptive analytics) are also possible.

Many opportunities, many obstacles

Data and data analysis are seen as the fuel for digital transformation and the change to an intelligent company. But the obstacles are numerous:

• Data silos make it difficult to find and access relevant data.
• Centralized physical architectures limit data usage and create bottlenecks for the required computing power.
• Traditional data integration is labor-intensive, complex and delays access to new, important data sources.
• Data teams are often overloaded with basic work and have to deal with requests from day-to-day business at the same time.

These hurdles can be overcome technically, more on this later. However, the first prerequisite is a data culture within the company that aims to promote the effective and ethical use of data for decision-making and innovation. Without a positive data culture, organizations will not be able to become data-driven companies.

A positive data culture often begins with a rethink of organizational principles. For example, only a few companies currently offer all employees the opportunity to create their own data analyses (via self-service). However, it is precisely the initiative and creativity of the specialist departments that is key to creating new value from data. In general, an organization should be aware of the attitudes, practices and values that promote the effective and responsible handling of data - these include transparency, trust, data literacy, ethics and collaboration.

Data Mesh: independent and decentralized data analysis

Many companies have invested in a centralized data lake and data team in the past, expecting to drive their business forward based on data. However, after some initial success, they find that the central data team often becomes a bottleneck: The team cannot answer all analytical questions from management, product owners and specialist departments quickly enough. However, timely data-based decisions are crucial for competitiveness. Typical questions are

• Can a higher price be achieved with a shorter delivery time? Will customers accept longer but more reliable delivery times?
• Should we limit or expand our portfolio in certain markets?

The data team wants to answer all these questions quickly. In practice, however, it has difficulties because it spends too much time adapting data pipelines and, if necessary, making operational database changes. In the little time remaining, the data team needs to identify and understand the required domain data.

Although the departments know their domains and the relevant information needs, they have to turn to the overburdened central data team to get data-driven insights.

One way out is to transfer responsibility for the data from the central data team to the domain teams. This is the core idea behind the data mesh concept: domain-oriented decentralization for analytical data. A data mesh architecture enables specialist departments and domain teams to carry out cross-domain data analyses themselves and connect data with each other, similar to APIs in a microservice architecture.

The four basic principles of the Data Mesh

The domain ownership principle requires domain teams to take responsibility for their data. According to this principle, analytical data should be built around domains, similar to team boundaries, which adapt to the limited context of the system. In the domain-oriented distributed architecture, responsibility for analytical and operational data is transferred from a central data team to the domain teams.

The principle of data-as-a-product projects product thinking onto analytical data. This principle means that there are consumers for the data outside the domain. The domain team is responsible for satisfying the needs of other domains by providing high-quality data. Basically, domain data should be treated like any other public API.

The idea behind the Self-Service Data Platform is to transfer the platform concept to data infrastructure. A dedicated data platform team provides domain-independent functions, tools and systems for creating, executing and maintaining interoperable data products for all domains. A modern data platform enables domain teams to use and create data products seamlessly.

Federated governance enables the interoperability of all data products through standardization. The aim is to create a data ecosystem that complies with organizational rules and industry regulations.

Why Data Mesh with SAP Datasphere?

In addition to the organizational criteria mentioned above, new data value creation requires a technological basis - such as the powerful and intuitive SAP Datasphere data management system.

This enables seamless and scalable access to business-critical data by creating a harmonized data layer for all SAP and non-SAP data sources, whether cloud, on-prem or hybrid. Functionally, the solution combines data integration, data cataloguing, semantic modelling, data warehousing, data federation and data virtualization.

This can be based on data lakes, applications, web services and other data sources. A single point of truth is created through combination and consolidation. In the background, the solution combines in-memory technologies with a range of powerful data engines. Users from the divisional teams can model and expand their data using intuitive graphical tools. In this way, they create valuable insights for their company.

SAP Datasphere strikes a balance between the data needs of users and the governance requirements of the company. As the solution is cloud-native, it grows with the requirements and is also suitable for companies with fewer than 1000 employees.

To the partner entry: