SAP Big Data - what is Big Data?

With the direct translation mass data one hits only one aspect. All normal data from the ERP system and other databases are also mass data.

With regard to the volume of data, we must speak of quantities that are too large for databases - too large in the absolute sense or in the cost/benefit sense.

The more interesting aspect is the degree of structure in the data. The ERP system contains 99 percent well-structured data, such as the MATART (material type) field in the MARA (material master) table.

One percent is free text like a delivery note. With Big Data, it's the other extreme and the exciting information is in the unstructured data areas. When and where a photo was taken is interesting, but what the picture shows is infinitely more important.

This is also accompanied by the type of data preparation. Whereas with databases it is a query such as "Total sales per month", in the above example we are suddenly talking about image analysis.

Even in cases that are not so extreme, such as log files, simple summations and counts are not performed. Databases are thus the worst choice for such data.

The most important definition of Big Data, however, is "all data that is not being used today to increase company profits." Creativity is the name of the game here. One of my recent projects involved writing down the utilization of servers in the data center - with the goal of reducing the number of servers.

An example: Sales are to be linked with information on how intensively customers have viewed the respective product on the website. For example, a product is advertised in the media.

Is this advertising being noticed? If so, we should see increased traffic on the associated product pages. Do prospective customers read the product page briefly, are immediately convinced, and then buy?

The web server already writes all page views to log files, but after a week they are deleted. So the data for this would be there, they are just not used yet.

The goal is maximum effectiveness and flexibility. A few years ago, Map Reduce on Hadoop was the ultimate, then came Apache Spark. It could do more, with better performance and greater power.

For a long time Apache Hive was the way to go, today it's Parquet Files. In such a dynamic environment, I don't want to spend a lot of resources on a potentially short-term solution, and I also want to have the openness to switch to something new at any time.

Currently, Apache Spark is such a powerful, but at the same time open solution. With it, the log files of the web server are broken down into rows and columns with one line of code. It is more complex to develop the logic of how the reading time per page can be derived from the history of page views.

If I finally add these and other key figures to the data warehouse, it enables combined analyses - for example, visualizing the key figures of sales, reading time and page impressions over time for a product.

Until recently, storing and processing secondary data was not attractively priced. The volume of data was too large, the information density too low, and the only way to process data effectively was with database-related tools.

The Apache Hadoop Filesystem (HDFS) can be used to form large filesystems from cheap PC components instead of buying an expensive disk array. Apache Spark can process these large data sets, with associated complex algorithms including statistical methods and machine learning.

Data warehouse tools, including those from SAP, have adapted to this situation and provide direct access to Hadoop files or send transformation tasks to a connected Spark cluster. A very simple way to read data from Hana is via the SAP Hana Spark Connector.