The future belongs to data streaming pipelines
They typically use point-to-point data pipelines to move data between operational databases and a centralized data warehouse or data lake. ETL (extract, transform and load) pipelines, for example, ingest data, transform it in regular batches and later forward it to a downstream analytical data warehouse. ETL pipelines and reverse ETL pipelines also send the results of data analyses that take place in the warehouse back to operational databases and applications.
Even though companies today often operate dozens to hundreds of point-to-point data pipelines, more and more IT managers are coming to the conclusion that point-to-point and batch-based data pipelines are no longer fit for purpose. Older pipelines are usually not very flexible and are perceived as "black boxes" by developers, as they cannot be adapted and are difficult to transfer to other environments. When operational processes or data need to be adapted, data developers therefore avoid changing existing pipelines. Instead, they add more pipelines and the associated technical debt. The bottom line is that traditional ETL pipelines require too much computing power and storage space, which can lead to scaling and performance issues as well as high operational costs as data volumes and requirements increase.
Data streaming pipelines are a modern approach to providing data as a self-service product. Instead of sending data to a centralized warehouse or analytics tool, data streaming pipelines can capture changes in real time, enrich them in the flow and send them to downstream systems. Teams can use their own self-service access to process, share and reuse data wherever and whenever it is needed.
In contrast to conventional pipelines, data streaming pipelines can be created using declarative languages such as SQL. This avoids unnecessary operational tasks with a predefined logic of required operations. This approach helps maintain the balance between centralized continuous observability, security, policy management, compliance standards and the need for easily searchable and discoverable data.
