Falling Back in Love with Data Pipelines

published 18.05.2020 04:30

Image of article 'Falling Back in Love with Data Pipelines'

This is the dawn of a new era in data engineering, and many of the early adopters of this latest approach to pipeline development have found it enables creation with far less time, code and maintenance.

Declarative programming allows data engineers to focus on the big picture, drawing on their experience and creativity to design what the optimal data pipeline should look like, rather than spending time repairing failures, tuning parameters or manually tracing code.

Fortunately, for the data engineer and their muse, the data pipeline, there are advances in pipeline technology, bridging the gap and offering a path toward reconciliation.

That said, data pipelines are never far from my thoughts and the confluence of those two things got me thinking about data engineers and how their initial love for creating data pipelines often turns to loathing.

By embracing this new method of development, data engineers can again see a happy future where more of their time and skills are devoted to the things for which they–and the business–have real passion.