The Demand and the Design: Navigating Data Engineering's Two Great Shifts
Data engineering today is in the midst of two major shifts — one of function and one of form. The first is obvious: AI is fundamentally redefining the function of data engineers at almost every level. Its insatiable appetite for data has created outsized demands of data engineering teams — demands needed for success and yet incredibly difficult to maintain. The second is a shift in form, in how data engineers must meet these new and growing demands. We've seen data engineers go from doing mostly rote, manual labor to more strategic execution, adopting best practices from software development to elevate the work they do. They're no longer mere data plumbers and pipeline constructors; they are the operational architects of any data-driven organization. And at this point, there is no going back.
When we think about modern data engineering, the focus is no longer on manually connecting each and every dot. That simply doesn't scale to meet the needs of AI. With exponentially increasing volumes of data rapidly becoming both available and usable, engineers need to work more efficiently to keep pace. That is where a more modern, declarative approach to building pipelines changes the whole game for data engineers. By abstracting away the minutiae of each step and instead focusing on the desired end state, data engineers have the power to multiply their productivity and make gains that previously seemed out of reach.
Take coding agents as an example. In a matter of months, these tools, including Cursor, Claude Code and Snowflake's Cortex Code, have revolutionized how we think about software development — and by extension, data engineering. How? For years, data engineering teams have been quietly adopting best practices from software-defined lifecycles. They're treating infrastructure as code and creating structured, version-controlled environments where data pipelines closely resemble stateless software code. Since these AI coding agents are trained heavily on software engineering problems, they are fairly readily able to adapt to this modern form of data engineering as well.
This shift in approach — to a more modern, declarative mindset — creates the right conditions to make AI tools functional. But more importantly, it provides the safety net needed to let AI operate at scale. In the past, fixing a pipeline meant running raw SQL commands directly in a production environment; but if something broke, it was incredibly complex to investigate what went wrong. Today, a modern approach means changes are checked into version control, tested and deployed only as a known good state. Having the ability to easily test changes and roll them back is a strict prerequisite before trusting AI to write or manage data workflows.
Now, trusting AI doesn't mean having blind faith. Instead, the key is building trust in the underlying data engineering process. We are already seeing organizations run thousands of data pipelines simultaneously, reaching a point where human oversight of every moving part is virtually impossible. Soon, we'll move toward agentic AI, where software agents will take on larger chunks of actual pipeline construction. Data engineers' roles will be elevated once again, moving away from writing individual scripts to advanced data modeling and system requirements. They'll function closer to the business, ensuring data availability and quality for AI, analytics and applications.
Ultimately, the future of data engineering isn't about writing better scripts to move data. It's about building the resilient systems that connect it for you. That is why Lead Developer Advocate Gilberto Hernandez wrote "Build Pipelines for AI: An Essential Guide to Smarter Data Engineering," a book designed to help you do just that. In it, he walks through the ITD (ingestion-transformation-delivery) framework for data pipelines and goes over the traditional approaches associated with each step — both their virtues and pitfalls. He highlights the modern tools and methods that can help data engineering teams adapt to the shifting landscape before them and prepare for the future that lies ahead.
