The Evolution of the Modern Data Engineer: From Coders to Architects

The role of data engineers is fundamentally changing. It’s no longer enough to focus on coding and pipelines, data engineers today must think bigger — about architectures and business goals, efficiency gains and agentic AI tools. With more responsibility and influence, and with AI use and adoption accelerating, data engineers are integral to overall business success. And this isn’t just my opinion — 72% of global business leaders agree, according to a recent report by MIT Technology Review Insights. 

The study, done in partnership with Snowflake, asked 400 senior technology executives from a broad range of industries to weigh in on the role of data engineers. The vast majority of those executives considered data engineers to be pivotal enablers of AI. But at the same time, data engineers are stretched incredibly thin, due to a proliferation of data and an AI-driven explosion in how data is being used.  This means that data engineers must become more productive by adopting AI, offloading infrastructure management to better platforms, and elevating themselves to become true partners to the business.

Going beyond tactical execution to strategic oversight

I see two major trends contributing to the raised profile of data engineers. The first is the sheer volume of information being generated and made available today — a function of data being easier to collect and cheaper to store than ever before. The second is a growing realization among top executives that without using data to inform decisions, their organizations are essentially flying blind. This is driving demand for AI to enable the use of data in more and more complex decisions than ever before.

By and large, the companies that use data effectively will make better decisions and, over time, outcompete those that don’t or believe they can’t.

To get full value out of the information they possess, companies need to travel a path from data to insights — and that road is built by data engineers. As more and more AI projects take off, the demands on data engineering teams are also growing. The survey showed that 77% of respondents see their data engineers’ workloads increasing. 

To keep up with the exponential growth in data volumes and AI projects, data engineers will need to be significantly more productive at scale. A company simply can’t grow the number of engineers in proportion to the workload, or at some point the whole company would be data engineers. So we are seeing things like AI copilots and coding tools that may eventually turn into autonomous agents to take on a lot of that work. 

Over time, I think the role of the data engineer will be less about writing code for every single pipeline and more about managing the infrastructure that AI agents are operating in. Data engineers will oversee orchestration across a lot of these pipelines, and set the rules and tests to make sure that the right data is coming in. But the actual construction of pipelines? It’s not hard to see a future where that will largely become the work of AI agents. 

And that should be great news to data engineers today. It means teams can focus more on tackling strategic problems, rather than routine, tactical busywork. They can consider questions like “What are our overarching goals?” and “How do we stop focusing on individual pipelines and think more about our overall data estate instead?” I believe this could be the next phase of data engineering, and today's shift toward architecting AI infrastructure is moving us there.

A changing function for the future of business

What this shift in role and responsibility really means, though, is that data engineers will need to be more familiar with business concepts. Consider how the ATM fundamentally changed the job of the bank teller — from a human cash dispenser to a customer-oriented banking specialist. The change ultimately required tellers to be more strategic thinkers, taking on more complex and high-value tasks. It upleveled the role overall, rather than eliminate the job altogether, as some had feared. In fact, the introduction of the ATM ultimately created more bank teller jobs because, with fewer tellers needed per branch, banks were able to open more branches, which lifted customer satisfaction through convenience and accessibility. 

In a similar vein, data engineers in the future will need to uplevel their thinking to prioritize understanding business goals and outcomes just as, say, business analysts do. They’ll have to ask themselves: What problem are we actually trying to solve? What insight are we trying to generate? What are we trying to improve? 

Understanding the ultimate needs and answers to those questions can create tremendous value for any business and, in turn, make data engineers even more valuable drivers of success. At Snowflake, we are committed to helping data engineers realize that potential by giving them the right tools to manage their growing workloads and enable AI at scale.

With all of that said, I invite you to participate in BUILD, our global virtual developers conference, from Nov. 4-7, and discover more insights from the MIT Tech Review survey report here.