Data Engineering: Building and Operating Reliable Data Systems in the Age of AI

Data engineering has expanded well beyond traditional reporting to support diverse workloads, from real-time streaming to predictive models. Artificial intelligence is accelerating this shift: according to a Snowflake-sponsored study by MIT Technology Review Insights, data engineers spent 37% of their time on AI projects in 2025 — nearly twice the share in 2023 — with executives projecting continued growth.

AI is impacting both sides of the engineering workflow. On the consumption side, AI demands richer semantic context, governed access and meticulous data preparation, while core priorities like pipeline reliability and consistent metrics remain essential. On the build side, AI-powered tooling is automating the manual effort required to develop and maintain those pipelines.

Data engineering responsibilities extend beyond individual pipelines to the architecture, data products and operational practices that allow organizations to use information reliably across different workloads.

Data engineers work across the data lifecycle, designing ingestion processes, developing transformations and data models, coordinating pipeline execution and maintaining the systems that deliver information to its consumers. Their responsibilities focus on data quality, freshness, reliability and governance, particularly when multiple applications depend on the same underlying information.

Historically, much of the discipline has centered on preparing reliable data for analytical consumption. Business intelligence applications need consistent metrics, data scientists need reproducible data sets, and operational applications depend on information delivered within defined latency and availability requirements.

Those same systems must now support AI applications that retrieve enterprise knowledge, generate queries and make decisions or take actions using business data. Alongside the underlying records, AI applications require sufficient context to interpret them correctly, including business definitions, relationships, metadata and access policies.

Without verified trust in the underlying data, organizations can’t safely automate operations or deploy AI applications that act on enterprise information. The emphasis on trustworthy data is reflected in dbt Labs’ 2026 State of Analytics Engineering report, where 83% of respondents identified increasing trust in data and data teams as a priority, up from 66% the previous year.

See how Snowflake supports the data engineering lifecycle, from ingestion and transformation to governed AI applications:

The data engineering lifecycle encompasses the processes that move information from source systems to the applications that depend on it. Depending on the workload, records might pass through ingestion services, storage layers and several transformations before they’re ready for consumption. Along the way, the system needs to preserve enough information to maintain consistent results as new records arrive and existing ones change.

Collecting and ingesting source data

Enterprise data comes from a variety of sources, including transactional systems such as CRM and ERP platforms, as well as application logs, event streams and external APIs. These sources generate different kinds of records and expose changes through different interfaces, which influences how information enters the processing environment.

For relational sources using change data capture (CDC), data ingestion often begins with an initial snapshot followed by a stream of subsequent inserts, updates and deletes. Maintaining a consistent downstream table requires coordinating the two and tracking which source changes have already been applied.

Other sources impose different constraints. An event stream can deliver messages out of order, for example, while an API connector often has to retrieve records through paginated requests subject to rate limits. When a pipeline combines these sources, processing logic must account for differences in delivery order and update frequency.

What the source exposes also determines which historical states can be reconstructed. If an application provides only its latest records, downstream processing can’t recover intermediate changes that were never captured.

For AI workloads, source information may also include documents and support conversations alongside structured records. Ingesting that content involves preserving relevant metadata as well as extracting text. A document’s source, effective date and access restrictions can determine whether an AI application should retrieve or use it.

Storing data for processing

Once ingested, records are typically written to a storage layer where they’re available for subsequent processing. Many implementations retain source data before applying business transformations, allowing engineers to inspect incoming records and, where sufficient history is preserved, reprocess them when requirements change.

The storage architecture determines how those records are organized and which processing capabilities are available. Data warehouses provide structured tables and analytical processing, while data lakes commonly retain files in object storage, including information that hasn’t yet been modeled. Lakehouse architectures add transactional table management and analytical processing over lake storage.

Organizations may use more than one approach. Raw events might remain in object storage while standardized and curated records are maintained in warehouse tables. Retaining the original information can be particularly useful when an upstream correction or a revised business rule requires historical reprocessing.

Where multiple processing engines need access to shared data, open table formats such as Apache Iceberg provide table metadata and snapshot management over files. Compatible engines can operate on the same underlying tables, reducing the need to maintain separate copies for every workload. Sharing the storage layer requires compatible catalog integration so the engines reference the intended tables and snapshots, along with access controls that govern which data each engine and its users can access.

How data is retained dictates how far back teams can reconstruct past states. If an aggregate calculation or business rule changes, an immutable event log allows teams to recalculate historical totals without re-extracting data from source systems. This design underpins modern extract, load and transform (ELT) architectures, where data is loaded into scalable cloud storage prior to transformation.

Transforming and modeling data

With the records available for processing, transformations can reconcile information from different sources and apply the rules required by downstream consumers. For a customer revenue table, for example, the workflow may need to associate account identifiers across systems, standardize transaction records and calculate revenue according to an approved business definition. Completed sales, settled payments and recognized revenue can represent different measures even when they originate from the same transaction data.

Historical corrections affect how that logic is implemented. Refunds may change totals for periods that have already been processed, for example, so an incremental transformation needs a way to identify which existing aggregates require revision. Processing only transactions newer than the previous run could leave those historical results incorrect.

Data modeling establishes how the processed information is represented for consumers. A reporting model might organize transactions into fact tables connected to customer and product dimensions, preserving historical customer attributes where necessary. That historical representation determines which questions the data can answer. The transformation logic must preserve and apply the appropriate definition.

For applications generating their own queries, business meaning also needs to be accessible beyond the physical tables. Semantic definitions can identify approved measures, dimensions and relationships so an AI application will use the intended concepts rather than inferring meaning from column names alone.

Other AI workloads may require preparing unstructured information for retrieval. Support documents, for example, may be extracted and indexed with account or product identifiers, source references and access metadata. Those relationships allow an application to combine retrieved text with curated transaction records while maintaining the relevant context.

Coordinating pipeline execution

As transformations accumulate, their dependencies determine how updates propagate through the system. For example, a customer churn scoring table may depend on aggregated behavioral features, which in turn depend on raw application telemetry and account reference data. Downstream processing must account for those dependencies so scoring models execute only after both activity streams and profile updates have fully resolved.

In procedural workflows, orchestration systems often coordinate ingestion and transformation jobs using schedules, dependency rules and execution state. Downstream tasks start when the configured conditions are satisfied, while independent transformations can run concurrently.

Declarative approaches move more of that coordination into the processing platform. Rather than scheduling every transformation separately, an engineer defines the desired output and, where supported, a freshness requirement. The platform manages refresh execution and dependencies according to its processing model.

Whether procedural or declarative, the workflow needs a consistent way to determine when results are ready for consumption. Some implementations publish dependent outputs together after processing succeeds; others allow tables to refresh independently and expose their respective freshness. The appropriate behavior depends on what downstream applications expect.

Publishing and serving data products

After processing, the resulting information is made available through the interfaces used by its consumers. Analytical applications may query curated tables directly, while operational services may use APIs or specialized serving layers with different latency requirements.

For AI applications, the published information may also need semantic definitions or retrieval interfaces that provide business context. An agent investigating customer activity, for example, could query curated transactions using an approved revenue metric, retrieve relevant support history and associate both results with the correct account.

The resulting tables and interfaces can be managed as a data product, with defined business meaning, quality expectations, freshness requirements and access controls. These specifications establish what downstream applications can expect from the information they consume, independently of how the underlying pipelines are implemented.

Before a pipeline enters production, engineers need to verify that its transformations produce the expected results and that changes won’t disrupt downstream consumers. Version-controlled definitions and automated testing support this process. With tools such as dbt, engineers can maintain SQL models, dependencies and tests together, using CI/CD to validate changes before deployment.

Testing also needs to account for how pipelines process existing data. For example, a revenue calculation that produces correct results during a full rebuild might behave differently when processing only new or modified transactions. For this reason, engineers test business logic alongside data quality requirements such as uniqueness, referential integrity and consistency with source totals.

Recovering from processing failures

Once a pipeline is running, interruptions are inevitable. To resume without losing records or processing the same changes twice, the system needs to know what was successfully completed before the interruption. Orchestration systems track completed executions, while streaming systems use checkpoints or offsets to record processing progress. That information establishes where processing should resume, although the pipeline also needs safeguards against duplicating changes when previously processed records are delivered again.

Recovery also has to account for work already completed downstream. If standardized transactions were updated before an interruption but the revenue aggregation wasn’t refreshed, the workflow needs to identify which results are current and resume the remaining processing. Publication rules govern what consumers can see until the dependent outputs reach the required state.

Detecting problems in the output

In a 2026 survey conducted by Monte Carlo and CDO Magazine, 61% of senior data and AI leaders reported experiencing data incidents where monitored metrics appeared normal despite an underlying critical issue. Of those surveyed, 47% said this happened sometimes and 14% said it happened often.

Execution monitoring shows which tasks ran, how long they took and whether they failed, but a successful run doesn’t necessarily mean the resulting data meets consumer requirements. An event-aggregation pipeline, for example, could continue completing every 15 minutes even after an upstream collector stops delivering user session telemetry. Although the execution history would show no failures, a freshness check comparing source timestamps against downstream records would reveal that active usage metrics are freezing in place.

Engineers use lineage to trace the affected output through its upstream tables and transformations. Combined with logs, repository history and quality results, that information helps establish where the problem originated and which downstream assets need correction.

Other problems can originate in upstream changes. Renaming a column might break a downstream query, while changing the meaning of an existing field can produce incorrect results without triggering an error. Data contracts establish expectations for incoming data, while semantic definitions preserve agreed business meanings, providing a basis for identifying disruptive changes during development and review. These practices are part of DataOps and data reliability engineering, producing the version-controlled code, execution records and validation results that engineers and AI agents use to investigate and repair pipelines.

As organizations develop more data products, maintaining the pipelines that support them creates additional engineering work. Beyond transformation logic, teams have to manage processing infrastructure, dependencies, scheduling and integrations between services. Modern data engineering architectures reduce that overhead in several ways:

• Integrated platforms and managed infrastructure: Combining ingestion, processing and data governance capabilities reduces the number of separate services and integrations engineers maintain. Cloud-managed infrastructure also handles selected provisioning, scaling and maintenance operations.
• Declarative processing: With declarative processing, engineers define the desired output and its freshness requirements, while the platform coordinates supported refresh operations and dependencies. This reduces the need for custom scheduling and incremental-processing logic.
• Open table formats: Apache Iceberg allows compatible engines to work with shared tables, reducing the synchronization pipelines required to maintain separate copies of the same data.

With less infrastructure and execution logic to maintain, engineers can organize more of their work around reusable data products. When an AI agent needs customer loyalty information, for example, the engineering team can build on existing account activity and churn-risk products, adding the required support history or governed interfaces rather than implementing another pipeline independently.

Under a data mesh approach, domain teams take responsibility for these products while shared platform teams provide common infrastructure and governance standards. Defined product requirements also establish a basis for AI agents to investigate, modify and validate the underlying engineering workflows.

Coding assistants already support SQL development, testing and documentation. Given a schema and transformation description, they can generate code that an engineer reviews and incorporates into the existing workflow.

But this is just the starting point. Snowflake’s Chris Child, VP of Product, Data Engineering, anticipates a broader shift: “Over time, I think the role of the data engineer will be less about writing code for every single pipeline and more about managing the infrastructure that AI agents are operating in.” This shift would place greater responsibility on engineers for architecture, business requirements and the controls governing automated work.

An agentic data engineering system can carry out a longer sequence of work to investigate a failed pipeline, including retrieving execution logs, examining repository files, checking dependencies, modifying code and running validation tools before proposing a repair. After identifying a likely cause, the agent can prepare a code change, run relevant tests and open a pull request for review. If the change affects business meaning or historical processing, it can flag the task for human review.

Pipeline repair with AI agents: an internal case study

This approach is being applied to production data engineering at Snowflake. To reduce the workload associated with Snowflake’s growing data environment, the team developed an AI agent that investigates monitored pipeline failures and prepares proposed repairs. Using operational logs and repository context, the agent identifies likely causes, modifies code and validates changes before opening a pull request for engineering review.

The team configured workflows with defined instructions, repository access, execution limits and notification destinations. Running agents as separate jobs allows multiple investigations to proceed concurrently, rather than routing every request through a single server.

In October 2026, Snowflake Engineering reported that its agent went from incident detection to a review-ready pull request in under 16 minutes on average. Over five months, the team reported hundreds of merged pull requests and improvements to more than 500 pipelines, including changes that reduced unnecessary compute consumption.

This implementation demonstrates how agents can take on substantial investigation and repair work while engineers retain approval over production changes. It also illustrates an intermediate step toward a broader vision where agents independently execute more of the work involved in maintaining data pipelines.

Toward autonomous data engineering

The path toward autonomous data engineering involves progressively delegating more engineering work to AI, building on established development practices and expanding automation as teams gain confidence in its reliability. A five-stage maturity model describes this progression:

• Stage 1 — Foundation: Engineers build and maintain pipelines themselves, establishing practices such as version control, automated testing and reliable deployment processes.
• Stage 2 — Copilots: AI assists with individual development tasks, including writing SQL and generating tests, while engineers make the decisions and carry out the work.
• Stage 3 — Human-in-the-loop agents: Agents investigate problems and propose changes, but engineers review and approve actions before they’re executed.
• Stage 4 — Human-on-the-loop agents: Agents carry out selected tasks independently within predefined boundaries. Engineers monitor their activity and intervene when necessary.
• Stage 5 — Autonomous data engineering: The envisioned operating model involves agents independently detecting failures, diagnosing causes, applying repairs and validating outcomes. Engineers define the architecture, policies and business requirements governing those operations.

As agents move from proposing changes to executing them independently, teams need to establish where that authority begins and ends. Tasks with established procedures and results that can be reliably validated are typically well-suited to autonomous execution, while changes involving unclear business requirements or unexpected conditions require human judgment.

Determining which tasks can be delegated also means establishing what constitutes a successful outcome. Engineers define the conditions an agent must satisfy before proceeding, the changes that require approval and when it must stop and request guidance.

As Chris Child explains, “Governance, interoperability and unified data are the preconditions that make it safe to trust autonomous workflows.” Those foundations support a gradual expansion of agent responsibility, with the scope of automation tied to the engineering context, validation capabilities and controls available for each task. The maturity model provides a direction for that progression without assuming every workload should be moved to fully autonomous operation.

Snowflake brings ingestion, processing, governance and AI development capabilities together in a managed platform, reducing the need to maintain separate infrastructure for every stage of the data lifecycle.

For ingestion, Snowflake Openflow provides managed connectivity, while Snowpipe and Snowpipe Streaming support file-based and streaming delivery. Engineers can build transformations using SQL, Apache Spark, Python or dbt Projects on Snowflake. Dynamic Tables provide a declarative option, allowing engineers to define transformations and freshness targets while Snowflake manages supported refresh operations and dependencies.

For workloads requiring interoperable access, Snowflake supports Apache Iceberg tables. Snowflake Horizon Catalog provides discovery, lineage and governance capabilities, while Horizon Context defines business metrics and relationships for applications using structured data. Cortex Analyst uses those definitions to generate SQL from natural-language questions, and Cortex Search supports retrieval from unstructured information.

AI is also changing how engineers work within the platform. Snowflake CoCo provides AI-assisted development using Snowflake context and supported tools. The internal pipeline-repair agent described earlier illustrates a more advanced pattern using CoCo Automations and CoCo Cloud Agents, in which agents investigate failures and prepare validated changes for engineering review. Together, these capabilities support the progression toward more autonomous data engineering workflows.

As data engineering platforms assume more responsibility for processing and AI agents begin handling development and maintenance tasks, the role of the data engineer is evolving. Engineers bring the technical and business understanding needed to design data architectures, define reliable data products and make decisions about how information is processed, modeled and delivered. Snowflake’s vision for autonomous data engineering is built on the expertise of engineers who design and shape the systems and data products on which the business depends.