Skip to content

Vodafone Czech Republic Cuts BigQuery Pipeline Costs 30% With Data Engineering Agent

Vodafone Czech Republic used Google's Data Engineering Agent to convert legacy BigQuery stored procedures into Dataform pipelines. The company reports halved execution times and a 30% cost reduction.

2 min readBased on 1 source0 views
CLOUD

TL;DR

  • Vodafone Czech Republic migrated legacy BigQuery stored procedures into modular Dataform pipelines using Google's Data Engineering Agent.
  • The company reports pipeline execution times cut by half and a 30% reduction in costs, with a 90% cut in migration time.
  • Parallelizing the code raised costs until the team asked the agent to materialize intermediate tables.

The problem with legacy stored procedures

Vodafone Czech Republic runs many BigQuery stored procedures to transform data for business use. They date from an earlier migration from an on-premises data warehouse. Many contain 2,000 or 3,000 lines of code and run one after another.

As the Google Cloud Blog describes it, such monoliths are hard to maintain. Logic is often copied and pasted, testing is manual and rare, comments tend to go out of date, and scheduling and dependencies are hard-coded. Dependencies between the stored procedure jobs were tracked by hand in a spreadsheet.

Migrating with the Data Engineering Agent

The Data Engineering Agent is a Gemini-based tool available in BigQuery pipelines and Dataform. It takes natural language instructions, plus any code supplied in an instruction file, and builds a Dataform pipeline. Vodafone used it directly in Dataform.

The agent split the code into smaller steps and ran independent ones in parallel. It also identified dependencies between tables. Dataform now handles execution order and logs, and engineers can run only part of a pipeline, which helps with debugging.

The cost trade-off

Parallelization was not free. Vodafone found that splitting logic into parallel streams could cancel out the caching that BigQuery applies to a base table in a sequential stored procedure. Each thread could then trigger a full table scan, and costs went up.

The fix was to have the agent create materialized views for intermediate tables. These views pre-cache the data so parallel queries stay fast and cheap. Vodafone also gave the agent specific guidance on which staging tables to parallelize. Vodafone credits the caching change with a 30% reduction in costs.

Reported results and next steps

Vodafone reports that the agent reproduced all stored procedures with 1:1 logic parity and minimal manual intervention. It cut migration time by 90% compared with manual conversion, which the company says would previously have taken weeks. Post-migration benchmarks show the new pipelines running twice as fast.

These figures come from Vodafone and Google Cloud, and the source does not include independent measurements or the underlying workload sizes. Vodafone plans to expand the agent to its Romanian, Irish, German and UK markets, create a central Dataform template repository, and migrate SAS code, BTEQ scripts and MicroStrategy workloads.

Why it matters

Teams running large legacy SQL estates in BigQuery can see a concrete path from monolithic stored procedures to modular, tested and documented Dataform pipelines. The Vodafone case also shows that parallelism can increase BigQuery spend if caching is lost, so cost checks should be part of any migration. Engineers may spend less time on repetitive conversion work and more on validating design choices, which Vodafone describes as a shift in how data engineers work.

Sources

This article was written by Blozhik from the reporting below. Read the originals for full detail.

GoHow Vodafone is using the Data Engineering Agent to improve their data pipelinesGoogle Cloud Blog
Was this useful?

Keep reading

Discussion · 0

Sign in to join the discussion. Liking does not need an account.Sign inRegister
Loading comments…
blozhikDeveloper news from across the web, explained. Every story links to its original sources.