Google Cloud Makes Apache Spark 4.2 Runtime Generally Available in Managed Service
Google Cloud has made Managed Service for Apache Spark 4.0 generally available, with a new runtime built on Apache Spark 4.2, Java 25 and Python 3.13. The release also adds an AI-assisted console and a faster serverless path.
TL;DR
- Managed Service for Apache Spark 4.0 is now generally available, built on Apache Spark 4.2 with Java 25 and Python 3.13.
- The runtime adds an AI agent for diagnosing failed jobs, IDE integration through the Data Agent Kit, and the Lightning Engine accelerator.
- Serverless workloads gain fallback across VM families and zones, and quotas are now set in Data Compute Units (DCUs).
A new runtime for serverless and cluster workloads
Google Cloud's Managed Service for Apache Spark 4.0 is now generally available. The release centers on a new runtime that packages Apache Spark 4.2, including Spark Declarative Pipelines and Real-Time Mode. It also ships Scala 2.13, Java 25, Python 3.13 with Pixi for dependency management, Apache Iceberg 1.12 and Delta Lake 4.4.
The same stack now serves both serverless and managed cluster deployments. Teams can move PySpark, Scala and Spark SQL workloads between the two without changing component versions, which removes a common source of drift between environments.
Google positions the release for what it calls agentic workloads, where AI agents submit and troubleshoot pipelines around the clock. The company says customer use of serverless Spark has nearly doubled over the past year, with millions of jobs running each month.
Console and troubleshooting changes
The redesigned console focuses on the job lifecycle. A new Overview page offers a samples gallery, Lakehouse public datasets and one-click authoring in VS Code through Cloud Shell. Batch jobs, interactive serverless sessions and managed clusters now appear in one Jobs listing.
Platform admins can also define reusable runtime profiles, which are in preview. A profile captures hardware, network and autoscaling settings once, so developers can pick a profile and run code.
The most notable addition is agentic troubleshooting. When a job fails or slows down, an AI agent reviews logs, metrics and query plans, then suggests a likely root cause and a fix. The agent works within existing IAM permissions. For teams that do not want to rely on automated diagnosis, the logs and metrics remain available in the same view.
Editor integration and performance
The Data Agent Kit connects Apache Spark to VS Code, Claude Code and Antigravity. During setup, a developer selects Apache Spark and other services, and the kit configures Model Context Protocol (MCP) servers, agent skills and API access. MCP is an open protocol that lets AI tools call external services through a standard interface.
On performance, Google describes three changes. Serverless 4.0 workloads start faster on the Premium tier at no additional cost. New serverless workloads run on the Premium tier by default, using Gen4 virtual machines and Hyperdisk storage. Lightning Engine, a native C++ vectorized engine built on Apache Gluten and Velox, is included in the 4.0 runtime.
Google says Lightning Engine delivers up to 3.25x the price-performance of the leading high-speed Apache Spark alternative, and up to 4.9x faster performance than open-source Apache Spark with zero code changes. These comparisons come from the vendor, and the sources do not name the alternative or describe the benchmark. For Apache Iceberg tables, built-in caching speeds up file scans by up to 50%. For BigQuery reads, Lightning Engine's Arrow-based reader is up to 40% faster than the default path.
Capacity and quotas
Serverless capacity is now more flexible. Serverless 4.0 workloads are no longer tied to a single VM instance family, disk type or zone. If the preferred infrastructure is constrained during a local demand peak, the service picks the best available option in the same region instead of failing the job. Once capacity returns, workloads move back to the default Gen4 and Hyperdisk setup.
Quotas for serverless deployments are now set in DCUs rather than per machine family. Admins manage one unified DCU quota instead of separate CPU quotas for each machine type. Attaching workloads to the Lakehouse runtime catalog is also automatic, replacing the legacy Hive Metastore.
Why it matters
For data engineers, the main change is a newer and more uniform runtime. Pinning Spark 4.2, Java 25 and Python 3.13 across both serverless and cluster jobs simplifies testing and upgrades. Teams that still rely on the Hive Metastore will need to plan the move to the Lakehouse runtime catalog, since attachment is now automatic.
Platform teams gain more control through runtime profiles and DCU quotas, but they should check how capacity fallback to other VM families and zones fits their cost and compliance requirements. Developers who work in VS Code or Claude Code can run Spark jobs without leaving their editor, provided they accept AI-generated suggestions and check them before deploying.
Sources
This article was written by Blozhik from the reporting below. Read the originals for full detail.
GoWhat’s new in Managed Service for Apache Spark 4.0Google Cloud Blog