How Google Revolutionized Interactive SQL and Modern Analytics
The Untold Story of Dremel and Its Lasting Influence on Big Data and SQL at Scale
In 2010, Google published a paper that didn’t make front-page news, but if you’ve queried a terabyte-scale dataset in seconds lately, you’ve likely felt its grandiose impact.
The system was called Dremel, and it introduced a new model for interactive, ad hoc analysis at web scale: something that, until then, was more wishful thinking than reality.
That same year, Google launched BigQuery, built on top of Dremel’s architecture. Fast forward a decade, and BigQuery has become a core pillar of the modern data stack: a serverless, petabyte-scale, fully-managed data warehouse.
But what made Dremel truly revolutionary wasn’t just raw speed. It was how it reimagined the entire stack, from how data is stored to how queries are executed at scale, ushering in architectural ideas that have since become cloud-native best practices.
This post is (as always) a deep dive into those ideas, drawn from the retrospective paper “Dremel: A Decade of Interactive SQL Analysis at Web Scale”, which reflects on how Dremel’s design not only stood the test of time but quietly shaped the trajectory of modern analytics systems.
Here’s what Dremel pioneered:
Disaggregated compute and storage: Compute and storage were decoupled long before it was fashionable. Dremel scaled both independently, allowing elastic query execution over shared data.
In situ analysis: Instead of copying data into rigid warehouses, Dremel enabled querying data where it lived, alongside tools like MapReduce and Flume. Today, this is the foundation of data lakehouse architectures.
Serverless computing: Dremel was internal-only, but it introduced a serverless, pay-per-use model before “serverless” was a buzzword. This vision later evolved into BigQuery’s now-familiar pricing and provisioning model.
Columnar storage for semistructured data: Column stores weren’t new, but Dremel’s ability to efficiently encode and query nested, semistructured data was. It extended columnar storage into JSON-like territory, long before solutions like Parquet and ORC became popular.
SQL for everyone: Dremel’s SQL-like query language eventually became ANSI-compliant and shared across products like Spanner, reinforcing the now-universal trend: even for big data, SQL remains the lingua franca.
The Dremel paper wasn’t just a description of a system: it was a glimpse into how data infrastructure at Google operated at a level most in the industry hadn’t yet imagined.
Mike Franklin, in introducing the journal version of the paper, noted that what was once “eye-opening” is now common practice. And he was totally right.
This retrospective isn’t just about history. It’s about the ideas behind Dremel that made it enduring, and how those same principles power today’s cloud-native data platforms.
If you’ve ever run a BigQuery job, written a SQL query over a massive dataset, or wondered how modern systems deliver sub-second latency over petabytes, you’ve seen Dremel’s legacy in action.
Let’s unpack the architecture, design decisions, and long-term impact of one of the most quietly influential systems in modern computing.
The Return of SQL, From Rejection to Renaissance
In the early 2000s, Google helped ignite the Big Data revolution, not with SQL, but by walking away from it.
The company’s infrastructure was optimized for scale, built on swarms of cheap, failure-prone machines. Systems like GFS and MapReduce became the backbone of batch processing at Google.
Data was stored in massive chunks, processed in parallel, and often manipulated using custom tools like Sawzall, a scripting language designed to tame MapReduce’s C++ underbelly.
For transactional workloads, the answer was Bigtable, a NoSQL store that famously powered much of Google’s web infrastructure.
And SQL? That was considered a relic. Too rigid, too centralized. The internal consensus was blunt: “SQL doesn’t scale.”
But raw scalability came at a cost: iteration speed. Writing MapReduce jobs (even with Sawzall) was cumbersome. An analysis job could take hours (in a few cases, even some days) to write, debug, and run. Teams were solving the scalability problem, but sacrificing key properties like usability, velocity, and flexibility.
Then came Dremel.
Dremel flipped that trade-off on its head. It reintroduced SQL, not as an afterthought, but as the core interface for querying massive, nested datasets.
Suddenly, analysts could write a declarative SQL query and run it interactively over trillions of rows. Jobs that previously took hours could now be iterated in seconds.
This wasn't just a technical improvement: it was a paradigm shift. The ability to explore petabyte-scale data ad hoc, from dashboards or custom tools, was transformative. Teams could now ask new questions, faster—and more often.
But Dremel’s SQL wasn’t your textbook dialect. It was designed from the ground up to work with hierarchical, nested schemas, reflecting the real structure of Google’s data, often encoded in Protocol Buffers.
Unlike traditional SQL systems that normalized everything into flat tables and required expensive joins, Dremel embraced denormalization.
A single nested record could contain thousands of repeated fields, making it ideal for complex log data, without flattening or duplicating. This was one of Dremel’s key innovations: columnar querying over deeply nested, semistructured data.
For a while, Dremel didn’t even support joins, and it didn’t need to. Its model was fast enough, and expressive enough, for most of Google’s analytical needs.
F1 and the Rise of Distributed SQL
Around the same time, a parallel SQL renaissance was unfolding inside Google, but on the transactional side. The Ads team, frustrated with the limitations of sharded MySQL and complex handoffs to scalable backends like Bigtable or Mesa, launched F1: a new SQL engine that combined traditional relational features with Bigtable’s distributed foundation.
By 2013, Google Ads had fully migrated to F1, and other internal teams followed shortly thereafter.
The success of F1 led to Spanner, which extended these ideas into a globally-distributed, strongly-consistent SQL database. Today, Spanner backs most of Google’s transactional systems.
Suddenly, SQL was no longer “legacy.” It was the lingua franca again, spanning batch analytics (Dremel), transactional systems (F1, Spanner), and even hybrid systems that blurred the line between OLTP and OLAP.
One SQL to Rule Them All
But success came with fragmentation. Dremel, F1, Spanner, and other systems like PowerDrill, Procella, and Tenzing all developed their own SQL dialects. Each had its quirks.
Developers and analysts needed to learn multiple idiosyncratic flavors of SQL to work across these platforms.
To clean up this mess, Google launched GoogleSQL, a unification effort to standardize SQL across all its engines. The project produced:
A single SQL dialect, ANSI-compliant with key extensions for structured data.
A common parser and compiler front-end.
A shared library of SQL functions.
Compliance tests and tools for validation, formatting, and fuzzing.
This effort led to ZetaSQL, which is now open source and powers systems like BigQuery, Spanner, and Cloud Dataflow.
Google’s internal SQL finally became unified, and externally available.
The Broader SQL Revival
Google wasn’t alone in its journey. The broader open-source and enterprise worlds went through a similar cycle. First came MapReduce and HDFS, then a wave of NoSQL systems that ditched SQL in favor of scale. But as complexity mounted and iteration speed slowed, the pendulum swung back.
Projects like Hive, Spark SQL, and Presto brought declarative querying back to distributed systems. SQL returned as the default interface for Big Data. Dremel, it turns out, wasn’t just ahead of its time, it was right all along.
Unlocking Scalability Through Decoupling
In the early days of Dremel, everything was tightly coupled.
Originally conceived in 2006 by Andrey Gubarev as a "20% project", Dremel ran on a few hundred shared-nothing servers.
Each machine stored a slice of data on its own local disk, and the design philosophy was straightforward: keep things close to maximize performance. At the time, this was the dominant architectural pattern in analytics, squeeze performance from dedicated hardware, with data co-located on directly-attached disks.
It worked well, until it didn’t.
As Dremel’s adoption grew, so did its unsolvable pain points. Managing a dedicated fleet became increasingly difficult, especially as datasets ballooned to petabyte scale and workloads became more unpredictable.
Engineers were spending more and more time coordinating capacity than innovating.
Disaggregated Storage
The first major shift came in 2009, when Dremel migrated to Borg, Google’s cluster manager (a precursor to Kubernetes).
Suddenly, Dremel lived in a shared world. And with shared infrastructure came shared spindles, disks now juggled workloads from many services, leading to unpredictable performance.
To cope, Dremel adopted a replicated storage strategy: each data partition was stored on three different disks, managed independently. This replication improved reliability and allowed the system to scale far beyond its original boundaries. Trillion-row tables became commonplace.
But the underlying design remained tightly coupled: storage was still fundamentally local. Want to add more storage? You had to add compute. Want to scale query capacity? You had to move data.
Even minor software changes had to account for complex replication logic. Worst of all, the data was "locked in", accessible only through Dremel itself.
It became clear that solving these challenges piecemeal was leading Dremel into a familiar shape: a distributed filesystem, much like GFS. So, the team took the plunge.
The first experiments replacing local disk storage with GFS were pretty disappointing. Performance dropped by almost an order of magnitude. Queries stalled waiting for hundreds of thousands of tablet files to open.
The original Dremel metadata layout was designed for disk seeks, not network hops. Latency was the bottleneck.
But with perseverance (and a lot of tuning) the team turned it around. Improvements in storage format, smarter metadata handling, and aggressive prefetching flipped the script. Eventually, Dremel running on disaggregated storage outperformed the local-disk setup in both throughput and latency.
The benefits stacked up:
No more loading delays. Sharded tables no longer had to be copied onto local disks.
Greater elasticity. Dremel clusters could ingest new workloads without resizing.
Simplified operations. GFS, and later Colossus, offered robust guarantees and freed teams from managing low-level storage.
Data liberation. Once disaggregated, datasets could be shared and accessed outside of Dremel.
With the migration to Colossus, Google’s distributed, multi-master filesystem, the system gained yet another leap in scalability and robustness.
Disaggregated Memory
Storage wasn’t the only thing ripe for disaggregation. After Dremel’s original paper was published, support for distributed joins was introduced via a shuffle primitive, inspired by MapReduce.
The initial implementation used local RAM and disk to buffer sorted intermediate results. It actually worked, but at scale, tight coupling became a bottleneck again.
Two major issues emerged:
Quadratic scaling. As the number of data producers and consumers grew, co-located shuffle couldn’t handle the communication explosion efficiently.
Resource fragmentation. Storage tied to compute led to poor isolation and stranded resources, problems that only worsened under multi-tenant workloads.
The team decided to disaggregate shuffle, too.
In 2012, a new shuffle layer was built on top of Colossus, separating intermediate shuffle storage from compute. Several designs were tested, including dedicated shuffle services, but by 2014, the team landed on a solution that supported fully in-memory query execution.
The impact was dramatic:
Latency dropped by an order of magnitude.
Shuffles scaled an order of magnitude larger.
Overall service cost fell by more than 20%.
Today, this disaggregated in-memory shuffle layer underpins both BigQuery and Google Cloud Dataflow, handling massive intermediate state across thousands of workers with tight coordination and low latency.
In BigQuery’s case, this memory system now accounts for 80% of the service’s total memory footprint.
Some Lessons from Disaggregation
Disaggregation wasn’t just a performance win: it changed the game for how data systems are built. Several key themes emerged:
Elasticity and cost-efficiency. Decoupling storage, memory, and compute made it possible to scale each layer independently, improving utilization and enabling more efficient resource provisioning.
Industry-wide adoption. This trend extends beyond Dremel. Systems like Spanner, Aurora, Snowflake, and Azure SQL Hyperscale all embrace variants of disaggregated storage and compute. Even RAM, hard to disaggregate in the traditional sense, is often packaged into specialized, value-added services like Dremel’s in-memory shuffle.
Richer APIs. Modern disaggregated systems are no longer built on raw block I/O. APIs now embed encryption, access control, filtering, and even aggregation. Some of these capabilities are pushed all the way into hardware, like Oracle’s SPARC M7.
Services, not components. The line between “resource” and “service” is blurring. A disaggregated shuffle buffer isn’t just a memory pool: it’s a distributed, resilient, query-aware data service.
Disaggregation, in short, helped Dremel (and later BigQuery) escape the bottlenecks of tightly coupled design. It unlocked scale, simplified operations, and laid the foundation for the cloud-native data architectures we see today.
The Rise of the Query Fabric
In the early days, Dremel was designed as a monolithic SQL engine with a vertically integrated execution path.
Query parsing, planning, execution, and result serving happened within the bounds of a relatively fixed set of services. As Dremel scaled up and diversified its data sources, this architectural rigidity began to fray.
Supporting federated sources, multiple file formats, and massive memory-disaggregated operations stretched the boundaries of what a tightly-coupled engine could handle. Something had to give.
So it did. What began as a tightly-scoped engine slowly unfolded into something more modular, more distributed: a query fabric.
The term “query fabric” wasn’t part of the original design vocabulary, but it captures the essence of what Dremel became. Query processing was no longer a function owned by a single engine.
Instead, it evolved into a choreography of distributed components, each specialized, loosely coupled, and stitched together by a common runtime and protocol.
Parsing might happen in one layer, optimization in another, and execution might span dozens of systems across disaggregated memory and storage.
This evolution paralleled (and in many ways prefigured) the emergence of modern data mesh and lakehouse architectures. It recognized a simple truth: in large organizations, data is everywhere, and no one system will ever “own” all of it.
Runtime Modularity and Logical Query Plans
One of the foundational shifts in Dremel’s architecture was the abstraction of query logic away from physical execution.
Instead of generating physical plans tightly bound to local compute and storage, Dremel began producing portable, logical plans that could be interpreted and executed across heterogeneous runtimes.
This meant a single query could simultaneously:
Read Parquet files from Google Cloud Storage,
Join with transactional tables served by F1,
Apply filters using BigTable’s row-key indexing,
Shuffle intermediate results into disaggregated memory,
And aggregate everything using Dremel’s vectorized execution engine.
This was only possible because of a shared understanding of logical plans across engines: we’re talking about plans that were declarative, extensible, and, crucially, serializable.
The abstraction barrier opened new doors: now Dremel could act as both an execution engine and a query router, dispatching subqueries to where they could be most efficiently run.
Shared Infrastructure, Differentiated Experience
The move toward a modular query fabric also encouraged Dremel’s engineers to rethink the boundaries between infrastructure and user experience.
For example, multiple teams inside Google shared much of the same underlying infrastructure for resource scheduling, file format handling, and query planning. But each provided a different user experience: some prioritized latency, others scale, others concurrency.
Rather than fight this diversity, Dremel embraced it. Components were refactored to be reusable. Storage engines could be swapped in a practical way. Planners could be layered efficiently. This created an internal ecosystem where innovation could happen in parallel, and improvements in one system often cascaded into others.
Today, this architectural philosophy powers not just BigQuery, but an entire family of analytics tools at Google, each tailored for different personas but harmonized under the same infrastructural umbrella.
A Fabric with Opinions
Despite its modularity, the Dremel query fabric was never completely neutral. It carried embedded opinions about how analytics should work: columnar data, SQL as the sole main language, scalability through disaggregation, and cost-efficiency via federation.
These opinions helped guide its evolution, acting as a kind of architectural compass. They also made integration with certain external systems easier (e.g., object stores and columnar formats), and others harder (e.g., write-heavy transactional systems or semi-structured data with evolving schemas).
As Dremel pushed forward, these opinions were gradually reified into APIs, configuration surfaces, and service contracts, making the fabric not only scalable, but programmable.
From Engine to Ecosystem
What began as a columnar query engine designed to power internal analytics at Google has grown into a foundational component of a much larger data ecosystem.
Dremel’s evolution reflects a broader shift in the way we think about analytics in the cloud era: not as a pipeline of rigidly staged operations, but as a dynamic interplay of storage formats, compute engines, federated sources, and so called “shared infrastructure”.
In situ analysis liberated data from the constraints of traditional warehousing, allowing users to query it where it lives. Disaggregated execution scaled compute and memory independently, paving the way for performance at planetary scale.
The emergence of a query fabric turned a monolithic engine into a modular, programmable system capable of spanning formats, sources, and runtimes. Throughout this journey, Dremel served not only as a tool, but as a conceptual blueprint for modern cloud-native analytics platforms.
Yet, this evolution also highlighted enduring tensions: between performance and generality, control and convenience, flexibility and governance.
The future of analytics is not purely in situ, nor purely managed: it is hybrid. It is about giving users the power to choose the right trade-off for each job, while ensuring that the system can reason, adapt, and optimize across a messy, heterogeneous landscape.
In that sense, Dremel’s story is not just about technical innovation. It is a journey about architectural humility.
It’s the recognition that no single engine will rule them all, but a well-designed fabric can weave them together into something greater than the sum of their parts.