When writing data to Colossus, BigQuery makes some decision about initial sharding strategy which evolves based on the query and access patterns. It provides client-driven replication and encoding.
Colossus handles cluster-wide replication, recovery and distributed management. Colossus is Google’s latest generation distributed file system and successor to GFS (Google File Systems). During encoding various statistics about the data is collected which is later used for query planning.īigQuery leverages Capacitor to store data in Colossus. Once all column data is encoded, it’s written back to Colossus. During the import process, BigQuery encodes every column separately into Capacitor format. You can import your data into BigQuery storage via Batch loads or Streaming. Unlike ColumnIO, Capacitor enabled BigQuery to directly operate on compressed data, without decompressing the data on the fly. In 2016, Capacitor replaced ColumnIO - the previous generation optimized columnar storage format. column is stored in a separate Capacitor file which enables BigQuery to achieve very high compression ratio and scan throughput. As you may expect, each field of BigQuery table i.e. BigQuery stores data in a columnar format known as Capacitor. The most expensive part of any Big Data analytics platform is almost always disk I/O. This makes BigQuery more economical and scalable compared to its counterparts. It is important to note, BigQuery architecture separates the concepts of storage (Colossus) and compute (Borg) and allows them to scale independently - a key requirement for an elastic data warehouse. Dremel implements a multi-level serving tree to execute queries which are covered in more detail in following sections.įigure-1: A high-level architecture for BigQuery service. Dremel jobs read data from Google’s Colossus file systems using Jupiter network, perform various SQL operations and return results to the client. Borg - Google’s large-scale cluster management system - allocates the compute capacity for the Dremel jobs.
As illustrated below, a BigQuery client (typically BigQuery Web UI or bg command-line tool or REST APIs) interact with Dremel engine via a client interface. In fact, BigQuery service leverages Google’s innovative technologies like Borg, Colossus, Capacitor, and Jupiter. Dremel is just an execution engine for the BigQuery. By incorporating columnar storage and tree architecture of Dremel, BigQuery offers unprecedented performance. 10,000 foot viewīigQuery and Dremel share the same underlying architecture. Original Dremel papers were published in 2010 and at the time of publication Google was running multiple instances of Dremel ranging from tens to thousands of nodes. Dremel is Google’s interactive ad-hoc query system for analysis of read-only nested data. High-level architectureīigQuery is built on top of Dremel technology which has been in production internally in Google since 2006. For instance, for best query performance, it is highly beneficial to understand how BigQuery allocates resources and relationship between the number of slots and query performance. Having said that, a good understanding of BigQuery architecture is useful when implementing various BigQuery best-practices including controlling costs, optimizing query performance, and optimizing storage. To get started with BigQuery, your must be able to import your data into BigQuery, then be able to write your queries using SQL dialects offered by BigQuery. That’s the whole idea of BigQuery - you don’t need to worry about architecture and operation. Overall, you don’t need to know much about underlying BigQuery architecture or how this service operates under the hood. BigQuery exposes simple client interface which enables users to run interactive queries. The pricing model is quite simple - for every 1 TB of data processed you pay $5. BigQuery service manages underlying software as well as infrastructure including scalability and high-availability. There are no servers to manage or database software to install. In addition, BigQuery now integrates with a variety of Google Cloud Platform (GCP) services and third-party tools which makes it more useful.īigQuery is serverless, or more precisely data warehouse as a service. Since inception, BigQuery has evolved into a more economical and fully-managed data warehouse which can run blazing fast interactive and ad-hoc queries on datasets of petabyte-scale. BigQuery was first launched as a service in 2010 with general availability in November 2011. Google’s BigQuery is an enterprise-grade cloud-native data warehouse. A Deep Dive Into Google BigQuery Architecture
0 kommentar(er)
