View all newsletters
Receive our newsletter - data, insights and analysis delivered to you
  1. Technology
  2. Cloud
August 29, 2019

BigQuery’s Big New Back-End: Data Streaming Now 10 Times Faster, Says GCP

Default streaming API quota up from 100,000 to 1,000,000 rows per second per project

By CBR Staff Writer

Google Cloud Platform (GCP) says it has radically revamped BigQuery, the enterprise data warehouse managed service that is arguably the flagship product in GCP’s stack.

The move is intended to remove data streaming bottlenecks and boost performance/capacity significantly for streaming workloads like those from the IoT.

A rebuild of its back-end means it’s now 10-times faster, with the default Streaming API quota lifted from 100,000 to 1,000,000 rows per second per project. Maximum bytes per second are also up from 100MB per table to 1GB per project.

There are also now no table-level limitations.

(BigQuery has a SQL interface, can be accessed via the GCP Console, a web UI, using a command-line tool, or by making calls to the BigQuery REST API using client libraries such as Java, .NET, or Python. It is commonly used to ingest and analyse data; another recent upgrade lets users run TensorFlow machine learning models in it…)

Automatic re-clustering now available.png

There have been no changes to the streaming API as a result of the new back-end.

Content from our partners
Green for go: Transforming trade in the UK
Manufacturers are switching to personalised customer experience amid fierce competition
How many ends in end-to-end service orchestration?

BigQuery Capacity Boost

GCP has also expanded support for federated queries, or external data sources that you can query directly through BigQuery, to include its own Cloud SQL.

The company’s Evan Jones said in a blog that other upgrades include “automatic re-clustering”.

He explained: “Efficiency is essential when you’re crunching through huge datasets. One key best practice for cost and performance optimization in BigQuery is table partitioning and clustering.

“As new data is added to your partitioned tables, it may get written into an active partition and need to be periodically re-clustered for better performance. Traditionally, other data warehouse processes like “VACUUM” and “automatic clustering” require setup and financing by the user. BigQuery now automatically re-clusters your data for you at no additional cost and with no action needed on your part.”

(When customers create and use clustered tables in BigQuery, their charges are based on how much data is stored in the tables and on the queries they run against the data).

See also: Google Eases Access to Public Data Sets

Topics in this article : , , ,
Websites in our network
Select and enter your corporate email address Tech Monitor's research, insight and analysis examines the frontiers of digital transformation to help tech leaders navigate the future. Our Changelog newsletter delivers our best work to your inbox every week.
  • CIO
  • CTO
  • CISO
  • CSO
  • CFO
  • CDO
  • CEO
  • Architect Founder
  • MD
  • Director
  • Manager
  • Other
Visit our privacy policy for more information about our services, how New Statesman Media Group may use, process and share your personal data, including information on your rights in respect of your personal data and how you can unsubscribe from future marketing communications. Our services are intended for corporate subscribers and you warrant that the email address submitted is your corporate email address.