Google brings BigQuery SQL cells to open-source notebooks

Google has introduced the %%bqsql IPython cell magic for BigQuery DataFrames, extending a SQL-and-Python notebook workflow beyond Colab Enterprise into the wider open-source ecosystem.

The tool is intended to let data scientists and data engineers run SQL queries against local pandas DataFrames in notebook environments, then pass the results back into Python without manually moving data between in-memory objects and temporary tables.

The workflow supports open-source packages including Jupyter, pandas and BigFrames, as well as the BigQuery sandbox. In practice, users can set up a local Python environment or work in Colab, load notebook extensions, point BigFrames to a Google Cloud project ID, and begin querying local DataFrames with SQL syntax.

The approach centres on a notebook cell magic called %%bqsql. When users reference a local pandas DataFrame inside braces in a SQL cell, the query engine treats that object like a table. Query results can then be saved to a named BigFrames DataFrame for use in later Python or SQL cells.

This is meant to reduce a common point of friction in notebook-based analytics. Teams that prefer SQL for filtering, joins and aggregations often have to export or stage intermediate results before switching back to Python libraries for plotting, modelling or other analysis.

Hybrid workflow

Google illustrated the feature with an example using USDA wheat data stored in an Excel file. In that workflow, pandas downloads the file and reads a worksheet into a local DataFrame. The user then cleans column names, removes rows with missing values in Python, and uses SQL cells for further filtering and transformation.

One query narrows the dataset to yearly entries and stores the result in a BigFrames DataFrame called yearly. A second SQL step extracts year values with regular expressions, converts them into timestamps, and writes the output to another BigFrames DataFrame called timeseries.

From there, the user can return to Python for visualisation. Because BigFrames DataFrames implement the pandas API, charting methods can be called directly on the SQL-derived data structure, with only summarised chart data returned to the notebook rather than the full dataset.

Google presented the feature as a way to split notebook pipelines into smaller steps, rather than forcing users into a single large SQL statement or an all-Python transformation path. The same pattern can also be applied to larger datasets by replacing an initial local pandas DataFrame with a BigQuery DataFrame reference.

Open-source push

The launch also expands Google's effort to make BigQuery workflows more accessible in standard data science tools. SQL cells were first introduced in Colab Enterprise, and this move brings a similar interaction model to environments commonly used by developers working with open-source notebooks.

BigFrames sits at the centre of that strategy. The library is designed to give Python users a pandas-like interface while using the BigQuery engine underneath for data processing. The new cell magic adds a more direct SQL entry point.

Google also pointed to a separate BigQuery accessor on standard pandas DataFrames, which allows SQL scalar functions to run directly on local pandas data. That would let users call community user-defined functions from projects such as BigQuery Utils, BigFunctions and CARTO Analytics Toolbox for BigQuery from a pandas workflow.

The BigQuery sandbox can be used to test the notebook pattern without a credit card, although some advanced features remain restricted in that environment. A project ID is still required so query usage can be allocated correctly.

Notebook economics

The release reflects a broader industry effort to bridge the long-standing divide between SQL-first analytics teams and Python-first data science teams. In many organisations, notebook users alternate between database querying and local scripting, creating repeated handoffs between tools and data formats.

By embedding SQL directly in notebook cells and keeping outputs addressable in later code, Google is seeking to keep more of that work inside a single interactive environment. That may appeal to analysts who want SQL readability for transformations and Python libraries for visualisation and statistical work.

Some advanced functions become available once a billing account is connected to a Google Cloud project, including time-series forecasting through bigframes.bigquery.ai.forecast, which can be called from a SQL-and-Python workflow.

Google described the model as a chain that moves from a local pandas DataFrame into SQL filtering, back into BigFrames objects for further transformation, and then into Python visualisation or a downloaded pandas DataFrame.