Where's the data?: Clarify using Database Connectors with Notebook Kit

[zachbogart]

I'm a little confused about Database Connectors and where the data source is located when using Notebook Kit. I'd appreciate some clarification about possible workflows using this new setup.

I was reproducing the taxi example and that works well locally:

• I can download the .parquet file and then put it in the docs/ folder in my case. Then a sql cell can then see it and I'm off to the races.

Now I want to try to recreate the deployment result, like on the connectors page (do an analysis and share). I got confused in the section on Notebook Kit not having the drivers installed and further along. Let's say I want to share my lovely taxi results by building them into a site that deploys using GitHub (npm build into dist). Here's what I'm running into locally:

Starting simple

• Installed duckdb driver
• Works well in Desktop, but run into trouble when I try to run npm run dev and view as a page. Is there something else I need to add/setup?

On Desktop 👇

Running preview with Notebook Kit 👇

Going further
For deploying, how do I set it up so I'm not git tracking the giant .parquet file? (that's where my mind is going)

• It makes sense, but I don't see the database file within the notebook-kit repo. So are you storing the file locally, running the notebook, then deploying the results? Is GitHub Actions doing the query for you?
• This might be related to caching? I notice in the notebook-kit that the data seems to be cached, though it was recommended to gitignore .observable/ entirely.
• How should we save results for deploying a notebook? (essentially reproducing the database connectors result)

Appreciate your help, thanks!

[mbostock]

Are you running an old version of Notebook Kit? What version of Notebook Kit are you running? (The error suggests it’s looking for db which would only happen prior to the release of database connectors in 1.1.0.)

[zachbogart]

Gosh darn it, you are absolutely right! Worked like a charm once running the right version. :)

[mbostock]

As to your other questions…

How do I set it up so I'm not git tracking the giant .parquet file?

You can add a file to your .gitignore file if you want to avoid adding it to your repository, as we do here:

notebook-kit/.gitignore

Line 4 in 77dff68

docs/fhvhv_tripdata_2025-06.parquet

(You could also put the large file into the .observable/cache file where it can live alongside the cached query results, and just be careful not to accidentally delete it.)

I don't see the database file within the notebook-kit repo. So are you storing the file locally, running the notebook, then deploying the results? Is GitHub Actions doing the query for you? This might be related to caching? I notice in the notebook-kit that the data seems to be cached, though it was recommended to gitignore .observable/ entirely.

Yes, I have the Parquet file on my local computer, and I have committed the cached query results to source control. You can bypass what is listed in your .gitignore by using git add -f. I ran the following command to add the cached query results to git:

git add -f docs/.observable/cache/duckdb-3dax5o7ca20m13tp.json docs/.observable/cache/duckdb-3i571kxkj7d5iqkm.json docs/.observable/cache/duckdb-3te2snbouq6be1hp.json docs/.observable/cache/duckdb-4nmx131pqy4guuau.json

The reason that I did this is to avoid needing to either store the large Parquet file in git, or to download it every time we build the site in CI. Either of those are certainly possible, but it would make building the documentation slower and less reliable, and since these results never change (unless I re-run the queries), I wanted to keep things simple.

How should we save results for deploying a notebook? (essentially reproducing the database connectors result)

For point-in-time (stable) analysis, especially if you are not collaborating on analysis, it’s reasonable to commit the saved query results to git; that is what I have done for the documentation site. This is reasonable even if you ignore the .observable/cache directory by default. On the other hand, if you want to collaborate on analysis or if you have live data, then you probably want to either commit your source data to git (perhaps using Git LFS), or fetch it during deploy, and then compute the query results during deploy so that they are kept up-to-date.

[zachbogart]

Excellent explanation. So cached results or live data can be tracked, with considerations about size and access frequency.

Wondering when caching happens. If I change the query I get a new json file. Then change back, no new file. Is it checking if existing query/data output already has been cached?

[mbostock]

Right, if you edit the cell back to something you’ve previously cached, then it restores the old results — which is great for quickly reverting or comparing versions of queries. Query caches are also shared across cells (and even notebooks in the same folder), so you can even paste the old query into a new cell and see the results side-by-side.

In Observable Desktop, when you re-run a “clean” SQL cell (that is, without editing the query, and either via shift-enter or by clicking the Play button or by clicking the cache age in the cell toolbar), it uses the since option to the underlying DatabaseClient to bypass the stale cached results and compute new results. In Observable Notebook Kit, you can delete the corresponding file from .observable/cache (or delete the entire cache) to get fresh results.

[zachbogart]

Cool, thanks!

[zachbogart]

Success deploying example, thanks for your help!

• https://tk.cookwcode.com/hello-data-connectors

🚕