Can data loaders reference other cells?

[martinswan]

Is it possible for data loader cells to reference other cells? For example, if I have a Python cell of:

from datetime import date

print(date.today(), end = "")

and name the output today with a format of text, can I reference that today value in a second Python cell?

[mbostock]

Not currently. Data loaders are self-contained programs and cannot reference the output of other cells; outputs can only be consumed in non-data loader cells. You can read and write to the file system, or a database, however.

It’s possible (even likely) we’ll figure out some way of doing this in the future, but it hasn’t been designed yet. Input welcome!

[mbostock]

To elaborate a bit…

In Node.js cells, we could easily parse the JavaScript to find unbound references — we do this already in JavaScript cells (and Markdown, HTML, SQL, etc.). But the semantics are different in that we need to serialize referenced values to make them available to Node.js from another process (another cell). We could implicitly use JSON to serialize the values, but that’s quite lossy: you can’t serialize functions, and even dates are coerced to strings. And yes we implicitly serialize interpolated values in SQL cells, and that generally feels okay; but it feels more surprising when going from JavaScript to Node.js, or vice versa, or Node.js to Node.js. So we thought that data loader cells should be explicit about serialization, and write to standard output instead of exposing top-level variables.

Even with explicit serialization, we could offer a mechanism to read the output of other data loader cells (and probably SQL cells… but probably not “live” cells that run in the browser). We’d need to design the syntax. For example, in a Node.js cell perhaps we could offer something like FileAttachment("foo") from observablehq:stdlib that allows you to read the output of another data loader as a file. Or perhaps it would be Output("foo") that deserializes the referenced value directly in Node.js, in the same way that it would be exposed in the browser. And there could be some similar API in Python.

Or instead of treating each data loader as a separate program, we could use long-lived processes with code injection as Jupyter does. That would make it easier to share variables within processes: Node.js cells could refer to top-level variables defined in other Node.js cells, likewise for Python, and likewise for JavaScript in the browser (and Markdown, HTML, etc. with JavaScript interpolation). That sounds useful, although I could see it being confusing how variables can be referenced across cells but only within the same process (the same cell type, mostly).

I’m hesitant to adopt the long-lived process model for Node.js and Python if it exacerbates the statefulness problem (in order to run a Node.js cell, you may need to run other Node.js cells that define referenced variables). I don’t want so much effort from the user to run cells manually to get back into the right state. Perhaps if cell execution were still automatic, it might not be so bad? Still, it’s also undesirable in a cloud-hosted environment because you need to keep your Node.js and Python processes running indefinitely, just in case the user wants to re-run a cell. And it would probably mean you couldn’t parallelize data loader execution… Complicated!

[martinswan]

Thanks for your reply. My mental model for asking this question was based on how Databricks notebooks work, but I am guessing that system is architected similarly to Jupyter notebooks and has the tradeoffs you mentioned. I appreciate that it sounds complicated!

[martinswan]

One other thought on this. Would it be the right trade-off to use stdin to pipe in values from other data loaders? Similar to how we can name our outputs, maybe a UI for selecting other cells' inputs could be added?

The python or node code could then reference stdin to parse out the data from the inputs. Or maybe that could be magically done for the user somehow?

[mbostock]

The reason that we use stdout for the output, rather than writing to a file, is mostly because data loader cells can only produce a single value; but if we allow data loaders cells to depend on other cells, we would want to allow references to multiple cells, so we would want to use (named) files rather than stdin. This is similar to how in most programming languages functions take multiple arguments but return a single value.

A practical consideration, though, is that it’s easier to detect when a process writes to stdout than to detect when it creates a file; the latter would probably require some sort of virtual file system. So for inputs, I’m not sure we’d be able to expose the outputs of other data loader cells as virtual files… I mean, maybe, if I can figure how? Short of that I was anticipating we’d provide some helper code in each supported interpreter (the Output("foo") I mentioned) which would read a file under the hood to access the serialized output of another data loader cell.