Planning: zebra-scan MVP

[arya2]

Motivation

• Create a prototype minimum viable product for zebra-scan
• Iterate on the code reasonably quickly
• Explore the scope of planned and unplanned zebra-scan features
• Inform the design of zebra-scan

Assumptions

These assumptions are from the MVP scope google doc.

Let’s assume the following:

• No external client (i.e. no RPCs)
• User has the necessary technical skills to query our database/datastore/however-we-store-the-scanning-results directly
• User controls their whole machine (so file permissions and secret data deletion don’t matter)
• We want to store the scanning results so that they can be made available to the User at a time of their choosing

Proposed assumptions:

• There is only one user/account/security domain

[arya2]

Could we make the MVP of zebra-scan its own binary, run separately from zebrad with a read-only handle for getting blocks from zebra-state? Maybe from a method hidden behind a feature so it's still excluded from zebrad?

And then as a separate process, could it print the results and nothing else directly to stdout? (so zebra-scan > results.txt would stream the results to a file)

[teor2345]

There are 4 different but related design questions here, so I'm going to respond to them in order:

Could we make the MVP of zebra-scan its own binary, , run separately from zebrad...

We could, but I think that's more complicated than adding an async task to zebrad behind the existing shielded-scan feature.

Adding a new binary involves a whole lot of decisions around how that binary should work, including loading an entire config document, launching an async executor, and maybe command-line argument processing, and a bunch of other stuff that comes with running a binary (like logging).

Adding a task to zebrad involves loading one new config field, and copying some existing async task code.

(And then there's the common scanner code no matter which alternative we choose.)

with a read-only handle for getting blocks from zebra-state?

This might be more complicated than you expect, because RocksDB has different APIs for concurrent access from different processes.

So the user either has to shut down their zebrad instance so the scanner can open the database using zebra-state, or we need to add new database code to open a read-only secondary instance:
https://docs.rs/rocksdb/latest/rocksdb/struct.DBCommon.html#method.open_as_secondary

Zebra's ReadStateService is actually holding a read-write database instance underneath, which is required for it to get correct results from recently written blocks.

Accessing the non-finalized state from another process could also be complicated.

And then as a separate process, could it print the results and nothing else directly to stdout? (so zebra-scan > results.txt would stream the results to a file)

I think writing results to a flat file might be simpler. But a separate binary or stdout aren't actually required to make that happen?

Writing to a flat file and writing to standard output are both simple in Rust: println!() or File::open() then writeln!(). However, the serialization formats are more complex, because they need to be ASCII, and they can't include certain characters. Also, processing a large number of results from a flat file can be complicated, because some spreadsheet programs can't open them, and lookups can be very slow. It also gets complicated to write results to a file from concurrent code.

Initially, it's more complicated to write results to a new column family in an existing database (or a new column family using the same database code, but a different database directory). But after that setup, things are simpler. We can use the existing binary serialization formats. Lookups are fast. Concurrent writing just works.

And either way we have to define data structures and serialization code.

We could time-box the database changes, and fall back to a flat file if they take too long?
I'll start a different discussion about minimal database changes.

[arya2]

We could, but I think that's more complicated than adding an async task to zebrad behind the existing shielded-scan feature.

Ah, okay. It would be fine to put it in zebrad and to stream the output to a file as well, we already have logic for atomically updating a file with new entries.

I do think it would be easier to read the viewing keys from the zebrad config.

Accessing the non-finalized state from another process could also be complicated.

Yeah, I meant to simplify only scanning the finalized state, but it would be throw-away code.

I think writing results to a flat file might be simpler. But a separate binary or stdout aren't actually required to make that happen?

I agree 👍

Initially, it's more complicated to write results to a new column family in an existing database (or a new column family using the same database code, but a different database directory). But after that setup, things are simpler. We can use the existing binary serialization formats. Lookups are fast. Concurrent writing just works.

For the MVP, I'm happy with whatever's easiest for the implementor between those.

They're likely the same effort, so using a database would be better because we can continue using it later. (Perhaps in a separate database?).

We could time-box the database changes, and fall back to a flat file if they take too long?

That sounds good, both options seem to require the same effort. (or if using a database is preferable, I think we've reduced scope enough to comfortably afford committing it without much risk to the timeline).

[teor2345]

We could, but I think that's more complicated than adding an async task to zebrad behind the existing shielded-scan feature.

Ah, okay. It would be fine to put it in zebrad and to stream the output to a file as well, we already have logic for atomically updating a file with new entries.

Atomically, but not concurrently!

So we don't get half-written files, but it's still possible to overwrite file + result A with file + result B, when what's needed is file + result A + result B. This doesn't matter for the peer cache because we only read it once at startup, then we write the entire file from memory every time.

But there's no guarantee that the results will fit in memory. (And that [edit: keeping all the results in memory] kind of misses the point.)

[mpguerra]

let's concentrate on the design for writing results to persistent storage as that's the only item for which we don't already have issues created in the MVP Epic

[teor2345]

That discussion is here:
#7975 (comment)

[teor2345]

Create a prototype of zebra-scan as quickly as possible.

A prototype is different to a minimum viable product, so I'm not sure if this is a goal of an MVP?

Instead, I think the goal is already stated in the MVP document, but it's buried halfway down:

do whatever is simplest and useful

Or the Wikipedia definition:

A minimum viable product (MVP) is a version of a product with just enough features to be usable by early customers who can then provide feedback for future product development.

A focus on releasing an MVP means that developers potentially avoid lengthy and (possibly) unnecessary work. Instead, they iterate on working versions and respond to feedback, challenging and validating assumptions about a product's requirements.

[arya2]

A prototype is different to a minimum viable product, so I'm not sure if this is a goal of an MVP?

Ah, I had a prototype in mind.

Thinking of an MVP, optionally logging the single user's Sapling balance or displaying it in a progress bar may be nice and easy.

But probably requires storing the unspent nullifiers.

[teor2345]

Shielded balances require decrypting the value of unspent notes, which needs a full viewing key, and then storing and summing those values.

We could track the number of scanned blocks, and the number of discovered transactions using progress bars?
It would provide a helpful user experience. But I'm not sure if it's minimal.

It might be the simplest way to display scanning progress.

[arya2]

We could track the number of scanned blocks, and the number of discovered transactions using progress bars?

Actually storing the results in a different database and letting clients on the same machine access that database directly does seem like an MVP, I don't think there's any need to log balances or discovered transaction counts.

So Zebra can still avoid storing more than viewing keys and a list of discovered transaction hashes for an account.

[teor2345]

I don't think we'll get direct database access in an MVP, see my detailed reasons here: #7975 (reply in thread)

But technically sophisticated users will be able to write zebrad code to access the database, or dump its results in the format they want.

This is in line with our MVP assumptions:

• No external client (i.e. no RPCs)
• User has the necessary technical skills to query our database/datastore/however-we-store-the-scanning-results directly

[teor2345]

Let's talk about how to store results generated by the MVP.

The current suggestions are:

• write to standard output
• write to a flat file
• use mmap-sync
• write to one file for each client
• write to a new column family in the existing state database
• write to a new column family in a new database at a different path

Please edit this comment to add new suggestions to the list.

[teor2345]

For standard output, the user will need to configure Zebra to redirect logs to another file. This config already exists, but we might need to silence some intro messages. Or they can use zebrad start error.

For a flat file we'll need a config to define the file path, but no special logging config is required.

However, flat file serialization formats are more complex, because they need to be ASCII, and they can't include certain characters. Also, processing a large number of results from a flat file can be complicated, because some spreadsheet programs can't open them, and lookups can be very slow. It also gets complicated to write results to a file from concurrent code.

[teor2345]

For a new database, we can re-use the existing state database code, and pass it a different state path and column family names.

Re-using the existing database adds a new column family name, and doesn't need to change the state path. But the results are harder to delete, because they are mixed with publicly available cached data. This is a security issue.

Either way, we can use the existing binary serialization formats. Database lookups are fast. Concurrent writing just works.

Here's a rough sketch of the minimum necessary code changes:

• Add db_path, db_version, and column_family_list arguments to ZebraDb::new()
• For example, db_path is state for zebra-state, and could be private-scan for zebra-scan. The other arguments are existing constants.
• Export ZebraDb from zebra-state
• Create a new ZebraDb in zebra-scan
• Create a new column family containing:

key	value
SaplingScanningKey + transaction::Hash	empty

Or we can use TransactionLocation, it doesn't really matter, they both have existing serialization implementations. But the hash is easier to construct.

SaplingScanningKey is currently a string, so we can just serialize it using as_bytes(). We can use a binary serialization later when we sort out the key types.

Then we can just increment the database version whenever we change the format. (Deleting previous versions can be left for later, we have the code, but it needs path changes.)

Edit: It's easier to use ZebraDb here instead of DiskDb, so we can do format upgrades later on.

[arya2]

write to a new column family in a new database at a different path

I like this option, clients on the same machine could directly access a read-only instance of the database without requesting anything from Zebra, so could a web server that handles requests from remote clients (the scan task is coupled with the server to add/remove viewers).

[teor2345]

write to a new column family in a new database at a different path

I like this option, clients on the same machine could directly access a read-only instance of the database without requesting anything from Zebra, so could a web server that handles requests from remote clients (the scan task is coupled with the server to add/remove viewers).

They could, but it will take some work to get secondary instances working, and any client would likely need information from zebra-state, such as block heights, transaction locations, or transaction data. So going through zebrad might be necessary.

[teor2345]

write to a new column family in a new database at a different path

I like this option, clients on the same machine could directly access a read-only instance of the database without requesting anything from Zebra, so could a web server that handles requests from remote clients (the scan task is coupled with the server to add/remove viewers).

They could, but it will take some work to get secondary instances working, and any client would likely need information from zebra-state, such as block heights, transaction locations, or transaction data. So going through zebrad might be necessary.

[mpguerra]

MVP is done, closing this discussion