BioCypher overarching CLI (working title biotope)

[slobentanzer]

Here we discuss CLI-related things like orchestration and data streams

[slobentanzer]

@jdreo @mbaric758 as discussed, here are two high-level diagrams for the options we have. The CLI would be able to be in command of multiple steps, probably best to manage that using a config and not a huge amount of CLI parameters, but I know how programmers think. ;)

Core at end of pipeline

flowchart LR

    %% User-Facing Schema Layer
    SC["Schema Configuration<br>(Controls Adapters<br>→ Biocypher Mapping)"]

    %% Main Pipeline
    A["User CLI Invocation"]
    B["Adapter 1<br>(Generates Node/Edge Stream)"]
    B2["Adapter 2<br>(Generates Node/Edge Stream)"]
    B3["... Other Adapters ..."]
    C["Named Pipe or<br>STDOUT Middleware<br>"]
    D["Fusion Step<br>(DuckDB or Map-Reduce)"]
    E["Biocypher Core<br>(Tuple Writing or<br>other format)"]

    %% Connections
    A --> B
    A --> B2
    A --> B3

    SC -.-> B
    SC -.-> B2
    SC -.-> B3
    SC --> E

    B -->|Stream| C
    B2 -->|Stream| C
    B3 -->|Stream| C

    C --> D
    D --> E

Loading

Core after adapters

flowchart LR

    SC["Schema Configuration<br>(Controls Adapters to Biocypher)"]
    A["User CLI Invocation"]
    B["Adapter 1<br>(Generates Node/Edge Data)"]
    B2["Adapter 2<br>(Generates Node/Edge Data)"]
    B3["... Other Adapters ..."]
    E["Biocypher Core<br>(Receives Adapter Data)"]
    O["Biocypher Output<br>Adapter Stream"]
    M["Named Pipe or<br>STDOUT Middleware"]
    F["Fusion Step<br>(DuckDB or Map-Reduce)"]

    %% Schema config influences adapters and Biocypher
    SC -.-> B
    SC -.-> B2
    SC -.-> B3
    SC --> E

    %% CLI triggers adapters
    A --> B
    A --> B2
    A --> B3

    %% Adapters feed data into Biocypher
    B --> E
    B2 --> E
    B3 --> E

    %% Biocypher outputs a stream for fusion
    E --> O
    O --> M
    M --> F

Loading

These may be a bit too high-level though, lacking some of the procedural details. Please modify to add your own view, particularly with respect to ontoweaver and the fusion aspects.

[slobentanzer]

edited for readability

[jdreo]

CLI arguments VS config files

I don't know about programmers in general, but a good programmer will certainly thinks that there is no difference between CLI arguments and configuration files. In ontoweave, we actually managed that problem with the classical POSIX/XDG approach (using argparse, I don't know if click allows to do that easily), in which the config is loaded as a sequence:

1. hard-coded values
2. system/distribution config files
3. user config files
4. environment variables
5. CLI arguments

So, in essence, there is absolutely no opposition between arguments and config files, both are really just different ways to load the same configuration.

Flowchart

Here is an attempt at an overview. I did not draw biotope, since it's just orchestrating everything.

flowchart LR

%% User-Facing Schema Layer
CS[("Schema")]
CY[("Ontology")]
CO[("Output")]
CM[("Mapping")]

%% Main Pipeline
AR["Raw Adapter<br>(serialize on stream)"]
AO["OntoWeaver Adapter<br>(in-mem or stream)"]
AB["BioCypher Adapter<br>(in-mem or stream)"]
M["Middleware<br>(e.g. FIFO+socat)"]
T["BioCypher generic output"]
F["Fusion Step<br>(Map-Reduce, over DuckDB)"]
O["BioCypher output<br>(any supported format)"]

%% Connections

AR ==>|FIFO Named Pipe| M
AO ==>|FIFO Named Pipe| M
AO ==> T
AO ==> O
AB ==>|FIFO Named Pipe| M
AB ==> T
M ==> T
T ==> F
AB ==> O
F ==> O

%% Config files
CS -.-> T
CS -.-> AO
CS -.-> O

CY -.-> AO
CY -.-> T

CY -.-> O

CM -.-> AO

O -.-> CO

Loading

[slobentanzer]

Yes, thanks for the clarification; we also follow POSIX/XDG so far. I meant as a default I would prefer the config, but of course the params should be available at every layer. Agree with the diagram, looks good to me for a prototype to go ahead with.

Re Click, if I believe the reviews, it makes these things even easier than using argparse; I am particularly interested in what folks mean by "well-integrated with pytest," as we make heavy use of that also in the benchmarks. Could be a way to even include the various benchmark executions in a biotope module. Will have a look at the Click docs soon, maybe do a draft with cursor as well.

Potential modules

biotope build: BioCypher and OntoWeaver wrapper (the above)
biotope chat: BioChatter wrapper
biotope extract: BioGather wrapper
biotope benchmark: BioChatter and BioGather benchmark execution CLI
biotope view (?): GUI applications

[slobentanzer]

I played around a bit and didn't have an issue with biotope as a name even after sleeping on it, so I renamed the repo to biotope. I also like Click, it is really easy to parameterise even a modular CLI with it, and the pytest integration really does make sense (you don't need subprocesses to test anything).

I also created a biotope init submodule that creates the initial project layout for the user (so far mainly nonsensical dummy content) and also creates a .biotope folder to track metadata along the lines of .git.

Feel free to check it out and also, if there's time/priority, contribute a module for the pipeline, particularly ontoweaver. I am thinking we could also have biotope build for what is called "BioCypher generic" in the diagram, and maybe biotope weave for the ontoweaver-derived functionalities? Maybe also biotope fuse?