Handle "just dependencies" better

[brettviren]

This may be an issue for Spack itself and not so much wire-cell-spack.

It is desirable to use a Spack view to both aggregate spack-installed packages providing the dependencies for developing a package as well as a location for developers to install that package using its native build system.

Specifically we want such a Spack view for developing WCT.

Ideally we may form this view so that it excludes the spack-installed version of WCT. This is because the Spack-installed files may be found during the native build of WCT and lead to some version mismatch problems..

The following options exist to have our cake and eat it too but all are imperfect.

Install, view and purge

spack install wire-cell-toolkit
spack view add -i local wire-cell-toolkit
rm local/include/WireCell* local/lib/libWireCell*

This suffers from taking time to install WCT only to remove it and needing to manually remove correct files.

Install and view of just dependencies

spack install wire-cell-dependencies
spack view add -i local wire-cell-dependencies

A Spack package wire-cell-dependencies was made to install everything but WCT. It consists of a copy-paste of most of wire-cell-toolkit Spack package.py and so is a source of drift over time (update one copy and not the other). In principle, we could make w-c-t depend on w-c-d to avoid this source of drift but then we'd need to version the dependencies in w-c-d as well as the dependency on w-c-d in w-c-t which feels confusing.

Install, view but exclude

spack install wire-cell-toolkit
spack view -e wire-cell-toolkit add -i local wire-cell-toolkit

This makes use of a feature to build a view on a seed but exclude that seed. It still suffers in that the spack install takes a bit longer than needed to install the ignored WCT.

There is spack install --only dependencies wire-cell-toolkit to avoid that but then the spack view will not work as it requires the seed to have been instantiated even when ignred.

[ligerlac]

Hey @brettviren
Regarding your second proposal: we could avoid the drift between the versions in w-c-d and w-c-t with a github actions workflow that automatically keeps them in sync.

Is there a reason why we suggest developers to use spack view rather than spack develop? I could, e.g., envision a developer workflow as follows:

spack repo add wct-spack
spack install wire-cell-dependencies
spack env create my-env
spack env activate my-env
spack develop -p /path/to/wire-cell-tookit --no-clone wire-cell-toolkit@master
spack add wire-cell-toolkit
spack install

[brettviren]

Hi @ligerlac

Regarding your second proposal: we could avoid the drift between the versions in w-c-d and w-c-t with a github actions workflow that automatically keeps them in sync.

That sounds good to me.

Is there a reason why we suggest developers to use spack view rather than spack develop?

I don't know if they are good reasons but:

• I did not know about spack develop until just now.

• Given that ignorance, I think spack view best fits how I personally like to develop against a Spack base. But this is very much by-construction. (I added the view command).

I have tried to use Spack "environments" for development environments in the distant past. It has been too long to remember what exact issues I hit with spack env but I remember them not adding anything useful to me on top of "views" for development. From your example workflow, it seems spack develop is another layer so (again, in my ignorance) I'm skeptical it is better (for me!).

However, I think it is very possible that develop may be more optimal than lower level view. The reason is that currently to create a new development area I use the rather "policy laden" script:

https://github.com/WireCell/wire-cell-spack/blob/master/scripts/wct-make-dev

If develop could take on all or some of this "policy" then it could be a good improvement over by-hand assembly of the development area. Though, I still have vague concern about adding too many "layers" to the development area and dependency on environment.

In any case, I think there is plenty of room for "more than one way to do it". So, feel free to update the docs. The main README has gotten a bit long. Probably everything starting from "WCT view" and any additions for env/develop should be factored into new files, say under docs/.

BTW, looking at your workflow inspired me to make #11.

[brettviren]

Since I did not fully understand what bothered me when I wrote it, let me expand on the "feels confusing" of the second option (Install and view of just dependencies). Maybe it can be overcome.

If we have no w-c-d then w-c-t fills with lines like:

# inline w-c-t/package.py
depends_on("tbb@2019", when='@0.19:)
depends_on("tbb@2020", when='@0.2:)

Note, we do not have lines like this now but we really should. TBB had a breaking change a few years back and WCT had to adapt. The above versions are just examples but I believe that older WCT will not work with newer TBB and vice versa.

Over time and in general, such versioned dependencies grow independently for each package.

If we move these lines into w-c-d we must have w-c-t depend on w-c-d by its version

# w-c-d/package.py
depends_on("tbb@2019", when='@1)
depends_on("tbb@2020", when='@2)

# delegated w-c-t/package.py
depends_on("w-c-d@1", when='@0.19:)
depends_on("w-c-d@2", when='@0.2:)

The versioning of w-c-d effectively flattens all version changes of all dependencies into a single labeling. That's fine if we make changes linear with calendar time. We make w-c-d v1, v2, v3, ....

If we discover a old bug and need a different dependency from a previous "dependency release" we must reflect a branching in the w-c-d version string, eg by "inserting" w-c-d v1.1. And, then add corresponding new depends_on("w-c-d@1.1",when="...") line to w-c-t.

Typing that out downgrades my "feels confusing" to merely "requires some care".

So, @ligerlac if you want to have a go at this, please do so!