Use version-controlled SwellStaticFiles

[mranst]

Description

I set up a local repository for SwellStaticFiles using git-annex. This works by storing files in an annex location and having git only manage symlinks to those files, updating the links whenever changes are made. This makes for a more lightweight process. We can keep this as a local repository for now, but we can eventually add remotes such as s3 or other backup locations. I propose to pin versions of those files to worktrees in the static files location (for example, I called the preliminary versions v0.0.0). We can make changes in the develop version, then create a tag and new worktree when we want to switch versions. These files are identical to the ones Swell has been using up until now so there's zero diff, I did add a basic readme briefly explaining git-annex in the repository.

[mathomp4]

I'm going to mention @tclune here. Only because this is a good reminder to look at git-annex. @mranst says it's working for him, so maybe some of the weird NFS issues we had in the past are now better?

[mranst]

Closes #767

[Dooruk]

I'm going to mention @tclune here. Only because this is a good reminder to look at git-annex. @mranst says it's working for him, so maybe some of the weird NFS issues we had in the past are now better?

What were the issues you guys had @tclune, so we are aware?

Hmm this is neat, it is interesting how it works. The software itself seems to be uptodate, so that is encouraging. I have a few very basic questions.

• Who is controlling the git-annex installation on NCCS? Is the one you are using an outdated version?

• We need to keep the develop files write protected.

• "For 'release' of new versions, a new worktree should be created under this location". Sorry, it's not clear to me how would one test with and update the static files. Say one wants to add new files or edit files. They would first need to create a new Swell git branch, create a local copy of annex files (all symlinks), and then update the files they want on the actual static file location, which is write protected ideally?

If you think it might be easier to demonstrate then type, please let me know.

[tclune]

The only issues that we had were that the requirements never converged to that solution. E.g., we wanted to use it for versioning of input data, but software council liked having sibling directories for the different versions. (Also had to convince Arlindo not to invent a home-grown solution on multiple occasions!)

There were also some weirdness issues with the fact that annex uses hardlinks when on the same filesystem and symlinks when on different file systems. I forget the details, but we did find some ways to work around the technical issues. (@mathomp4 - do you remember this one?)

Basically annex has remained a solution that is almost ready to deploy if/when we (GMAO/IMVI) want to pull the trigger. I have long forgotten the details of how to install/use/deploy git-annex. Forgotten for the 3rd time now probably. :-) my vague recollection is that it was easy to install on my laptop. Not sure why/if we requested NCCS to install at this remove. (@mathomp4?)

[mathomp4]

The current git-annex/10.20240532 is in NCCS space, so they must have installed it. I suppose on ... May 32, 2024? Weird. Maybe it isn't a date version.

The latest version I see in brew is 10.20260525, but I think git-annex is like vim: release often; I see a news item for 10.20260601.

As for our testing, @tclune did figure out a clever way to do things via symlinks that seemed to avoid the NFS issues git-annex has (or had at the time of testing). Maybe things are better now?

[mranst]

* Who is controlling the `git-annex` installation on NCCS? Is the one you are using an outdated version?

We can look at installing our own version

* "For 'release' of new versions, a new worktree should be created under this location". Sorry, it's not clear to me how would one test with and update the static files. Say one wants to add new files or edit files. They would first need to create a new Swell git branch, create a local copy of annex files (all symlinks), and then update the files they want on the actual static file location, which is write protected ideally?

You can clone the repository as long as you are on the local filesystem, make changes to files and commit them to branches. My idea here was to have separate versions as worktrees in this location, but devs are free to clone the repository and test out files in their own workspace.

My understanding is that local git repositories don't have any notion of access controls beyond the os level. So anyone with access to advda, (or whatever group it is under) is going to have to have write access to this location (or at least the .git folder), otherwise they wouldn't be able to do anything. So this sort of relies on the assumption that anyone who needs to edit these files has advda access and is going to be responsible about it (which is how it is currently). My goal with this was to provide some basic version tracking, unfortunately I don't think there's going to be a super formalized process for review without invoking a hosting service like Github

[Dooruk]

Sounds and looks good, I would like to see how this works in practice. Ideally files here shouldn't change often so we don't need to worry about version control as much as like the SWELL source code.

I'm looking into this now, git or git annex command gives me the following error:

git-annex: Git refuses to operate in this repository,
probably because it is owned by someone else.

To add an exception for this directory, call:
        git config --global --add safe.directory /gpfsm/dnb10/projects/p61/SwellStaticFiles/versions/develop

This is fine for develop branch (only a few should be able to write there). For a new branch (say v20260611) do I need to create a different folder?

[mranst]

Sounds and looks good, I would like to see how this works in practice. Ideally files here shouldn't change often so we don't need to worry about version control as much as like the SWELL source code.

I'm looking into this now, git or git annex command gives me the following error:

git-annex: Git refuses to operate in this repository,
probably because it is owned by someone else.

To add an exception for this directory, call:
        git config --global --add safe.directory /gpfsm/dnb10/projects/p61/SwellStaticFiles/versions/develop

This is fine for develop branch (only a few should be able to write there). For a new branch (say v20260611) do I need to create a different folder?

That message is more of a security warning for your behalf, not the repository's (to make sure that you're not unintentionally running any malicious code through git hooks). All you need to do is run that command and you can modify the repo, if you have filesystem permissions. I was thinking that we'd use worktrees in different directories to maintain branches, though it might get confusing in practice