Semaphore Repository Workspace Corruption During Parallel Task Execution

[HTGEK]

Hi Team,

We are observing repository workspace corruption when enabling “Allow Parallel Tasks” for a Semaphore task template executing Ansible + Python workflows from a GitHub repository.

Environment:

• Semaphore Community Edition
• GitHub HTTPS repository
• Parallel GitLab-triggered API requests (~20–30 concurrent)
• Semaphore task template configured with “Allow Parallel Tasks”
• Python scripts executed from cloned repository workspace

Current behavior:
When parallel execution is enabled, tasks intermittently fail during repository update/checkout phase before Ansible/Python execution starts.

Observed errors:

error: cannot open '.git/FETCH_HEAD': No such file or directory
fatal: Unable to read current working directory: No such file or directory
BUG: refs/files-backend.c:3175: initial ref transaction called with existing refs
Failed updating repository: signal: aborted (core dumped)

The issue appears related to concurrent mutation/update of the same repository workspace under:

/opt/semaphore/tmp/project_x/repository_x_template_xx

Questions:

1. What is the recommended Semaphore-supported approach for safe parallel execution against the same Git repository/template?
2. Is there a way to force isolated/fresh repository checkout per parallel task execution?
3. Is there a repository cache/workspace locking configuration recommended for high concurrency?
4. Are there known Git workspace corruption issues in Semaphore Community Edition under parallel task execution?
5. Is there a recommended runner/repository configuration for 20–30 simultaneous API-triggered task executions?

Important context:

• Sequential execution works fine
• Parallelism is a hard requirement for the workflow
• Existing shell-script-based workflow handled parallel requests successfully
• Failure occurs before Ansible/Python logic starts
• The issue appears isolated to Semaphore Git workspace/repository handling under concurrency

Any guidance/recommended production-grade configuration would be greatly appreciated.

Thanks.

[Lujain-ALghamdi]

This looks like a race condition around the shared repository workspace rather than anything in your Ansible/Python code.

The errors:

cannot open '.git/FETCH_HEAD'
Unable to read current working directory
initial ref transaction called with existing refs

are what I'd expect if multiple jobs are updating or replacing the same checkout at the same time.

Since sequential runs work fine and the failures happen before the playbook starts, I'd first try giving each parallel execution its own working copy instead of letting 20–30 jobs operate on the same Semaphore-managed repository directory.

For that level of concurrency, I'd probably keep a read-only/local mirror of the repo and create a temporary checkout/worktree for each job, for example:

WORKDIR=$(mktemp -d)
git clone /path/to/repo-mirror "$WORKDIR"
cd "$WORKDIR"

# run the job here

rm -rf "$WORKDIR"

Another option would be separate runners/instances if that's practical in your setup.

I wouldn't try to solve this with a Git lock around the shared workspace unless you're OK with effectively serializing the repository update step. That would avoid concurrent Git mutations, but it also defeats part of the reason for enabling parallel tasks.

So for 20–30 concurrent API-triggered runs, I'd aim for:

• one isolated checkout/workspace per execution
• no concurrent writes to the same .git directory
• cleanup after each job
• separate runners/workers if you need to scale the concurrency further

I'd also test the same workload with isolated clones outside Semaphore. If that runs reliably, it would be a pretty strong indication that the shared repository workspace is where the contention is happening.

I don't see a documented Semaphore setting that guarantees a fresh isolated repository checkout for every parallel execution, so I'd be interested to hear from the maintainers whether this is expected behavior with Allow Parallel Tasks or something that should be handled internally