Skip to content

Commit 17273e5

Browse files
Fix build error with --build-template conda/micromamba:v2 for singularity (#972)
Signed-off-by: munishchouhan <hrma017@gmail.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
1 parent 9703852 commit 17273e5

File tree

8 files changed

+47
-149
lines changed

8 files changed

+47
-149
lines changed

docs/api.md

Lines changed: 8 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -218,12 +218,12 @@ Returns the name of the container request made available by Wave.
218218
| `mambaImage` | Name of the Docker image used to build Conda containers. |
219219
| `commands` | Command to be included in the container. |
220220
| `basePackages` | Names of base packages. |
221-
| `baseImage` | Base image for the final stage of multi-stage Conda/Pixi builds. |
221+
| `baseImage` | Base image for the final stage of multi-stage Conda/Pixi Docker builds. Has no effect on Singularity builds. |
222222
| `pixiOpts` | Pixi build options (when `type` is `CONDA` and `buildTemplate` is `conda/pixi:v1`). |
223223
| `pixiImage` | Name of the Docker image used for Pixi package manager (e.g., `ghcr.io/prefix-dev/pixi:latest`). |
224224
| `cranOpts` | CRAN build options (when `type` is `CRAN`). |
225225
| `rImage` | Name of the R Docker image used to build CRAN containers (e.g., `rocker/r-ver:4.4.1`). |
226-
| `buildTemplate` | The build template to use for container builds. Supported values: `conda/pixi:v1` (Pixi with multi-stage builds), `conda/micromamba:v2` (Micromamba 2.x with multi-stage builds). Default: `conda/micromamba:v1` template. |
226+
| `buildTemplate` | The build template to use for container builds. Supported values: `conda/pixi:v1` (multi-stage build using Pixi), `conda/micromamba:v2` (multi-stage build using Micromamba 2.x). Both produce smaller images by excluding the package manager from the final stage (Singularity still uses a single-stage build). Default: `conda/micromamba:v1`. |
227227
| `nameStrategy` | The name strategy to be used to create the name of the container built by Wave. Options: `none`, `tagPrefix`, `imageSuffix`. | |
228228

229229
#### Response
@@ -493,7 +493,7 @@ Get status of build against `buildId` passed as path variable
493493
}
494494
```
495495

496-
- Create Singularity image with Pixi v1 template (multi-stage build):
496+
- Create Singularity image with Pixi v1 template (single-stage build):
497497

498498
**Request**
499499

@@ -535,7 +535,7 @@ Get status of build against `buildId` passed as path variable
535535
}
536536
```
537537

538-
- Create Singularity image with Micromamba v2 template (multi-stage build):
538+
- Create Singularity image with Micromamba v2 template (single-stage build):
539539

540540
**Request**
541541

@@ -578,19 +578,15 @@ Get status of build against `buildId` passed as path variable
578578
```
579579

580580
:::note
581-
Multi-stage build templates (`conda/pixi:v1` and `conda/micromamba:v2`) create optimized container images by separating the build environment from the final runtime environment. This results in smaller container images that only contain the installed packages and runtime dependencies, without the build tools.
581+
Multi-stage build templates (`conda/pixi:v1` and `conda/micromamba:v2`) create optimized Docker container images by separating the build environment from the final runtime environment. This results in smaller container images that only contain the installed packages and runtime dependencies, without the build tools.
582582
:::
583583

584584
:::important
585-
**Image Requirements for Singularity Builds with Pixi and Micromamba v2 Templates**
585+
**Singularity builds use single-stage**
586586

587-
When building Singularity images (`format: "sif"`) with the `conda/pixi:v1` or `conda/micromamba:v2` build templates, base images (via `baseImage` option) must have the `tar` utility installed.
587+
When building Singularity images (`format: "sif"`) with the `conda/pixi:v1` or `conda/micromamba:v2` build templates, single-stage builds are used. Singularity's proot-based builder cannot preserve file permissions when transferring files across stages, so the conda/pixi environment is installed directly in a single stage using the mamba/pixi image as the base.
588588
589-
This requirement exists because Singularity's multi-stage builds use `proot` to emulate filesystem operations, which cannot reliably copy directory structures between build stages. The templates support this by compressing the environment into a tarball (`tar czf`) and extracting it (`tar xzf`) in the final stage.
590-
591-
Docker builds are not affected by this requirement. Docker's native `COPY --from=build` directive handles directory copying directly.
592-
593-
Most standard base images (Ubuntu, Debian, Alpine, etc.) include `tar` by default, but minimal or distroless images may require explicit installation.
589+
As a result, the `baseImage` option has no effect on Singularity builds — it only applies to Docker builds. The multi-stage image size optimization is also not available for Singularity.
594590
:::
595591
596592
### GET `/v1alpha1/builds/{buildId}/status`

docs/cli/use-cases.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,8 @@ The Wave CLI supports build templates for creating container images from Conda p
9393
| Template | Description |
9494
|-----------------------|------------------------------------------------------------------------------------------------------------------|
9595
| `conda/micromamba:v1` | Single-stage build using Micromamba v1 (default). The final image includes the package manager. |
96-
| `conda/micromamba:v2` | Multi-stage build using Micromamba 2.x. Produces smaller images without the package manager in the final stage. |
97-
| `conda/pixi:v1` | Multi-stage build using [Pixi][pixi] package manager. Produces smaller images with faster dependency resolution. |
96+
| `conda/micromamba:v2` | Multi-stage build using Micromamba 2.x. Produces smaller images by excluding the package manager from the final stage (Singularity still uses a single-stage build). |
97+
| `conda/pixi:v1` | Multi-stage build using [Pixi][pixi] package manager. Produces smaller images with faster dependency resolution (Singularity still uses a single-stage build). |
9898

9999
**Related CLI arguments**
100100

@@ -110,11 +110,11 @@ Multi-stage build templates offer several advantages:
110110
- **Reproducibility**: Lock files are generated for exact package version tracking.
111111
- **Security**: Fewer binaries in the final image reduces the attack surface.
112112

113-
**Singularity build requirements**
113+
**Singularity builds use single-stage**
114114

115-
When building Singularity containers (`--singularity`) with the `conda/micromamba:v2` or `conda/pixi:v1` templates, base images must have the `tar` utility installed. This is required because Singularity's multi-stage builds use `proot`, which cannot copy directory structures directly between build stages. The templates work around this by compressing the environment with `tar` and extracting it in the final stage.
115+
When building Singularity containers (`--singularity`) with the `conda/micromamba:v2` or `conda/pixi:v1` templates, single-stage builds are used. Singularity's proot-based builder cannot preserve file permissions when transferring files across stages, so the conda/pixi environment is installed directly in a single stage using the mamba/pixi image as the base.
116116
117-
Docker builds are not affected by this requirement.
117+
As a result, the `baseImage` option has no effect on Singularity builds — it only applies to Docker builds. The multi-stage image size optimization is also not available for Singularity.
118118
119119
**Example usage**
120120
@@ -126,7 +126,7 @@ wave \
126126
--build-template conda/pixi:v1
127127
```
128128
129-
Build a container using the Micromamba v2 multi-stage template:
129+
Build a container using the Micromamba v2 template:
130130
131131
```bash
132132
wave \
Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
BootStrap: docker
22
From: {{mamba_image}}
3-
Stage: build
43
%files
54
{{wave_context_dir}}/conda.yml /scratch/conda.yml
65
%post
@@ -10,17 +9,6 @@ Stage: build
109
echo ">> CONDA_LOCK_START"
1110
cat environment.lock
1211
echo "<< CONDA_LOCK_END"
13-
tar czf /opt/conda.tar.gz -C /opt conda
14-
15-
Bootstrap: docker
16-
From: {{base_image}}
17-
Stage: final
18-
%files from build
19-
/opt/conda.tar.gz /opt/conda.tar.gz
20-
%post
21-
cd /opt
22-
tar xzf conda.tar.gz
23-
rm conda.tar.gz
2412
%environment
2513
export MAMBA_ROOT_PREFIX=/opt/conda
2614
export PATH="$MAMBA_ROOT_PREFIX/bin:$PATH"
Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,12 @@
11
BootStrap: docker
22
From: {{mamba_image}}
3-
Stage: build
43
%post
54
micromamba install -y -n base {{channel_opts}} {{target}}
65
{{base_packages}}
76
micromamba env export --name base --explicit > environment.lock
87
echo ">> CONDA_LOCK_START"
98
cat environment.lock
109
echo "<< CONDA_LOCK_END"
11-
12-
Bootstrap: docker
13-
From: {{base_image}}
14-
Stage: final
15-
%files from build
16-
/opt/conda /opt/conda
1710
%environment
1811
export MAMBA_ROOT_PREFIX=/opt/conda
1912
export PATH="$MAMBA_ROOT_PREFIX/bin:$PATH"
Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
BootStrap: docker
22
From: {{pixi_image}}
3-
Stage: build
43
%files
54
{{wave_context_dir}}/conda.yml /scratch/conda.yml
65
%post
@@ -12,19 +11,5 @@ Stage: build
1211
echo ">> CONDA_LOCK_START"
1312
cat /opt/wave/pixi.lock
1413
echo "<< CONDA_LOCK_END"
15-
tar czf /opt/pixi-env.tar.gz -C /opt/wave/.pixi/envs default
16-
ls -lh /opt/pixi-env.tar.gz
17-
18-
Bootstrap: docker
19-
From: {{base_image}}
20-
Stage: final
21-
# install binary from stage one
22-
%files from build
23-
/opt/pixi-env.tar.gz /opt/pixi-env.tar.gz
24-
/shell-hook.sh /shell-hook.sh
25-
%post
26-
mkdir -p /opt/wave/.pixi/envs
27-
tar xzf /opt/pixi-env.tar.gz -C /opt/wave/.pixi/envs
28-
rm /opt/pixi-env.tar.gz
2914
%environment
3015
export PATH="$MAMBA_ROOT_PREFIX/bin:$PATH"

src/test/groovy/io/seqera/wave/util/CondaHelperTest.groovy

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -168,10 +168,10 @@ class CondaHelperTest extends Specification {
168168
then:
169169
result.contains('BootStrap: docker')
170170
result.contains('From: mambaorg/micromamba:2-amazon2023')
171-
result.contains('Stage: build')
172171
result.contains('micromamba install -y -n base -f /scratch/conda.yml')
173-
result.contains('Bootstrap: docker')
174-
result.contains('From: ubuntu:24.04')
172+
result.contains('%files')
173+
result.contains('%post')
174+
result.contains('%environment')
175175
}
176176

177177
def 'should create v2 docker file with lock file'() {

src/test/groovy/io/seqera/wave/util/PixiHelperTest.groovy

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -63,12 +63,11 @@ class PixiHelperTest extends Specification {
6363
then:
6464
result.contains('BootStrap: docker')
6565
result.contains('From: public.cr.seqera.io/wave/pixi:0.61.0-noble')
66-
result.contains('Stage: build')
6766
result.contains('pixi init --import /scratch/conda.yml')
6867
result.contains('pixi shell-hook > /shell-hook.sh')
69-
result.contains('Bootstrap: docker')
70-
result.contains('From: ubuntu:24.04')
71-
result.contains('Stage: final')
68+
result.contains('%files')
69+
result.contains('%post')
70+
result.contains('%environment')
7271
}
7372

7473
def 'should use custom base image'() {

0 commit comments

Comments
 (0)