Too many open files - 1024 and 4096 soft and hard open file limit

[zmarty]

Sorry for being terse, I am writing this on my phone because I am in a locked down corporate environment.

I created a Python microsandbox image with more data analysis packages such as scikit using a custom Dockerfile. I noticed that inside the container I cannot properly load complex Python packages because I immediately get Too many open file errors as soon as Python tries to import these packages. I have verified that the container has 1024 and 4096 soft and hard limits respectively, which is very low. It should be closer to 65K.

I tried setting higher limits in /etc/security/limits.conf but I am having trouble having the file have higher limits, something overwrites the file. And I am not sure limits.conf is even used in microsandbox / libkrun containers.

I hope this is something easy but I failed to fix it. I need a working image with the most common common Python data analysis packages that has higher limits. Please help, thanks!

[appcypher]

Sorry for the late response. msb actually supports changing the limits but it is not exposed to the API yet.

https://github.com/microsandbox/microsandbox/blob/2fa57fd0609356c86447aeafdaae7324a61f5cb4/microsandbox-core/lib/vm/rlimit.rs#L36-L37

Specifically. Setting rlimit is missing from the sandbox config.

https://github.com/microsandbox/microsandbox/blob/2fa57fd0609356c86447aeafdaae7324a61f5cb4/microsandbox-core/lib/config/microsandbox/config.rs#L261-L290

The vanilla linux kernel has a hard 4096 limit but distros increase it to a mil.
https://github.com/torvalds/linux/blob/78f4e737a53e1163ded2687a922fce138aee73f5/include/uapi/linux/fs.h#L38

A temp soln would be to increase the limit but I'm not sure what the implications are. I will look at what Docker does.

[zmarty]

Docker has it configurable.

/etc/sysconfig/docker

OPTIONS="--default-ulimit nofile=1024000:1024000"

[zmarty]

@appcypher I gave it a shot and I tested it locally. Please check the warning in the commit though: #288

It would be awesome to have this functionality, we really need it to run useful Python code safely.

[zmarty]

I also created a multiarchitecture data science Docker image with the top libraries and tested that it can now successfully import and work with them: https://hub.docker.com/repository/docker/zmarty/microsandbox-python-data-science/tags

Here is the Dockerfile if you are curious:

# First stage: Build the portal binary with Python feature enabled
FROM rust:slim AS builder

# Install dependencies and build the portal binary
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    pkg-config \
    libssl-dev \
    && apt-get clean && rm -rf /var/lib/apt/lists/* \
    && mkdir -p /usr/src/microsandbox

WORKDIR /usr/src/microsandbox
COPY . .
RUN cargo build --release --bin portal --features python

# Second stage: Create the Python Data Science image
FROM python:latest

# Set environment variables and create user
ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_NO_CACHE_DIR=1 \
    DEBIAN_FRONTEND=noninteractive

ARG USER_NAME="python-user"
ARG USER_UID="1000"
ARG USER_GID="100"

# Install system dependencies, SQLite, create user, set up directories, and install Python packages in one layer
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    curl \
    git \
    wget \
    libssl-dev \
    ca-certificates \
    sqlite3 \
    libsqlite3-dev \
    graphviz \
    graphviz-dev \
    libgraphviz-dev \
    pkg-config \
    && apt-get clean && rm -rf /var/lib/apt/lists/* \
    # Create non-root user and necessary directories
    && useradd -m -s /bin/bash -N -u $USER_UID $USER_NAME \
    && mkdir -p /home/$USER_NAME/work \
    && chown -R $USER_NAME:$USER_GID /home/$USER_NAME \
    && mkdir -p /etc/microsandbox/portal \
    && chown -R $USER_NAME:$USER_GID /etc/microsandbox \
    # Install core Python packages
    && pip install --no-cache-dir --upgrade pip setuptools wheel \
    && pip install --no-cache-dir \
        # Development and testing tools
        black \
        flake8 \
        mypy \
        pytest \
        pytest-cov \
        requests \
        ipython \
        jupyter \
        jupyterlab \
        # Core data science packages
        numpy \
        pandas \
        scipy \
        scikit-learn \
        matplotlib \
        seaborn \
        plotly \
        bokeh \
        altair \
        # Statistics and analysis
        statsmodels \
        pymc \
        arviz \
        pingouin \
        # Database packages
        sqlalchemy \
        sqlite-utils \
        # File format support
        openpyxl \
        xlrd \
        xlsxwriter \
        h5py \
        tables \
        pyarrow \
        fastparquet \
        # Utilities
        tqdm \
        click \
        pydantic \
        python-dateutil \
        networkx \
        beautifulsoup4 \
        lxml \
        # Image processing
        pillow \
        opencv-python-headless \
        # Text processing
        nltk \
        textblob \
        spacy \
        # Symbolic math
        sympy \
        # Additional visualization
        wordcloud \
        pygraphviz

# Copy the portal binary from the builder stage and set permissions
COPY --from=builder /usr/src/microsandbox/target/release/portal /usr/local/bin/
RUN chmod +x /usr/local/bin/portal

# Switch to the non-root user
USER $USER_NAME
WORKDIR /home/$USER_NAME/work

# Download NLTK data (commonly used datasets)
RUN python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords'); nltk.download('vader_lexicon')" || true

# Set a command that starts portal and keeps the container running
CMD ["bash", "-c", "echo 'Python Data Science environment with microsandbox-portal ready' && RUST_LOG=debug portal & tail -f /dev/null"]

Build commands for Ubuntu amd64

Prerequisites (Ubuntu/Linux)

1. Install QEMU for cross-platform emulation:

   sudo docker run --privileged --rm tonistiigi/binfmt --install all

2. Create a multi-platform builder:

   sudo docker buildx create --name multiarch-builder --driver docker-container --use
   sudo docker buildx inspect --bootstrap

Build Command

For multi-architecture build and push to Docker Hub:

sudo docker buildx build --platform linux/amd64,linux/arm64 -t zmarty/microsandbox-python-data-science:latest -f sdk-images/python-data-science/Dockerfile . --push

[zmarty]

I tested the code using the custom image above and using a random csv dataset I found online, and here is the result - it was even able to create a sqlite3 DB and run queries on it:

I also want to mention that I cannot use this for real until it is in the main branch and there is a new release of microsandbox, because of the way the corporate environment I work on is set up ...

[appcypher]

Thanks for great detailed issue. One of the things I definitely want is not having to use our custom dockerfile to support a custom image. I will be working on the server and other aspects over the next few days to improve the general dev experience. It is going to be a significant overhaul mostly fixing the filesystem and networking.