feat: add python dependencies at runtime without rebuilding image

mlabeeb03 · mlabeeb03 · commit c7fe9fde0f58 · 2025-09-17T19:12:38.000+05:00
Since adding new python packages in LMS/CMS containers through tutor core requires rebuilding the openedx image, we have created this plugin that can add these dependencies at runtime without the need to rebuild images.
The plugin implements a live dependencies system with the following components:

A tutor do job (build_live_dependencies) that reads package names from LIVE_DEPENDENCIES config variable, installs them in a job runner container, zips the packages, and uploads to Django storage
A monitor_livedeps daemon in each container that watches for storage changes and triggers uwsgi restarts when packages are updated
An update_livedeps script that runs on uwsgi server start/restart, it downloads and extracts packages to a specified directory
Modified uwsgi configuration to run the monitor_livedeps script as a daemon and execute the update_livedeps script on start/restart
Dockerfile changes to set PYTHONPATH and copy required scripts into containers

The solution works with both Docker Compose and Kubernetes deployments by using persistent storage instead of mounted volumes.

Requirements: tutor-minio plugin or compatible S3 storage backend must be installed for persistent package storage
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -0,0 +1,27 @@
+name: Run tests
+
+on:
+  pull_request:
+    branches: [ release, main ]
+  push:
+    branches: [ release, main ]
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+         python-version: ['3.9', '3.12']
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Upgrade pip
+        run: python -m pip install --upgrade pip
+      - name: Install dependencies
+        run: |
+          pip install .[dev]
+      - name: Test lint, types, and format
+        run: make test
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,7 @@
+.*.swp
+!.gitignore
+TODO
+__pycache__
+*.egg-info/
+/build/
+/dist/
diff --git a/LICENSE.txt b/LICENSE.txt
diff --git a/MANIFEST.in b/MANIFEST.in
@@ -0,0 +1,2 @@
+recursive-include tutorlivedeps/patches *
+recursive-include tutorlivedeps/templates *
diff --git a/Makefile b/Makefile
@@ -0,0 +1,42 @@
+.DEFAULT_GOAL := help
+.PHONY: docs
+SRC_DIRS = ./tutorlivedeps
+
+# Warning: These checks are not necessarily run on every PR.
+test: test-lint test-types test-format test-pythonpackage  # Run some static checks.
+
+test-format: ## Run code formatting tests
+	ruff format --check --diff ${SRC_DIRS}
+
+test-lint: ## Run code linting tests
+	ruff check ${SRC_DIRS}
+
+test-types: ## Run type checks.
+	mypy --exclude=templates --ignore-missing-imports --implicit-reexport --strict ${SRC_DIRS}
+
+build-pythonpackage: ## Build the "tutor-livedeps" python package for upload to pypi
+	python -m build --sdist
+
+test-pythonpackage: build-pythonpackage ## Test that package can be uploaded to pypi
+	twine check dist/tutor_livedeps-$(shell make version).tar.gz
+
+format: ## Format code automatically
+	ruff format ${SRC_DIRS}
+
+fix-lint: ## Fix lint errors automatically
+	ruff check --fix ${SRC_DIRS}
+
+changelog-entry: ## Create a new changelog entry.
+	scriv create
+
+changelog: ## Collect changelog entries in the CHANGELOG.md file.
+	scriv collect
+
+version: ## Print the current tutor-livedeps version
+	@python -c 'import io, os; about = {}; exec(io.open(os.path.join("tutorlivedeps", "__about__.py"), "rt", encoding="utf-8").read(), about); print(about["__version__"])'
+
+ESCAPE = 
+help: ## Print this help
+	@grep -E '^([a-zA-Z_-]+:.*?## .*|######* .+)$$' Makefile \
+		| sed 's/######* \(.*\)/@               $(ESCAPE)[1;31m\1$(ESCAPE)[0m/g' | tr '@' '\n' \
+		| awk 'BEGIN {FS = ":.*?## "}; {printf "\033[33m%-30s\033[0m %s\n", $$1, $$2}'
diff --git a/README.rst b/README.rst
@@ -0,0 +1,42 @@
+Real time python package installation for Open edX
+==================================================
+
+This is a plugin for `Tutor <https://docs.tutor.edly.io>`_ that provides real time python package installation for Open edX platforms. This means you do not need to rebuild the openedx image whenever you add a new package. This is achieved by storing these packages in Django storage and downloading them inside each LMS/CMS container.
+
+Installation
+------------
+
+This plugin depends on the `tutor-minio <https://github.com/overhangio/tutor-minio>`_ plugin to store the packages. After installing tutor-minio run the following commands:
+
+``pip install git+https://github.com/overhangio/tutor-livedeps.git``
+
+``tutor plugins enable livedeps``
+
+
+Configuration
+-------------
+
+``LIVE_DEPENDENCIES`` (default: ``"[]"``)
+
+To add a new package to this config run 
+
+``tutor config save --append LIVE_DEPENDENCIES=package_name``
+
+To remove an old package from this config run 
+
+``tutor config save --remove LIVE_DEPENDENCIES=package_name``.
+
+Then run the following command to install the packages that are present in the ``LIVE_DEPENDENCIES`` config:
+
+``tutor dev/local/k8s do build-live-dependencies``
+
+
+Troubleshooting
+---------------
+
+This Tutor plugin is maintained by Muhammad Labeeb from `Edly <https://edly.io>`__. Community support is available from the official `Open edX forum <https://discuss.openedx.org>`__. Do you need help with this plugin? See the `troubleshooting <https://docs.tutor.edly.io/troubleshooting.html>`__ section from the Tutor documentation.
+
+License
+-------
+
+This work is licensed under the terms of the `GNU Affero General Public License (AGPL) <https://github.com/overhangio/tutor-minio/blob/release/LICENSE.txt>`_.
diff --git a/changelog.d/20250916_182709_mlabeeb03.md b/changelog.d/20250916_182709_mlabeeb03.md
@@ -0,0 +1 @@
+- [Feature] Add python dependencies at runtime without rebuilding image. (by @mlabeeb03)
diff --git a/changelog.d/scriv.ini b/changelog.d/scriv.ini
@@ -0,0 +1,18 @@
+[scriv]
+version = literal: tutorlivedeps/__about__.py: __version__
+categories =
+format = md
+md_header_level = 2
+new_fragment_template =
+    <!--
+    Create a changelog entry for every new user-facing change. Please respect the following instructions:
+    - Indicate breaking changes by prepending an explosion 💥 character.
+    - Prefix your changes with either [Bugfix], [Improvement], [Feature], [Security], [Deprecation].
+    - You may optionally append "(by @<author>)" at the end of the line, where "<author>" is either one (just one)
+    of your GitHub username, real name or affiliated organization. These affiliations will be displayed in
+    the release notes for every release.
+    -->
+
+    <!-- - 💥[Feature] Foobarize the blorginator. This breaks plugins by renaming the `FOO_DO` filter to `BAR_DO`. (by @regisb) -->
+    <!-- - [Improvement] This is a non-breaking change. Life is good. (by @billgates) -->
+entry_title_template = {%% if version %%}v{{ version }} {%% endif %%}({{ date.strftime('%%Y-%%m-%%d') }})
diff --git a/pyproject.toml b/pyproject.toml
@@ -0,0 +1,61 @@
+# https://packaging.python.org/en/latest/tutorials/packaging-projects/
+# https://hatch.pypa.io/latest/config/build/
+
+[project]
+name = "tutor-livedeps"
+description = "livedeps plugin for Tutor"
+authors = [
+  { name = "Muhammad Labeeb"},
+  { email = "muhammad.labeeb@arbisoft.com" },
+]
+license = { text = "AGPL-3.0-only" }
+
+readme = {file = "README.rst", content-type = "text/x-rst"}
+requires-python = ">= 3.9"
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: GNU Affero General Public License v3",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: Implementation :: CPython",
+    "Programming Language :: Python :: Implementation :: PyPy",
+
+]
+dependencies = [
+    "tutor>=20.0.0,<21.0.0",
+    "tutor-minio>=20.0.0,<21.0.0",
+]
+optional-dependencies = { dev = ["tutor[dev]>=20.0.0,<21.0.0"] }
+
+# These fields will be set by hatch_build.py
+dynamic = ["version"]
+
+[tool.hatch.version]
+path = "tutorlivedeps/__about__.py"
+
+# https://packaging.python.org/en/latest/specifications/well-known-project-urls/#well-known-labels
+[project.urls]
+Documentation = "https://github.com/overhangio/tutor-livedeps#readme"
+Issues = "https://github.com/overhangio/tutor-livedeps/issues"
+Source = "https://github.com/overhangio/tutor-livedeps"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["tutorlivedeps"]
+
+[tool.hatch.build.targets.sdist]
+# Disable strict naming, otherwise twine is not able to detect name/version
+strict-naming = false
+include = [ "/tutorlivedeps"]
+exclude = ["tests*"]
+
+[project.entry-points."tutor.plugin.v1"]
+livedeps = "tutorlivedeps.plugin"
diff --git a/tutorlivedeps/__about__.py b/tutorlivedeps/__about__.py
@@ -0,0 +1 @@
+__version__ = "20.0.0"
diff --git a/tutorlivedeps/__init__.py b/tutorlivedeps/__init__.py
diff --git a/tutorlivedeps/patches/.gitignore b/tutorlivedeps/patches/.gitignore
diff --git a/tutorlivedeps/patches/openedx-dockerfile-final b/tutorlivedeps/patches/openedx-dockerfile-final
@@ -0,0 +1,8 @@
+# Set the PYTHONPATH to include the live dependencies folder
+ENV PYTHONPATH=/openedx/live-dependencies/deps/lib/python3.11/site-packages
+
+# Copy live dependencies scripts and trigger file
+RUN mkdir -p /openedx/live-dependencies
+RUN touch /openedx/live-dependencies/uwsgi_trigger
+COPY --chown=app:app settings/livedeps/build/monitor_livedeps.py /openedx/live-dependencies/
+COPY --chown=app:app settings/livedeps/build/update_livedeps.py /openedx/live-dependencies/
diff --git a/tutorlivedeps/patches/uwsgi-config b/tutorlivedeps/patches/uwsgi-config
@@ -0,0 +1,6 @@
+# Run a daemon to detects changes in live dependencies zip file and triggers reloads
+attach-daemon = python /openedx/live-dependencies/monitor_livedeps.py
+# This file is touched by the monitor_livedeps.py script to initiate a reload
+touch-reload = /openedx/live-dependencies/uwsgi_trigger
+# This script downloads persistent packages on server starts/reload
+exec-asap = python /openedx/live-dependencies/update_livedeps.py
diff --git a/tutorlivedeps/plugin.py b/tutorlivedeps/plugin.py
@@ -0,0 +1,97 @@
+import os
+import typing as t
+from glob import glob
+
+import click
+import importlib_resources
+from tutor import config as tutor_config
+from tutor import hooks
+from tutor.commands.context import Context
+
+from .__about__ import __version__
+
+########################################
+# CONFIGURATION
+########################################
+
+hooks.Filters.CONFIG_DEFAULTS.add_items(
+    [
+        ("LIVEDEPS_VERSION", __version__),
+        ("LIVE_DEPENDENCIES", []),
+    ]
+)
+
+
+########################################
+# TEMPLATE RENDERING
+########################################
+
+hooks.Filters.ENV_TEMPLATE_ROOTS.add_items(
+    # Root paths for template files, relative to the project root.
+    [
+        str(importlib_resources.files("tutorlivedeps") / "templates"),
+    ]
+)
+
+hooks.Filters.ENV_TEMPLATE_TARGETS.add_items(
+    [
+        ("livedeps/build", "build/openedx/settings"),
+    ],
+)
+
+
+########################################
+# PATCH LOADING
+########################################
+
+
+for path in glob(str(importlib_resources.files("tutorlivedeps") / "patches" / "*")):
+    with open(path, encoding="utf-8") as patch_file:
+        hooks.Filters.ENV_PATCHES.add_item((os.path.basename(path), patch_file.read()))
+
+
+########################################
+# CUSTOM JOBS (a.k.a. "do-commands")
+########################################
+
+
+@click.command(
+    help="Build all live dependencies, zip them and upload to storage backend"
+)
+@click.pass_obj
+def build_live_dependencies(context: Context) -> t.Iterable[tuple[str, str]]:
+    """
+    Build the live dependencies and upload using Django's storage API.
+    You need to update the `LIVE_DEPENDENCIES` variable in the config file to add/remove packages.
+    """
+    config = tutor_config.load(context.root)
+    all_packages = " ".join(
+        package for package in t.cast(list[str], config["LIVE_DEPENDENCIES"])
+    )
+
+    script = f"""
+    pip install \
+    --prefix=/openedx/live-dependencies/deps \
+    {all_packages} \
+    && python3 -c '
+import os, shutil, tempfile
+from django.core.files.storage import storages
+from django.core.files.base import File
+
+DEPS_DIR = "/openedx/live-dependencies/deps"
+DEPS_KEY = "deps.zip"
+
+with tempfile.TemporaryDirectory(prefix="tutor-livedeps-") as zip_dir:
+    base = os.path.join(zip_dir, DEPS_KEY)
+    archive_path = shutil.make_archive(base[:-4], format="zip", root_dir=DEPS_DIR)
+
+    with open(archive_path, "rb") as f:
+        # TODO Use a separate storage for live dependencies
+        storages["default"].save(DEPS_KEY, File(f))
+'
+    """
+
+    yield ("lms", script)
+
+
+hooks.Filters.CLI_DO_COMMANDS.add_item(build_live_dependencies)
diff --git a/tutorlivedeps/templates/livedeps/apps/.gitignore b/tutorlivedeps/templates/livedeps/apps/.gitignore
diff --git a/tutorlivedeps/templates/livedeps/build/.gitignore b/tutorlivedeps/templates/livedeps/build/.gitignore
diff --git a/tutorlivedeps/templates/livedeps/build/monitor_livedeps.py b/tutorlivedeps/templates/livedeps/build/monitor_livedeps.py
@@ -0,0 +1,30 @@
+import datetime
+import os
+import time
+
+from django.core.files.storage import storages
+
+DEPS_DIR = "/openedx/live-dependencies/deps"
+DEPS_KEY = "deps.zip"
+DEPS_ZIP_PATH = DEPS_DIR[:-4] + DEPS_KEY
+TRIGGER_FILE = "/openedx/live-dependencies/uwsgi_trigger"
+
+# TODO Use a separate storage for live dependencies
+storage = storages["default"]
+
+while True:
+    if storage.exists(DEPS_KEY):
+        remote_ts = storage.get_modified_time(DEPS_KEY)
+
+        if os.path.exists(DEPS_ZIP_PATH):
+            local_ts = os.path.getmtime(DEPS_ZIP_PATH)
+            local_ts = datetime.datetime.fromtimestamp(
+                local_ts, tz=datetime.timezone.utc
+            )
+
+            if local_ts < remote_ts:
+                with open(TRIGGER_FILE, "a"):
+                    os.utime(TRIGGER_FILE, None)
+    # TODO What happens if download takes more than 10 seconds?
+    # This could keep initiating a reload in a loop.
+    time.sleep(10)
diff --git a/tutorlivedeps/templates/livedeps/build/update_livedeps.py b/tutorlivedeps/templates/livedeps/build/update_livedeps.py
@@ -0,0 +1,23 @@
+import os
+import shutil
+import zipfile
+
+from django.core.files.storage import storages
+
+DEPS_DIR = "/openedx/live-dependencies/deps"
+DEPS_KEY = "deps.zip"
+DEPS_ZIP_PATH = DEPS_DIR[:-4] + DEPS_KEY
+
+# TODO Use a separate storage for live dependencies
+storage = storages["default"]
+
+if storage.exists(DEPS_KEY):
+    if os.path.exists(DEPS_DIR):
+        shutil.rmtree(DEPS_DIR)
+    os.makedirs(DEPS_DIR, exist_ok=True)
+
+    with storage.open(DEPS_KEY, "rb") as remote_f, open(DEPS_ZIP_PATH, "wb") as local_f:
+        shutil.copyfileobj(remote_f, local_f)
+
+    with zipfile.ZipFile(DEPS_ZIP_PATH, "r") as zip_ref:
+        zip_ref.extractall(DEPS_DIR)
diff --git a/tutorlivedeps/templates/livedeps/tasks/lms/.gitignore b/tutorlivedeps/templates/livedeps/tasks/lms/.gitignore

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+recursive-include tutorlivedeps/patches *`
	`2`	`+recursive-include tutorlivedeps/templates *`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+- [Feature] Add python dependencies at runtime without rebuilding image. (by @mlabeeb03)`