IATI Register Your Data (API)

Summary

Product	IATI Register Your Data (API)
Description	The Register Your Data product is an API and web app design to make write changes into the new IATI Registry. This repository contains a FastAPI application
Website	None
Related
Documentation	Rest of README.md
Technical Issues	GitHub issues page
Support	IATI Support Website

High-level requirements

• Python 3.12.11
• Docker
• OpenSSL (or similar) to generate public/private key pairs for the audit log (on a local development instance, if necessary).

Running

Overview

Currently, the API application requires the following resources to run:

• A JSON Web Key Set server: the configuration details for the production server can be obtained from https://api.eu.asgardeo.io/t/iati/oauth2/token/.well-known/openid-configuration but in the future we should define a local IDP, such as SIOCODE-Open/local-idp, for local development and/or integration testing.
• Public/private key pair to encrypt the audit log (only the public key is required for running the application). These can be generated with OpenSSL using openssl genrsa -out private-key.pem 4096 && openssl rsa -in private-key.pem -pubout -out public-key.pem.
• .env configuration file (see .env.example and a description of the variables below).

The API application is NOT currently wired up to the CRM.

Configuration

The application is configured using a set of environment variables in a .env file. The application looks in the current directory for this file.

Variable	Description
JWKS_URI	URI for the JWKS, typically this should be obtained from https://api.eu.asgardeo.io/t/iati/oauth2/token/.well-known/openid-configuration or the Asgardeo console.
ACCESS_CHECK_ENDPOINT	True enables the /api/v1/access_check endpoint to enable applications to check that the API can be called using an access token.
APP_LOG_LEVEL	Log level for the application diagnostic log ("DEBUG", "INFO", "WARNING", "ERROR", or "CRITICAL" to match the standard Python logging levels).
APP_LOG_PATH	Location for the application diagnostic log.
AUDIT_LOG_PATH	Location for the audit log.
AUDIT_LOG_PUBLIC_KEY_PATH	Path to the public key for encrypting audit logs.
PROMETHEUS_PORT	Port to serve Prometheus metrics from.
JWT_AUDIENCE	Audience that we expect to find in JWTs from the identity server.

FineGrainedAuthorisation database migrations

The FineGrainedAuthorisation system uses a postgres database. Migrations are handled using alembic. Alembic is configured using the pyproject.toml file. If the FGA models change or new ones are added, you can get alembic to generate a migration file with the following command:

alembic -c pyproject.toml revision --autogenerate -m "Commit message"

Note: alembic cannot autogenerate the sqlalchemy commands or SQL code for every type of migration, so after it has autogenerated a migration, check the newly created file in alembic/versions to see what migration code is has created and if it is adequate.

The database is not currently automatically migrated whenever FastAPI starts, as this is not best practice. To migrate the database when a version is released, the following command (suiteable for CI/CD pipelines) can be used:

alembic -c pyproject.toml upgrade head

If you need to reset the database (will likely cause data loss) to the initial state, you can use the command:

alembic -c pyproject.toml downgrade base

Running locally in development mode

To run the API locally in development mode execute the following from the command line.

fastapi dev src/main.py

This uses the FastAPI CLI with a built-in uvicorn ASGI server. To run locally in production mode change dev to run.

Running in production mode from a Docker container

The docker container can be built with:

docker build . --tag rydapi

Running the API in production can be run directly from its Docker container with:

docker run \
  --mount type=bind,source=.env,target=/api/.env,readonly \
  --mount type=bind,source=./logs/,target=/api/logs/ \
  --mount type=bind,source=./keys/,target=/api/keys/,readonly \
  -p 8000:8000 \
  -p 9000:9000 \
  rydapi

This performs the following setup:

• --mount type=bind,source=.env,target=/api/.env,readonly: shares .env configuration file on the host with the container.
• --mount type=bind,source=./logs/,target=/api/logs/: allows the container to write logs to the /logs directory on the host.
• --mount type=bind,source=./keys/,target=/api/keys/,readonly: shares the /keys directory on the host with the container so public and private keys can be used by the container.
• -p 8000:8000: shares port 8000 for API traffic.
• -p 9000:9000: shares port 9000 for Prometheus metrics (assuming that this is the port as specified by the .env file.)

Care should be taken to make sure that the .env variables match the log (APP_LOG_PATH and AUDIT_LOG_PATH) and key directories (AUDIT_LOG_PUBLIC_KEY_PATH) and the Prometheus metric port (PROMETHEUS_PORT).

Development

Adding new dependencies

New dependencies are added to pyproject.toml. Once these have been added requirements.txt and/or requirements_dev.txt need to be regenerated. With:

pip-compile --output-file=requirements.txt --strip-extras

and/or

pip-compile --extra=dev --output-file=requirements_dev.txt --strip-extras

Checking and linting

Linting is setup with isort and black and checked with flake8. Static type checking is performed by mypy. Configurations are stored in pyproject.toml. To use these linters and checkers you will first need to install the development dependencies:

pip install -r requirements_dev.txt

Then the code can be linted with:

black .
isort .

And checked with flake8 and mypy:

flake8
mypy .

Security testing

Static Security Application Testing is performed with bandit. To use this tool you must also install the development dependencies and then test with:

bandit -c pyproject.toml -r .

Versioning

The version is set in pyproject.toml. When making updates, set the version to an appropriate value. The version is fixed to 0.1.0 until we have a working API on a dev instance (albeit with mocked data).

Audit Log Viewer

There is a command line application which can be used to decrypt and print the encrypted audit log files. This is primarily useful for use while development. This is the audit_log_viewer.py file.

Configuration

The audit_log_viewer.py cli is configured using the following environment variables:

AUDIT_LOG_PATH=/path/to/log/file
AUDIT_LOG_PRIVATE_KEY_PATH=/path_to_private_key

These values will be automatically loaded from a .env in the root of the repository.

Use

To display the audit log file configured by AUDIT_LOG_PATH use:

python src/audit_log_viewer.py

To decrypt and print log lines from stdin use:

python src/audit_log_viewer.py --use-stdin

The latter allows you to tail -f the encrypted log file and pipe it into the viewer.

License

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.