diff --git a/README.md b/README.md index 6c65100..6bddb81 100644 --- a/README.md +++ b/README.md @@ -27,12 +27,12 @@ Please, [let us know](https://github.com/Infineon/arduino-devops/issues) if you You can get started by enabling the following workflows for your Arduino assets: -* [Enable *compile-examples* build check for Arduino assets](tutorial) -* [Enable *release* management for Arduino assets](tutorial-release) +* [Enable *compile-examples* build check for Arduino assets](https://ifx-arduino-devops.readthedocs.io/en/latest/compile-examples/getting-started.html) +* [Enable *release* management for Arduino assets](https://ifx-arduino-devops.readthedocs.io/en/latest/release/getting-started.html) ## More information -Find the complete information in the repo [docs](docs/). +Find the complete information in the repo [docs](https://ifx-arduino-devops.readthedocs.io/en/latest). ## Contributing diff --git a/docs/compile-examples/getting-started.rst b/docs/compile-examples/getting-started.rst index 42c07f3..cd41660 100644 --- a/docs/compile-examples/getting-started.rst +++ b/docs/compile-examples/getting-started.rst @@ -12,7 +12,7 @@ Before you start, make sure you satisfy the following prerequisites: - Your Arduino asset (library or core) is hosted in a GitHub repository. - GitHub Actions is enabled in your repository. Check the `GitHub docs `_ to learn how to enable it. -- If your asset is an Arduino core, first you need to `configure the repository `_ to be able to generate the core package with the ``core-packager.py`` tool. +- If your asset is an Arduino core, first you need to :doc:`configure the repository <../scripts/arduino-packager>` to be able to generate the core package with the ``core-packager.py`` tool. Enabling the reusable workflow ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/img/release-from-ga-select-version.png b/docs/img/release-from-ga-select-version.png new file mode 100644 index 0000000..819c3f6 Binary files /dev/null and b/docs/img/release-from-ga-select-version.png differ diff --git a/docs/img/release-from-ga-wflow.png b/docs/img/release-from-ga-wflow.png new file mode 100644 index 0000000..701def3 Binary files /dev/null and b/docs/img/release-from-ga-wflow.png differ diff --git a/docs/img/release-from-release-new-release.png b/docs/img/release-from-release-new-release.png new file mode 100644 index 0000000..633d21f Binary files /dev/null and b/docs/img/release-from-release-new-release.png differ diff --git a/docs/img/release-from-release-tag.png b/docs/img/release-from-release-tag.png new file mode 100644 index 0000000..3150ceb Binary files /dev/null and b/docs/img/release-from-release-tag.png differ diff --git a/docs/img/release-wflow-success.png b/docs/img/release-wflow-success.png new file mode 100644 index 0000000..096e5f8 Binary files /dev/null and b/docs/img/release-wflow-success.png differ diff --git a/docs/overview/motivation.rst b/docs/overview/motivation.rst index 1550d05..9eaa5cb 100644 --- a/docs/overview/motivation.rst +++ b/docs/overview/motivation.rst @@ -11,12 +11,12 @@ Background The introduction of `arduino-cli `_ has been a significant step forward in enabling programmatic handling of the Arduino toolchain and processes. This has opened up possibilities for automation via scripts and command line interfaces, making it easier to integrate tasks like build verification and testing into continuous integration servers. -Current State and Objectives +Current state and objectives ---------------------------- While existing DevOps solutions, such as `compile-sketches action `_, provide some functionality, they do not fully meet our requirements. This project aims to fill this gap by providing a collection of utilities that solve specific DevOps use cases for managing Arduino assets. The scope of this project is intentionally open-ended, allowing it to evolve and adapt to emerging needs. -Key Features and Design Principles +Key features and design principles ---------------------------------- The project's features are designed to be used via command line interface, making it easy to integrate with any non-GUI CI/CD framework and environment. Python is the language of choice, ensuring cross-platform compatibility. Although our primary development and testing platform is Linux, we strive to ensure platform independence. @@ -25,7 +25,7 @@ When applicable, the features presented rely on the Arduino `platform `_. This allows for easy deployment and upgrading of workflows with minimal effort. Given GitHub's popularity among Arduino developers, we have chosen to focus on GitHub Actions initially. However, we believe it should be straightforward to port the workflows to other CI/CD frameworks, such as GitLab, Jenkins, or Travis. -Community Engagement and Contributions +Community engagement and contributions -------------------------------------- We believe these tools can be of general interest and applicability to any Arduino library or core developer. We encourage users to try them out, provide feedback, and contribute to making them more extensible and adaptable to various use cases. \ No newline at end of file diff --git a/docs/overview/usage.rst b/docs/overview/usage.rst index 6266058..76d3ff2 100644 --- a/docs/overview/usage.rst +++ b/docs/overview/usage.rst @@ -5,7 +5,7 @@ This repository is designed to automate DevOps processes for Arduino libraries a Thus, we recommend starting by adding the available `reusable workflows `_ to your GitHub-hosted library or core. Before running the workflows, you'll need to configure them with some minimal settings. -Step-by-Step Enablement +Step-by-step enablement ----------------------- To begin, we recommend following our Getting Started guides, which will walk you through the process of enabling the workflows: @@ -13,12 +13,12 @@ To begin, we recommend following our Getting Started guides, which will walk you - :ref:`Getting started with compile-examples ` workflow enablement. Learn how to set up the compile-examples workflow and start automating your build processes. - :ref:`Getting started with release ` workflow enablement. Discover how to enable the release workflow and streamline your library or core releases. -Advanced Customizations and Local Usage +Advanced customizations and local usage --------------------------------------- As you become more familiar with the workflows and scripts, you might need advanced customizations to tailor them to your needs. You can also use the scripts locally on your development machine. -Running Scripts Locally +Running scripts locally ~~~~~~~~~~~~~~~~~~~~~~~ To run the scripts locally, first ensure you have the required dependencies installed (see :ref:`Requirements `). diff --git a/docs/release/description.rst b/docs/release/description.rst new file mode 100644 index 0000000..9cf5c21 --- /dev/null +++ b/docs/release/description.rst @@ -0,0 +1,38 @@ +Description +------------ + +The ``release`` workflow provides an automated and standardized way to create releases for Arduino cores and libraries. +It is designed to be used in CI/CD pipelines, enabling seamless integration with GitHub Actions to automate the entire release process. + +.. image:: ../img/release-wflow-success.png + :alt: Release workflow run success + :width: 60% + :align: center + +| + +The workflow integrates with the :doc:`arduino-release.py <../scripts/arduino-release>` script to handle semantic versioning, changelog generation, and GitHub release creation. +It supports both Arduino cores and libraries, automatically detecting the asset type and applying the appropriate release procedures. + +The workflow is designed as a reusable GitHub Action that can be called from other repositories, providing a consistent release process across multiple Arduino assets. + +Implementation details +^^^^^^^^^^^^^^^^^^^^^^ + +From a high level, the GitHub Action workflow implements the following functionality: + +#. **Validate inputs** and determine the asset type (core or library). +#. **Setup the environment** with required tools and dependencies using the provided setup script. +#. **Execute the release process** using the arduino-release.py script with the specified version. +#. **Create a GitHub release** with automatically generated changelog and release notes. +#. **Upload release assets** including packaged cores or library archives. + +The workflow supports: + +- **Semantic versioning** with automatic version validation +- **Changelog generation** from commit history and pull request information +- **Multi-platform compatibility** for different operating systems +- **Flexible setup scripts** to accommodate different project requirements +- **Automatic asset packaging** for both cores and libraries + +Please explore the `release.yml `_ file to find out more about the implementation details of the reusable workflow. diff --git a/docs/release/getting-started.rst b/docs/release/getting-started.rst new file mode 100644 index 0000000..42de619 --- /dev/null +++ b/docs/release/getting-started.rst @@ -0,0 +1,204 @@ +.. _release_getting_started: + +Getting started +---------------- + +This section will guide you through the steps to setup the release workflow in your Arduino asset repository. + +Prerequisites +^^^^^^^^^^^^^^ + +Before you start, make sure you satisfy the following prerequisites: + +- Your Arduino asset (library or core) is hosted in a GitHub repository. +- GitHub Actions is enabled in your repository. Check the `GitHub docs `_ to learn how to enable it. +- You have appropriate permissions to create releases in your repository. +- If your asset is an Arduino core, ensure it's properly configured for packaging with the :doc:`arduino-packager.py <../scripts/arduino-packager>` tool. + +Setting up the reusable workflow +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In your locally cloned repository, follow these steps: + +0. Create a new branch, for example ``devops/release-workflow``. + + .. note:: + This is not strictly necessary, but recommended to avoid pushing changes directly to the main branch. + +1. Create a new file in the ``.github/workflows`` directory, for example ``release.yml``. + +2. Add the following content to the file: + + .. code:: yaml + + name: Release + + on: + push: + tags: + - '[0-9]+.[0-9]+.[0-9]+**' + workflow_dispatch: + inputs: + version: + description: 'Release version' + required: true + default: '' + type: choice + options: + - patch + - minor + - major + + jobs: + arduino-devops: + uses: Infineon/arduino-devops/.github/workflows/release.yml@latest + with: + version: ${{ inputs.version }} + secrets: inherit + + .. important:: + Notice the ``@latest`` at the end of the ``uses`` line. This will use the latest version of the reusable workflow. + This is particularly useful to automatically get the latest updates in the reusable workflow. + Keep in mind that this may introduce breaking changes in case of major version updates. + You can also specify a specific version, for example ``@0.4.0``, or a branch, for example ``@main``. + +3. Commit your changes and push the branch to the remote. + +Additional workflow options +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The GitHub Action reusable workflow provides some additional options often used in the release process: + +- ``setup-script`` + Add a setup script that will be executed before building the release asset. + For example: + + .. code:: yaml + + jobs: + compile-examples: + uses: Infineon/arduino-devops/.github/workflows/compile-examples.yml@latest + with: + setup-script: bash scripts/before-core-build-setup.sh + +- ``release-ssh-key`` + This is used to provide a secret SSH key from the repository secrets to the workflow. + + For example: + + .. code:: yaml + + jobs: + arduino-devops: + uses: Infineon/arduino-devops/.github/workflows/release.yml@latest + with: + version: ${{ inputs.version }} + secrets: + release-ssh-key: ${{ secrets.release_wflow_key }} + + This is required for Arduino libraries when using protection rules that prevent direct + commit to the default branch. + + As the workflow creates new commits and pushes them to the remote repository, the new + commits will be rejected. + + To avoid this, you can use the SSH key to push the changes to the remote repository. + + Follow the steps to allow GitHub Actions to push changes directly to the default (protected) branch: + + 1. Add a new `deploy key `_ to your repository with write access. + This is available in the repository settings under "Deploy keys". + 2. `Add the private key to your repository secrets `_ with the name ``release_wflow_key``. + 3. If you have any protection rules that prevent direct commits to the default branch, + `enable the bypass option `_ for deploy keys. + + More information about this approach in this `GitHub discussion `_. + +Creating a release +^^^^^^^^^^^^^^^^^^ + +Once the workflow is set up, you can create releases in different ways: + +Release via GitHub actions +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Navigate to the ``Actions`` tab of your repository in GitHub. +2. Select the ``Release`` workflow from the list. + + .. image:: ../img/release-from-ga-wflow.png + :alt: Release workflow selection + :width: 60% + :align: center + + | + +3. Click ``Run workflow`` and select the default branch (usually ``main`` or ``master``) and the version increment: major, minor, or patch. + + .. image:: ../img/release-from-ga-select-version.png + :alt: Release workflow select version + :width: 50% + :align: center + + | + +4. The workflow will automatically: + + - Validate the version format + - Generate a changelog + - Create a GitHub release + - Upload release assets + +5. Once the workflow completes, you will see all the release steps passing, and the new release will be available in the ``Releases`` section of your repository. + +Release via GitHub release section +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Navigate to the ``Releases`` section of your repository. + + .. image:: ../img/release-from-release-new-release.png + :alt: Release from release section + :width: 80% + :align: center + + | + +2. Click on the ``Draft a new release`` button. + + .. image:: ../img/release-from-release-tag.png + :alt: Set new tag manually + :width: 60% + :align: center + + | + +3. Fill in the release tag (e.g., ``0.9.0``), and click on ``Create new tag: 0.9.0``. + + .. warning:: + You are going to set the tag manually. It should be a valid semantic version (semver) format, with a valid increment. + +4. This will trigger the release workflow, and it will automatically: + + - Validate the version format + - Generate a changelog + - Create a GitHub release + - Upload release assets + +5. Once the workflow completes, the new release will be updated with the corresponding title, changelog and assets. + +Release from local push +~~~~~~~~~~~~~~~~~~~~~~~ + +In your local repository, use the ``arduino-script.py`` to create a new release. Assuming that you are in the root path of your repository, and that the script is located in the ``arduino-devops`` directory: + + .. code:: bash + + python arduino-devops/arduino-script.py new + +This will automatically modify the required files, create a new tag, and push the changes to the remote repository. +The release workflow will be triggered, verifying the version and creating the release. +Check the ``release`` section of your repository to see the new release. + + .. warning:: + In case of an Arduino library, make sure that you have the necessary permissions to push modifications directly to the default (usually ``main`` or ``master``) branch of the repository. + The script will not only create a tag, but also update the `library.properties` file with the new version and push the changes to the remote repository. + The remote will reject the push if you do not have the right permissions. diff --git a/docs/release/index.rst b/docs/release/index.rst index daca443..406a06d 100644 --- a/docs/release/index.rst +++ b/docs/release/index.rst @@ -1,16 +1,11 @@ Release -======= +------- -Description ------------ +This section contains documentation for the release workflow and related scripts. -.. _release_getting_started: +.. toctree:: + :maxdepth: 2 -Getting started ----------------- - -Usage ------ - -Asset creation --------------- + description + getting-started + diff --git a/docs/scripts/arduino-cli-install.rst b/docs/scripts/arduino-cli-install.rst index 97696ca..0482bca 100644 --- a/docs/scripts/arduino-cli-install.rst +++ b/docs/scripts/arduino-cli-install.rst @@ -1,2 +1,68 @@ arduino-cli-install -=================== \ No newline at end of file +=================== + +The ``arduino-cli-install`` script is a cross-platform Python utility that automates the installation of the `Arduino CLI `_ tool on Linux, macOS, and Windows systems. + +Arduino CLI is a command-line interface that provides all the functionality of the Arduino IDE in a scriptable and automatable way. It's particularly useful for continuous integration (CI) workflows, automated testing, and batch operations. + +Features +-------- + +- **Cross-platform support**: Works on Linux, macOS, and Windows +- **Version-specific installation**: Install any specific version of Arduino CLI +- **Automated download and setup**: Handles downloading, extracting, and placing the executable in the system PATH +- **Error handling**: Provides clear error messages if installation fails + +Usage +----- + +Basic usage: + +.. code:: bash + + python arduino-cli-install.py + +Where ```` is the specific version of Arduino CLI you want to install. + +Examples +^^^^^^^^ + +Install a specific version: + +.. code:: bash + + python arduino-cli-install.py 0.34.2 + +Installation process +-------------------- + +The script performs the following steps based on your operating system: + +**Linux and macOS:** + +1. Downloads the appropriate tar.gz archive from the Arduino CLI releases +2. Extracts the archive +3. Moves the executable to ``/usr/local/bin/`` (requires sudo privileges) +4. Cleans up temporary files + +**Windows:** + +1. Downloads the Windows zip archive using PowerShell +2. Extracts the archive +3. Moves the executable to ``C:\Windows\System32\`` (requires administrator privileges) +4. Cleans up temporary files + +Prerequisites +------------- + +- Python 3.x installed on your system +- Internet connection to download Arduino CLI +- Administrator/sudo privileges for system-wide installation +- On Linux/macOS: ``curl`` and ``tar`` commands available +- On Windows: PowerShell available + +.. note:: + The script installs Arduino CLI system-wide, making it available from any directory in your terminal or command prompt. + +.. warning:: + This script requires elevated privileges (sudo on Linux/macOS, administrator on Windows) to install Arduino CLI in system directories. Make sure you trust the source before running with elevated privileges. \ No newline at end of file diff --git a/docs/scripts/arduino-packager.rst b/docs/scripts/arduino-packager.rst index a6ad3be..c10ee6c 100644 --- a/docs/scripts/arduino-packager.rst +++ b/docs/scripts/arduino-packager.rst @@ -1,5 +1,227 @@ arduino-packager ================ -In the case of Arduino cores, it also takes care of packaging the core assets, which includes the sources and -the `package index manifest `_ with the core metadata, to be able to be installed by the Arduino IDE or CLI. \ No newline at end of file +The ``arduino-packager`` script generates the package assets for Arduino cores installation. +These include: + +- The archive *.zip* file with the core `required sources and metafiles `_. +- The `package index manifest `_ file, which contains the information required by Arduino (IDE and CLI) to install the core. + +It requires two files for the asset generation: + +- **The core packager YAML file** + + This file is used by the ``arduino-packager`` in both the creation of the sources archive and the package index JSON file. + It specifies which files need to be included in the package, and provides some of the field values of the package index JSON. + See section `Packager config YAML specification`_ to check all the configurable parameters. + +- **The package index manifest template** + + It includes the JSON schema that needs to be filled in for every release. Some of the keys are already pre-filled. These are constant for every release. + + The rest of the empty keys will be added by the ``arduino-packager.py`` script based on the core sources archive and the provided packager configuration YAML. + + +.. note:: + + Please locate these 2 files inside the ``package`` directory of the core root path. This is the default location where the tool will look for them. + + +Additionally, the script will also include all the previous releases (``platforms`` key) based on the JSON package index manifest of the previous release. +This way, the user will have access to all the existing versions of the core. + +Packager config YAML specification +---------------------------------- + +The YAML file supports the following keys: + + - package-name: + The name of the core package archive. + + - include: + List of files and directories to include in the core package + archive. + + - index-name: + The name of the index file without the ".json" extension. + + - server: + The server information for uploading the package. + This is open to different specifications and approaches + for the package sources archive and package index file hosting. + Currently, the only one supported is: + + **GitHub** + + The GitHub server should have the following keys: + + - type: + github + - owner: + The owner of the repository (user name or organization) + - repo: + The repository name + + From this information, we can generate: + + - The URL to download the package. + - The URL of the package index JSON file. + + + Optional keys: + + - exclude: + Specific files or subdirectories to exclude from any of the + directories in the include list. + + This exclude list will be processed after the include list. + Therefore, if a file is in both the exclude and + include lists, it will be excluded. + + - index-template: + The template for the package index file. + If not provided, the expected default value is: + "package/.template.json" + + Reference example YAML file: + + .. code:: yaml + + package-name: arduino-core-vendor-product + include: + - cores + - libraries + - variants + - boards.txt + - platform.txt + - LICENSE.md + - README.md + - post_install.sh + index-name: package_vendor_product_index + server: + type: github + owner: vendor-owner + repo: arduino-core-repo-name + + .. note:: + + Most of the configuration data in the YAML file can have a default value and/or can be auto-discovered. + Future versions of the scripts will try to provide such features. + This will simplify the configuration as much as possible, while providing alternatives through the YAML file to override the defaults. + +Package index manifest template +-------------------------------- + +The package index manifest template is a JSON file that contains the schema for the package index manifest. +Find the description of each field in the `Arduino CLI package index JSON specification `_. +This is the schema with the pre-filled keys containing dummy values, which can be fixed for every release: + +.. code:: json + + { + "packages": [ + { + "name": "company_vendor", + "maintainer": "company_or_person_maintainer", + "websiteURL": "url_to_the_documentation", + "email": "contact_email@email.com", + "platforms": [ + { + "name": " Boards", + "architecture": "", + "version": "", + "category": "Contributed", + "url": "", + "archiveFileName": "", + "checksum": "", + "size": "", + "help": { + "online": "" + }, + "boards": [ + { + "name": "board_a_name" + }, + { + "name": "board_b_name" + } + ], + "toolsDependencies": [ + { + "packager": "vendor", + "name": "tool_x", + "version": "x.y.z" + }, + { + "packager": "company", + "name": "tool_a", + "version": "a.b.c" + } + ] + } + ], + "tools": [ + { + "name": "tool_x", + "version": "x.y.z", + "systems": [ + { + "host": "i686-mingw32", + "archiveFileName": "tool_x-x.y.z-windows.zip", + "url": "https://url-to-package/tool_x-x.y.z-windows.zip", + "checksum": "SHA-256:a96002b4aebbd89b1ace6a2da3f591d27d168fe56c1decd401bfb7a88d509260", + "size": "305588533" + }, + { + "host": "x86_64-pc-linux-gnu", + "archiveFileName": "tool_x-x.y.z-linux.tar.gz", + "url": "https://url-to-package/tool_x-x.y.z-linux.tar.gz", + "checksum": "SHA-256:76fb2d76080c3c2966983d8d8053cb7082416b83d8f5f6caa3d69bc8287d7846", + "size": "212727150" + }, + { + "host": "x86_64-apple-darwin", + "archiveFileName": "tool_x-x.y.z-macosx.tar.gz", + "url": "https://url-to-package/tool_x-x.y.z-macosx.tar.gz", + "checksum": "SHA-256:081417273428cd7c929e8206c823c8681e53c84ee86f8ce95f71e726702e7d75", + "size": "206451774" + } + ] + }, + { + "name": "tool_a", + "version": "a.b.c", + "systems": [ + { + "host": "i686-mingw32", + "archiveFileName": "tool_a-a.b.c-windows.zip", + "url": "https://url-to-package/tool_a-a.b.c-windows.zip", + "checksum": "SHA-256:858d7370ea29bd70f22a017976248c7df82076d401a05420b1a479ebadb1f7b2", + "size": "4932550" + }, + { + "host": "x86_64-pc-linux-gnu", + "archiveFileName": "tool_a-a.b.c-linux.tar.gz", + "url": "https://url-to-package/tool_a-a.b.c-linux.tar.gz", + "checksum": "SHA-256:2ea4721039cbfd3859ddac87eda8c3a10284c7404a77fdcb69e9a96378ea880b", + "size": "4620637" + }, + { + "host": "x86_64-apple-darwin", + "archiveFileName": "tool_a-a.b.c-macos.zip", + "url": "https://url-to-package/tool_a-a.b.c-macos.zip", + "checksum": "SHA-256:563dd0ad0dfabc05a8832c068e85b330d93248cc907978b29c3d3d95710dc077", + "size": "4901567" + } + ] + } + ] + } + ] + } + +By default, the template will be searched in the ``package`` directory of the core root path, with the name ``.template.json``. + +You can override this by providing the ``index-template`` key in the packager YAML file. + + diff --git a/docs/scripts/arduino-release.rst b/docs/scripts/arduino-release.rst index c1b55ca..5edf66e 100644 --- a/docs/scripts/arduino-release.rst +++ b/docs/scripts/arduino-release.rst @@ -1,2 +1,376 @@ arduino-release -=============== \ No newline at end of file +=============== + +The ``arduino-release`` script is an utility for managing semantic versioning releases of Arduino assets (libraries and cores). It supports with the automation of the entire release process including version validation, CI workflow verification, git tagging, and library.properties file management. + +Overview +-------- + +This tool provides a complete solution for releasing Arduino projects following semantic versioning (semver) best practices. It integrates with GitHub Actions to ensure all CI workflows pass before creating releases, maintains proper version consistency across different files, and automates git operations. + +The script supports both **Arduino libraries** and **Arduino cores**, automatically detecting the asset type and applying the appropriate release workflow. + +Features +-------- + +- **Semantic versioning support**: Full semver compliance with validation +- **CI/CD integration**: Verifies GitHub Actions workflows before release +- **Automatic version bumping**: Supports major, minor, and patch increments +- **library.properties management**: Automatically updates version for libraries +- **Git automation**: Handles tagging, committing, and pushing +- **Branch protection**: Ensures releases only from main/master branches +- **Release verification**: Post-release validation capabilities +- **Asset type detection**: Automatically identifies libraries vs cores + +Supported asset types +--------------------- + +**Arduino libraries:** + - Detected by presence of ``library.properties`` file + - Automatically updates version field in ``library.properties`` + - Creates commit for version change before tagging + +**Arduino cores:** + - Detected by presence of ``platform.txt``, ``boards.txt``, ``cores/``, and ``variants/`` + - Only performs git tagging (no file modifications needed) + +Usage +----- + +The script provides three main subcommands: + +.. code:: bash + + python arduino-release.py [options] + +Commands +^^^^^^^^ + +**new** + Create a new release with version validation and CI checks. + +**verify** + Verify that all requirements are met for an existing release. + +**info** + Display information about the current repository state. + +Basic examples +^^^^^^^^^^^^^^ + +Create a new patch release: + +.. code:: bash + + python arduino-release.py new patch + +Create a new minor release: + +.. code:: bash + + python arduino-release.py new minor + +Create a new major release: + +.. code:: bash + + python arduino-release.py new major + +Create a specific version release: + +.. code:: bash + + python arduino-release.py new 2.1.0 + +Verify an existing release: + +.. code:: bash + + python arduino-release.py verify + +Get repository information: + +.. code:: bash + + python arduino-release.py info head-tag + python arduino-release.py info repo-name + python arduino-release.py info asset-type + +Detailed command reference +-------------------------- + +new +^^^ + +Creates a new release for the specified version. + +**Syntax:** + +.. code:: bash + + python arduino-release.py new [options] + +**Arguments:** + +``version`` + The new version to create. Can be: + + - **Bump type**: ``major``, ``minor``, or ``patch`` + - **Specific version**: Semver format like ``1.2.3``, ``2.0.0-rc.1`` + +**Options:** + +``--root-path `` + Path to the Arduino asset root directory. + + **Default**: Current working directory + +``--any-branch`` + Allow releases from any branch (bypasses main/master requirement). + + **Default**: false (releases only from main/master) + +**Process flow:** + +1. **Branch validation**: Ensures HEAD is on main/master branch +2. **CI verification**: Waits for and validates all GitHub Actions workflows +3. **Tag check**: Ensures HEAD is not already tagged +4. **Version calculation**: Determines new version from current tags +5. **Version validation**: Validates semver compliance and proper increment +6. **File updates**: For libraries, updates ``library.properties`` +7. **Git operations**: Creates commit (if needed), tag, and pushes to remote + +**Example process output:** + +.. code:: text + + --> Checking if repo HEAD in the default branch... [OK] + --> Waiting for all ci workflows completion for HEAD... [OK] + --> Checking if HEAD is not already tagged... [OK] + --> Previous last version: 1.2.0 + --> New bumped version: 1.3.0 + --> Updating "library.properties" file with new version... [OK] + --> Committing changes to library.properties file... [OK] + --> Creating new git tag... [OK] + --> Pushing new git tag... [OK] + +verify +^^^^^^ + +Verifies that all requirements are met for an existing release. + +**Syntax:** + +.. code:: bash + + python arduino-release.py verify [options] + +**Validation checks:** + +1. **CI workflows**: All GitHub Actions workflows must be successful +2. **Version validity**: Current tag must be valid semver increment from previous +3. **File consistency**: For libraries, ``library.properties`` version must match git tag + +**Use cases:** +- Post-release validation in CI/CD pipelines +- Troubleshooting release issues +- Ensuring release integrity + +info +^^^^ + +Displays information about the current repository state. + +**Syntax:** + +.. code:: bash + + python arduino-release.py info [options] + +**Available keys:** + +``head-tag`` + Shows the git tag of the current HEAD commit. + +``repo-name`` + Shows the repository name (without owner). + +``asset-type`` + Shows whether the asset is a "library" or "core". + +**Examples:** + +.. code:: bash + + $ python arduino-release.py info head-tag + 1.2.3 + + $ python arduino-release.py info asset-type + library + +Version management +------------------ + +Semantic versioning rules +^^^^^^^^^^^^^^^^^^^^^^^^^ + +The script enforces strict semantic versioning rules: + +**Major version (X.0.0):** +- Breaking changes +- Minor and patch must be 0 +- Can only increment by 1 + +**Minor version (0.X.0):** +- New features, backward compatible +- Patch must be 0 +- Major stays the same, minor increments by 1 + +**Patch version (0.0.X):** +- Bug fixes, backward compatible +- Major and minor stay the same, patch increments by 1 + +**Pre-release support:** +- Supports release candidates: ``1.0.0-rc.1`` +- Can release final version from RC: ``1.0.0-rc.1`` → ``1.0.0`` + +Version bump examples +^^^^^^^^^^^^^^^^^^^^^ + +.. code:: bash + + # From 1.2.3 to 1.2.4 (patch) + python arduino-release.py new patch + + # From 1.2.3 to 1.3.0 (minor) + python arduino-release.py new minor + + # From 1.2.3 to 2.0.0 (major) + python arduino-release.py new major + + # Specific version + python arduino-release.py new 1.5.0 + +CI/CD integration +----------------- + +GitHub actions integration +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The script integrates deeply with GitHub Actions: + +1. **Workflow Detection**: Automatically finds all workflows for the HEAD commit +2. **Status monitoring**: Waits for in-progress workflows to complete +3. **Success validation**: Ensures all workflows pass before proceeding +4. **Release workflow Skip**: Ignores workflows containing "release" in the name + +**Workflow status handling:** + +- ``in_progress`` / ``requested``: Waits with progress animation +- ``success``: Continues with release process +- ``failure``: Stops and exits with error + +**API requirements:** + +- Uses GitHub REST API (no authentication required for public repos) +- Requires internet connection +- Works with both public and private repositories + +Prerequisites +------------- + +**System requirements:** +- Python 3.x with required packages: ``requests``, ``semver`` +- Git installed and configured +- Internet connection for GitHub API access + +**Repository requirements:** +- Git repository with GitHub remote (https or ssh) +- At least one existing commit +- For libraries: valid ``library.properties`` file +- For cores: valid ``platform.txt``, ``boards.txt``, ``cores/``, ``variants/`` + +**Git configuration:** +- Git user.name and user.email configured (or will be set to GitHub Actions bot) +- Push permissions to the remote repository +- Proper authentication for git push operations + +Error handling +-------------- + +The script provides comprehensive error handling with detailed messages: + +**Common errors:** + +``Release is allowed only from permanent branches`` + Solution: Switch to main/master branch or use ``--any-branch`` + +``This HEAD is already tagged`` + Solution: Make new commits or use existing tag for verification + +``The new version is not a valid semver increment`` + Solution: Check version rules and use proper semver increments + +``All CI workflows must be successful`` + Solution: Fix failing workflows before creating release + +``Package index file not found`` + Solution: Ensure proper Arduino asset structure + +**Exit codes:** +- ``0``: Success +- ``1``: Error occurred (check error message for details) + +Advanced usage +-------------- + +**Release from feature branch:** + +.. code:: bash + + # Allow releases from any branch (use with caution) + python arduino-release.py new patch --any-branch + +**Custom root directory:** + +.. code:: bash + + # Release from specific directory + python arduino-release.py new minor --root-path /path/to/arduino/project + +**CI/CD pipeline integration:** + +.. code:: yaml + + # GitHub Actions workflow example + - name: Create Release + run: | + python arduino-release.py new patch + + - name: Verify Release + run: | + python arduino-release.py verify + +**Batch operations:** + +.. code:: bash + + # Get current version for other scripts + CURRENT_VERSION=$(python arduino-release.py info head-tag) + echo "Current version: $CURRENT_VERSION" + +Best practices +-------------- + +1. **Always run from main/master**: Use default branch protection unless absolutely necessary +2. **Verify CI first**: Ensure all tests pass before creating releases +3. **Use semantic versioning**: Follow semver rules for version increments +4. **Test releases**: Use the verify command to validate releases +5. **Automate in CI**: Integrate into GitHub Actions for consistent releases +6. **Backup important work**: Script modifies git history and files + +.. warning:: + This script performs git operations including creating commits, tags, and pushing to remote repositories. Always ensure your work is backed up and you have proper permissions before running. + +.. tip:: + Use the ``--any-branch`` flag carefully in development environments. For production releases, always use the default main/master branch requirement. \ No newline at end of file diff --git a/docs/scripts/pckg-install-local.rst b/docs/scripts/pckg-install-local.rst index b6a7f8e..351fa9a 100644 --- a/docs/scripts/pckg-install-local.rst +++ b/docs/scripts/pckg-install-local.rst @@ -1,2 +1,179 @@ pckg-install-local -================== \ No newline at end of file +================== + +The ``pckg-install-local`` script is a Python utility that enables local installation and testing of Arduino core packages during development. + +It allows to install an Arduino core without publishing it to a remote repository, from a local directory. +The `arduino-cli` can only install cores which are hosted in a remote server. +In CI/CD this is convenient for two use cases: + +- Working with the commit triggering the CI/CD pipeline for an Arduino core. +- Prevent the CI/CD downloads to increase the download statistics of the core package by installing it from a published release, which will skew the real user download statistics. + +Overview +-------- + +When developing an Arduino core, you typically need to test your changes before publishing. This script automates the process of: + +1. Starting a local HTTP server to host your package files (as arduino-cli cannot install from a local directory) +2. Creating a modified package index file with local URLs +3. Uninstalling any existing version of the core +4. Installing the core from your local development files +5. Cleaning up temporary files + +This tool can be used for cores generated by the :doc:`arduino-packager` tool. + +Features +-------- + +- **Local HTTP server**: Automatically starts a local server on a random port (8000-8200 range) +- **Package index modification**: Creates a local version of your package index with localhost URLs +- **Clean installation**: Removes existing installations and staging files before installing +- **Arduino CLI integration**: Uses Arduino CLI commands for proper core management +- **Verbose logging**: Optional detailed output for debugging +- **Cross-platform support**: Works on Linux, macOS, and Windows + +Usage +----- + +Basic usage: + +.. code:: bash + + python pckg-install-local.py + +With options: + +.. code:: bash + + python pckg-install-local.py --pckg-dir /path/to/package --verbose + +Command line options +^^^^^^^^^^^^^^^^^^^^ + +``--pckg-dir `` + Path to the package directory containing the Arduino core package files. + + **Default**: Current working directory + + **Example**: ``--pckg-dir ./build/`` + +``--verbose`` + Enable verbose logging output to see detailed information about the installation process. + + **Example**: ``--verbose`` + +``-v, --version`` + Display the version of the pckg-install-local tool. + + **Example**: ``--version`` + +Package directory structure +--------------------------- + +The package directory must contain: + +1. **Package index file**: A ``.json`` file (e.g., ``package_mycore_index.json``) +2. **Core archive**: A ``.zip`` file containing the Arduino core (e.g., ``arduino-core-mycore-1.0.0.zip``) + +Example directory structure: + +.. code:: text + + package/ + ├── package_mycore_index.json # Package index manifest + └── arduino-core-mycore-1.0.0.zip # Arduino core archive + +Installation process +-------------------- + +The script performs the following steps: + +1. **Server setup** + + - Starts a local HTTP server on a random port (8000-8200) + - Serves files from the specified package directory + +2. **Package index modification** + + - Locates the package index JSON file in the directory + - Creates a local copy with ``_local`` suffix (e.g., ``package_mycore_index_local.json``) + - Updates the package URL to point to the local server + +3. **Core cleanup** + + - Uninstalls any existing version of the core using ``arduino-cli core uninstall`` + - Removes staging files from Arduino's installation directory + - Cleans up previous local package index files + +4. **Core installation** + + - Updates Arduino CLI core index + - Installs the core using the local package index URL + - Lists installed cores to verify installation + +Prerequisites +------------- + +- **Python 3.x** installed on your system +- **Arduino CLI** installed and available in your system PATH +- **Package files** generated by the arduino-packager tool or manually created +- **Network access** for Arduino CLI to update its index (internet connection) + +.. note:: + The script requires Arduino CLI to be properly installed and configured. You can install it using the :doc:`arduino-cli-install` script. + +Examples +-------- + +**Install from current directory:** + +.. code:: bash + + python pckg-install-local.py + +**Install with verbose output:** + +.. code:: bash + + python pckg-install-local.py --verbose + +**Install from specific package directory:** + +.. code:: bash + + python pckg-install-local.py --pckg-dir ./build/package --verbose + +**Check version:** + +.. code:: bash + + python pckg-install-local.py --version + +Troubleshooting +--------------- + +**"Package index file not found" error:** + Ensure your package directory contains a ``.json`` file (package index manifest). + +**"Package archive file not found" error:** + Ensure your package directory contains a ``.zip`` file (Arduino core archive). + +**"Server could not start" error:** + The script tries random ports between 8000-8200. If all ports are busy, try closing other applications or restart your system. + +**Arduino CLI command fails:** + Verify that Arduino CLI is properly installed and accessible from the command line: + + .. code:: bash + + arduino-cli version + +**Core installation fails:** + Check that your package index file is valid JSON and contains the required Arduino core package structure. + +.. warning:: + This script modifies Arduino CLI's core installations. Make sure to backup any important Arduino configurations before running it. + +.. tip:: + Use the ``--verbose`` flag to see detailed logging output when troubleshooting installation issues. \ No newline at end of file