From 20c318365af5b5e65bc2f38c65830d8e1e5da24a Mon Sep 17 00:00:00 2001 From: Baptiste Date: Thu, 24 Jul 2025 15:25:41 +0200 Subject: [PATCH 1/6] Entire rework of the generator guide --- .../documentation.instructions.md | 118 +++- docs/docs/guides/generator.mdx | 664 +++++++++++------- 2 files changed, 509 insertions(+), 273 deletions(-) diff --git a/.github/instructions/documentation.instructions.md b/.github/instructions/documentation.instructions.md index 3d34df9dbe..1ea1d7b33b 100644 --- a/.github/instructions/documentation.instructions.md +++ b/.github/instructions/documentation.instructions.md @@ -5,6 +5,8 @@ applyTo: "**/*.mdx" This master prompt serves as a comprehensive guide for AI systems tasked with writing technical documentation for Infrahub by OpsMill. The prompt defines the objectives, structure, tone, style, and key considerations necessary to produce clear, useful, and accurate documentation tailored to the needs of Infrahub users. +The documentation structure follows the [Diataxis framework](https://diataxis.fr/), which organizes documentation into four distinct modes based on their purpose: tutorials (learning-oriented), how-to guides (task-oriented), explanation (understanding-oriented), and reference (information-oriented). This prompt focuses primarily on how-to guides and explanation documentation to ensure content meets users' specific needs effectively. + ## 🧑‍💻 Role Definition The assumed role for generating documentation is that of an Expert Technical Writer and MDX Generator. @@ -40,6 +42,22 @@ The documentation is both an onboarding and a reference tool, serving developers - Informative over promotional: Focus on explaining how and why, not on marketing. - Consistent and structured: Follow a predictable pattern across sections and documents. +### For Guides + +- Use conditional imperatives: "If you want X, do Y. To achieve W, do Z." +- Focus on practical tasks and problems, not the tools themselves +- Address the user directly using imperative verbs: "Configure...", "Create...", "Deploy..." +- Maintain focus on the specific goal without digressing into explanations +- Use clear titles that state exactly what the guide shows how to accomplish + +### For Topics + +- Use a more discursive, reflective tone that invites understanding +- Include context, background, and rationale behind design decisions +- Make connections between concepts and to users' existing knowledge +- Present alternative perspectives and approaches where appropriate +- Use illustrative analogies and examples to deepen understanding + ## 📄 Source and Style References Refer to the project style guides and templates provided in the current repository: @@ -64,24 +82,29 @@ If you can't find any of the references described above, please mention it in th Adjust complexity and terminology accordingly, erring on the side of accessibility. -## 🪵 Document Structure and Patterns +## 🪵 Document Structure and Patterns Following Diataxis + +### Guides Structure (Task-oriented, practical steps) -If working on a **guide** or **tutorial**, please follow this structure: +How-to guides help users solve real-world problems and achieve specific goals with Infrahub. They are goal-oriented, focused on tasks, and follow a logical sequence of actions. ```markdown - Title and Metadata - - Title of the guide (YAML frontmatter) + - Title should clearly state what problem is being solved (YAML frontmatter) + - Begin with "How to..." to signal the guide's purpose - Optional: Imports for components (e.g., Tabs, TabItem, CodeBlock, VideoPlayer) - Introduction - - Brief overview of the guide's purpose - - Context or use case for the guide + - Brief statement of the specific problem or goal this guide addresses + - Context or real-world use case that frames the guide + - Clearly indicate what the user will achieve by following this guide - Optional: Links to related topics or more detailed documentation - Prerequisites / Assumptions - What the user should have or know before starting - - Optional: Environment setup or requirements + - Environment setup or requirements + - What prior knowledge is assumed - Step-by-Step Instructions - Step 1: [Action/Goal] - - Description of the step + - Clear, actionable instructions focused on the task - Code snippets (YAML, GraphQL, shell commands, etc.) - Screenshots or images for visual guidance - Tabs for alternative methods (e.g., Web UI, GraphQL, Shell/cURL) @@ -91,39 +114,54 @@ If working on a **guide** or **tutorial**, please follow this structure: - Step N: [Action/Goal] - Continue as needed - Validation / Verification - - How to check that the step(s) worked (e.g., inspecting in the Web UI) + - How to check that the solution worked as expected - Example outputs or screenshots -- Advanced Usage / Improvements - - Optional: Further enhancements, best practices, or next steps - - Optional: Migration, abstraction, or optimization tips -- Reference / Additional Resources - - Links to related guides, topics, or external resources + - Potential failure points and how to address them +- Advanced Usage / Variations + - Optional: Alternative approaches for different circumstances + - Optional: How to adapt the solution for related problems + - Optional: Ways to extend or optimize the solution +- Related Resources + - Links to related guides, reference materials, or explanation topics - Optional: Embedded videos or labs for further learning -- Conclusion / Summary - - Recap of what was achieved - - Optional: Success messages or next actions -- Appendix / Notes - - Optional: Additional notes, troubleshooting, or FAQ ``` -If working on a **topic** please follow this structure: +### Topics Structure (Understanding-oriented, theoretical knowledge) + +Topic or explanation documentation helps users understand concepts, background, and context. It's understanding-oriented and provides theoretical knowledge that serves the user's study of Infrahub. ```markdown - Title and Metadata - - Title of the topic (YAML frontmatter) + - Title should clearly indicate the topic being explained (YAML frontmatter) + - Consider using "About..." or "Understanding..." in the title - Optional: Imports for components (e.g., Tabs, TabItem, CodeBlock, VideoPlayer) - Introduction - - Brief overview of the topic's purpose + - Brief overview of what this explanation covers + - Why this topic matters in the context of Infrahub + - Questions this explanation will answer - Main Content Sections - - Overview / Summary - - Concepts / Definitions - - Key terms, concepts, or background information - - Architecture / Design (if applicable) - - Diagrams, images, or explanations of structure - - UI/UX Behavior - - Integration / Interactions - How this feature interacts with others - - Advanced Topics -- Links to related guide or external resources + - Concepts & Definitions + - Clear explanations of key terms and concepts + - How these concepts fit into the broader system + - Background & Context + - Historical context or evolution of the concept/feature + - Design decisions and rationale behind implementations + - Technical constraints or considerations + - Architecture & Design (if applicable) + - Diagrams, images, or explanations of structure + - How components interact or relate to each other + - Mental Models + - Analogies and comparisons to help understanding + - Different ways to think about the topic + - Connection to Other Concepts + - How this topic relates to other parts of Infrahub + - Integration points and relationships + - Alternative Approaches + - Different perspectives or methodologies + - Pros and cons of different approaches +- Further Reading + - Links to related topics, guides, or reference materials + - External resources for deeper understanding ``` ## ✅ Quality and Clarity Checklist @@ -134,3 +172,23 @@ Before submitting documentation, validate: - Instructions are clear, with step-by-step guidance where needed - Markdown formatting is correct and compliant with Infrahub's style - Spelling and grammar are checked + +### For Guides + +- The guide addresses a specific, practical problem or task +- The title clearly indicates what will be accomplished +- Steps follow a logical sequence that maintains flow +- Each step focuses on actions, not explanations +- The guide omits unnecessary details that don't serve the goal +- Validation steps help users confirm their success +- The guide addresses real-world complexity rather than oversimplified scenarios + +### For Topics + +- The explanation is bounded to a specific topic area +- Content provides genuine understanding, not just facts +- Background and context are included to deepen understanding +- Connections are made to related concepts and the bigger picture +- Different perspectives or approaches are acknowledged where relevant +- The content remains focused on explanation without drifting into tutorial or reference material +- The explanation answers "why" questions, not just "what" or "how" diff --git a/docs/docs/guides/generator.mdx b/docs/docs/guides/generator.mdx index 04f4f5569a..af0f6b9864 100644 --- a/docs/docs/guides/generator.mdx +++ b/docs/docs/guides/generator.mdx @@ -1,34 +1,130 @@ --- -title: Creating a Generator +title: How to Create a Generator --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -# Creating a generator in Infrahub +# How to create a generator in Infrahub -Within Infrahub a generator is defined in an [external repository](../topics/repository). However, during development and troubleshooting it is easiest to start from your local computer and run the Transform using [infrahubctl generator]($(base_url)infrahubctl/infrahubctl-generator). +Generators in Infrahub allow you to automatically create or modify data based on existing information in your database. They are defined in [external repositories](../topics/repository) and can be developed and tested locally using [infrahubctl generator]($(base_url)infrahubctl/infrahubctl-generator). -The goal of this guide is to develop a Generator and add it to Infrahub, we will achieve this by following these steps. +In this guide, you'll learn how to develop a generator and integrate it into Infrahub. -1. Create a Git repository -2. Create an entry for the generator within an .infrahub.yml file. -3. Identify the relevant data you want to extract from the database using a [GraphQL query](../topics/graphql), that can take an input parameter to filter the data -4. Write a Python script that uses the GraphQL query to read information from the system and generates new data based on the response -5. Test the generator with infrahubctl -6. Push changes to GitHub and add the repository to Infrahub as an external repository -7. Validate that the generator works by triggering it through a proposed change +By the end, you'll be able to: -## Preparations +- Create a generator that reads data from Infrahub using GraphQL +- Generate new objects based on that data +- Run the generator both locally and through Infrahub's CI pipeline -What your generator will look like will depend on what your schema looks like and the intended outcome. The generator described here will be very generic and also not useful in a real world scenario, it is only meant to describe how the generators work. +## What you'll build -As the default Infrahub schema doesn't have a lot of object types to use as a test, we will illustrate how this could work by adding two nodes to the schema. +You'll build a simple generator that: -### Create and load example schema +1. Reads information about Widget objects from the database +2. Creates a number of Resource objects based on each Widget's count property +3. Automatically runs when changes to Widget objects are proposed -Create a **widgets.yml** file with the following: +## Steps overview -```yaml title="widgets.yml" +1. Prepare the data model by creating a schema and objects +2. Create a GraphQL query to fetch data from Infrahub +3. Implement a Python generator that creates the data +4. Tie everything together in an `.infrahub.yml` file +5. Test the generator locally with infrahubctl +6. Deploy the generator to Infrahub +7. Validate that the generator works in Infrahub's CI pipeline + +## Prerequisites + +Before you begin, you'll need: + +- Basic knowledge of Infrahub, Python, GraphQL, YAML, and Git +- Familiarity with Generators and how they work in Infrahub (see [Generators topic](../topics/generator)) +- An Infrahub instance running locally or remotely +- A Git repository connected to Infrahub (see [Adding an External Repository guide](./repository)) +- [infrahubctl]($(base_url)infrahubctl/infrahubctl) installed and configured locally +- The repository cloned locally where you'll develop the generator + +## Step 1: Setting up the data model + +Since the default Infrahub schema doesn't include the object types we need for this example, we'll first create a simple schema with Widget and Resource nodes. + +### Create a new branch + +:::warning Version Control & Branching + +We recommend creating a new branch in your Git repository for your generator development to avoid affecting the main branch. In this guide, we'll use a branch named `create-widget-generator`. + +```shell +git checkout -b create-widget-generator +``` + +::: + +### Create group + +:::info Object file or UI + +If you prefer a "as code" approach, use the object file method. Otherwise, you can create the group directly in the Infrahub UI. + +::: + + + + 1. Create a file named `groups.yml` in the `objects` directory of your repository: + + ```yaml title="objects/groups.yml" + --- + apiVersion: infrahub.app/v1 + kind: Object + spec: + kind: CoreStandardGroup + data: + - name: "widgets" + ``` + + 2. Update the `.infrahub.yml` file to include the new object file: + + ```yaml title=".infrahub.yml" + --- + objects: + - file_path: objects/groups.yml + ``` + + 3. Commit the changes to your repository: + + ```shell + git add objects/groups.yml .infrahub.yml + git commit -m "Add widgets group" + git push + ``` + + + + 1. Open the **Infrahub UI** in your browser + 2. Select the branch `create-widget-generator` + 3. Navigate to the **Groups** view (Object Management > Groups) + 4. Create a **Standard group** named `widgets` + + + +:::success + +To verify the group was created successfully, navigate to the Groups view in the Infrahub UI and check that the `widgets` group appears in the list. + +::: + +### Create and load schema + +:::warning Schema Creation + +This step is only necessary if you don't already have the schemas affected by the generator in your instance. + +::: + +1. Create a `widgets.yml` file in the `schemas` directory of your repository: + +```yaml title="schemas/widgets.yml" # yaml-language-server: $schema=https://schema.infrahub.app/infrahub/schema/latest.json --- version: '1.0' @@ -36,8 +132,9 @@ version: '1.0' nodes: - name: Widget namespace: Test - label: "Widget" - default_filter: name__value + label: Widget + human_friendly_id: + - name__value display_labels: - name__value attributes: @@ -46,311 +143,392 @@ nodes: unique: true - name: count kind: Number + - name: Resource namespace: Test - label: "Resource" - default_filter: name__value + label: Resource + human_friendly_id: + - name__value display_labels: - name__value attributes: - name: name kind: Text unique: true + ``` -Load the following schema using the [infrahubctl schema]($(base_url)infrahubctl/infrahubctl-schema) command. +:::info Schema Loading -```shell -infrahubctl schema load widgets.yml -``` +Choose the method you already use to manage schema changes in your Infrahub deployment. -```shell -schema 'widgets.yml' loaded successfully -1 schema processed in 8.453 seconds. -``` +::: -### Add two new nodes + + + 2. Update the `.infrahub.yml` file to include the new schema file: -Perform these steps in the [Infrahub UI](http://localhost:8000). + ```yaml title=".infrahub.yml" + --- + schemas: + - schemas/widgets.yml + ``` -1. Create two new widget objects - - One with the name `widget1` and count 1 - - One with the name `widget2` and count 2 -2. Create a [Standard group](http://localhost:8000/objects/CoreGroup) called "**widgets** -3. Add both of the created objects to the new group + 3. Commit and push the changes to your repository + -:::info -Any widget object must be added to or be part of the **widgets** group to be tied to the generator. -::: + + 2. Load the following schema using the [infrahubctl schema]($(base_url)infrahubctl/infrahubctl-schema) command. + + ```shell + infrahubctl schema load schemas/widgets.yml --branch=create-widget-generator + ``` + + ```shell + schema 'widgets.yml' loaded successfully + 1 schema processed in 8.453 seconds. + ``` -## 1. Create GitHub repository + + -Follow GitHub's instructions on [creating a new repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-new-repository). +:::success -:::info Initialize with a README -For ease, it is suggested to **Initialize this repository with: Add a README file**. +You should see new entries for `Widget` and `Resource` in the left-hand side menu of the Infrahub UI. -This allows you to clone the repository down without any extra commands. ::: -Once the repository is created, [clone the repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) down and change directory into the cloned folder. +### Create widget objects + +Now that you have the schema loaded, you need to create objects that will be used by the generator: + +:::warning Object Creation -## 2. Create an .infrahub.yml file +This step is only necessary if you don't already have the objects affected by the generator in your instance. -The [.infrahub.yml](../topics/infrahub-yml) file allows you to tie the different [components of a generator](../topics/generator) together into a working unit. +::: -:::info Convert query to Infrahub SDK objects -We provide `convert_query_response` option to be toggled to be able to access objects from the GraphQL query as Infrahub SDK objects rather than the raw dictionary response. +:::note Object Creation Method -This allows you to manage the returned data with helper methods on the SDK objects such as `save`, `fetch`, etc. on the returned data rather than having to build a payload to send back to Infrahub to manage the objects. +You can also use an object file to create the widgets, though it's less relevant here than for the group creation. -Read more on the [Infrahub Python SDK]($(base_url)python-sdk/introduction). ::: - - - - - - ```yaml title=".infrahub.yml" - # yaml-language-server: $schema=https://schema.infrahub.app/python-sdk/repository-config/latest.json - --- - generator_definitions: - - name: widget_generator - file_path: "example_generator/widget_generator.py" - targets: widgets - query: widget_query - class_name: WidgetGenerator - parameters: - name: "name__value" - - queries: - - name: widget_query - file_path: "example_generator/widget_query.gql" - ``` - - - - - ```yaml title=".infrahub.yml" - # yaml-language-server: $schema=https://schema.infrahub.app/python-sdk/repository-config/latest.json - --- - generator_definitions: - - name: widget_generator - file_path: "example_generator/widget_generator.py" - targets: widgets - query: widget_query - convert_query_response: true - class_name: WidgetGenerator - parameters: - name: "name__value" - - queries: - - name: widget_query - file_path: "example_generator/widget_query.gql" - ``` - - +1. Open the **Infrahub UI** in your browser +2. Select the branch `create-widget-generator` +3. Navigate to the **Widget** view +4. Create a new widget: + - Name: `widget1` + - Count: `1` + - Member of groups: `widgets` +5. Create a second widget: + - Name: `widget2` + - Count: `2` + - Member of groups: `widgets` + +## Step 2: Create the GraphQL query + +Next, create a GraphQL query that fetches the data your generator needs to process. + +### Create a query file + +Create a `widget_query.gql` file in the `queries` directory of your repository: + +```graphql title="queries/widget_query.gql" +query Widgets($name: String!) { + TestWidget(name__value: $name) { + edges { + node { + __typename + id + name { + value + } + count { + value + } + } + } + } +} +``` - - - This defines a generator definition with the following properties: - - **name**: a unique name for the generator - - **file_path**: the relative file path to the file containing the generator as seen from within a Git repository - - **targets**: the name of a group of which the members will be a target for this generator - - **query**: the name of the GraphQL query used within this generator - - **convert_query_response**: convert the result of the GraphQL query to SDK InfrahubNode objects - - **class_name**: the name of the Python class in the generator file - - **parameters**: the parameter to pass to the generator GraphQL query, in this case this we will pass the name of the object (widget) as the name parameter - - - Here the `name` refers to the query's name and `file_path` should point to the GraphQL file within the repository. - - +### Test the query -See [this topic](../topics/infrahub-yml) for a full explanation of everything that can be defined in the `.infrahub.yml` file. +To test the query you can use **Infrahub's GraphQL Sandbox**. -## 3. Identify the relevant data +:::note GraphQL Sandbox -Here we define a GraphQL query that we will use to gather the information that will be passed into the generator. +Access the sandbox by clicking your user icon in the bottom left corner and selecting **GraphQL Sandbox**. This tool allows you to run GraphQL queries and provides an interactive way to explore the schema. - - - ```graphql title="example_generator/widget_query.gql" - query Widgets($name: String!) { - TestWidget(name__value: $name) { - edges { - node { - name { - value - } - count { - value +::: + +1. Copy the above GraphQL query in the main section + +2. In the **variables** section, add: + +```json +{ + "name": "widget1" +} +``` + +3. Click the **Execute** button, this should return a response like: + +```json +{ + "data": { + "TestWidget": { + "edges": [ + { + "node": { + "__typename": "TestWidget", + "id": "185526eb-2114-ce20-390a-c51aac78460a", + "name": { + "value": "widget1" + }, + "count": { + "value": 1 } } } - } - } - ``` - - - Here we must provide `__typename` and `id` within the query so the SDK can convert the query to the correct type and properly store within the SDK. - - ```graphql title="example_generator/widget_query.gql" - query Widgets($name: String!) { - TestWidget(name__value: $name) { - edges { - node { - __typename - id - name { - value - } - count { - value - } - } - } + ] } } - ``` +} +``` - - +:::success -Create a local directory on your computer where we will store the generator files. +You now have a working GraphQL query that retrieves the necessary data from Infrahub. You might want to keep the result of the query for later reference, as it will be useful when implementing the generator. + +::: + +## Step 3: Implement the generator + +Now create a Python class that implements your generator logic. The generator creates `TestResource` objects equal to the widget's count value. + +The class must: + +- Inherit from `InfrahubGenerator` +- Implement an async `generate()` method that processes data from your GraphQL query + +:::info + +If you aren't using `convert_query_response`, you can access the widget data directly from the `data` parameter passed to the `generate()` method. +If you're working with `protocols`, you can create `TestResource` objects using strict typing. + +::: + +1. Create a file named `widget_generator.py` in your `generators` directory: + +```python title="generators/widget_generator.py" +from infrahub_sdk.generator import InfrahubGenerator + +class WidgetGenerator(InfrahubGenerator): + async def generate(self, data: dict) -> None: + # Access the widget as an SDK object + widget = self.nodes[0] # or self.store.get(data["TestWidget"]["edges"][0]["node"]["id"]) + widget_name: str = widget.name.value + widget_count: int = widget.count.value + + # Create resources based on widget count + for count in range(1, widget_count + 1): + payload = { + "name": f"{widget_name.lower()}-{count}", + } + obj = await self.client.create(kind="TestResource", data=payload) + await obj.save(allow_upsert=True) -```shell -mkdir example_generator ``` -## 4. Create a Python Generator - -The Generator class needs to implement a `generate` function that receives a `data` parameter that contains the response from the GraphQL query. - -The goal of this generator will be to create a number of resources that depends on the set count of the widgets. - - - - - ```python title="example_generator/widget_generator.py" - from infrahub_sdk.generator import InfrahubGenerator - - class WidgetGenerator(InfrahubGenerator): - async def generate(self, data: dict) -> None: - # We can now access the nodes as typical SDK objects - widget = data["TestWidget"]["edges"][0]["node"] - widget_name: str = widget["name"]["value"] - widget_count: str = widget["count"]["value"] - for count in range(1, widget_count + 1): - payload = { - "name": f"{widget_name.lower()}-{count}", - } - obj = await self.client.create(kind="TestResource", data=payload) - await obj.save(allow_upsert=True) - ``` - - - - ```python title="example_generator/widget_generator.py" - from infrahub_sdk.generator import InfrahubGenerator - - class WidgetGenerator(InfrahubGenerator): - async def generate(self, data: dict) -> None: - # We can now access the nodes as typical SDK objects - widget = self.nodes[0] # or self.store.get(data["TestWidget"]["edges"][0]["node"]["id"]) - widget_name: str = widget.name.value - widget_count: str = widget.count.value - - for count in range(1, widget_count + 1): - - payload = { - "name": f"{widget_name.lower()}-{count}", - } - obj = await self.client.create(kind="TestResource", data=payload) - await obj.save(allow_upsert=True) - ``` - - +## Step 4: Tie everything together in the .infrahub.yml file - -Store this class within a new file called **widget_generator.py** within the **example_generator** directory. +Now that you have your GraphQL query and Python generator, edit your [.infrahub.yml](../topics/infrahub-yml) file to tie everything together. -## 5. Test the generator using infrahubctl +1. Add the following configuration to the file `.infrahub.yml`: -Using infrahubctl you can first verify that the **.infrahub.yml** file is formatted correctly by listing available generators. +```yaml title=".infrahub.yml" +--- +generator_definitions: + - name: widget_generator + file_path: "generators/widget_generator.py" + targets: widgets + query: widget_query + convert_query_response: true + class_name: WidgetGenerator + parameters: + name: "name__value" + +queries: + - name: widget_query + file_path: "queries/widget_query.gql" +``` + +For a complete explanation of the `.infrahub.yml` file format, see the [infrahub.yml topic](../topics/infrahub-yml). + +2. Verify the configuration + +Check that your `.infrahub.yml` file is correctly formatted by listing available generators: ```shell infrahubctl generator --list ``` +:::success + +If successful, you'll see output like: + ```shell Generators defined in repository: 1 -widget_generator (widget_generator.py::Generator) Target: widgets +widget_generator (generators/widget_generator.py::WidgetGenerator) Target: widgets ``` -:::warning +::: -When running a generator with `infrahubctl` the [SDK tracking]($(base_url)python-sdk/topics/tracking) feature isn't used. The reason for this is that internally Infrahub uses the ID of the generator_definition to control the tracking, this isn't available from the outside. For this reason it is recommended to create test branches when developing generators and validating the results. +## Step 5: Test the generator locally -::: +Before deploying your generator to Infrahub, test it locally using the `infrahubctl` command-line tool. -```shell -infrahubctl branch create test-branch1 -``` +### Run the generator -Then we can try to run the generator within this branch. +Run the generator on both of your widget objects: ```shell -infrahubctl generator widget_generator --branch=test-branch1 name=widget1 -infrahubctl generator widget_generator --branch=test-branch1 name=widget2 +infrahubctl generator widget_generator --branch=create-widget-generator name=widget1 +infrahubctl generator widget_generator --branch=create-widget-generator name=widget2 ``` -Now you should see the tree [TestResource](http://localhost:8000/objects/TestResource?branch=test-branch1) objects within `test-branch1` one for the first widget and two for the second one. +### Verify the results -![resources](../media/guides/generator_pc_3.png) +1. Select the branch `create-widget-generator` from the branch selector +2. Navigate to **Resource** objects +3. You should see: + - One resource object for `widget1` (named `widget1-1`) + - Two resource objects for `widget2` (named `widget2-1` and `widget2-2`) -Merge the changes into the **main** branch using [infrahubctl branch]($(base_url)infrahubctl/infrahubctl-branch) command. +![Generated resource objects](../media/guides/generator_pc_3.png) -```shell -infrahubctl branch merge 'test-branch1' -``` +:::warning -## 6. Commit and add repository to Infrahub +You might want to cleanup this development data before merging your changes into the main branch. -The root directory should include the **.infrahub.yml** file and **example_generator** folder. +::: + +:::success -- `.infrahub.yml`: Contains the definition for the generator +Your generator worked as expected, it's now time to deploy it to Infrahub. -Within the **example_generator** folder you should now have 2 files: +::: -- `widget_query.gql`: Contains the GraphQL query -- `widget_generator.py`: Contains the Python code for the generator +## Step 6: Deploy to Infrahub -Before we can run our generator in an Proposed Change (PC) pipeline we must push the files up to the remote Git repository we created in **Step 1**. +Now that you've tested your generator, deploy it to Infrahub. + +### Verify repository structure + +Ensure your repository has the following structure: + +:::note Schemas & Objects + +Depending on your organization, you might also have `schemas` and `objects` directories in your repository. + +::: + +```shell +your-repository/ +├── .infrahub.yml +├── generators/ +│ └── widget_generator.py +└── queries/ + └── widget_query.gql +``` + +### Commit and push your code + +Upload your generator code to the repository: ```shell git add . -git commit -m "Add generator" +git commit -m "Add widget generator" git push ``` -Now that we have the latest changes, [add your repository to Infrahub](./repository) and then the generators will be executed as part of the proposed change CI pipeline. +### Confirm the generator is imported + +After merging your changes, confirm that the generator is imported correctly by checking the Infrahub UI. + +1. Open the **Infrahub UI** in your browser +2. Select the `main` branch +3. Navigate to the **Generators Definition view** (Actions > Generator Definitions) +4. You should see your `widget_generator` listed there + +If you don't see it verify the status of the repository in the **Repository view** and ensure the sync status is `synced`. + +### Proposed changes and merge + +Now merge your development branch `create-widget-generator` into `main`. + +:::note Merging Options + +We recommend using Infrahub's **Proposed Changes** feature to merge your changes. This allows you to review both data and code changes before merging. + +::: + +1. Create a **Proposed Change** + - Source branch: `create-widget-generator` + - Destination branch: `main` + - Name: `Add widget generator` +2. Navigate to the **Data** and **Files** tabs to review the changes +3. Back to the **Overview** tab, click the **Merge** button + +:::success + +Your generator is now deployed to Infrahub in the `main` branch and ready to be used. + +::: + +## Step 7: Run the generator in Infrahub's CI pipeline + +Now let's verify that your generator automatically runs when new widgets are created: + +1. Create a new branch named `add-widget-3` +2. Navigate to the **Widget** objects in the Infrahub UI +3. Create a new widget: + - Name: `widget3` + - Count: `3` + - Member of groups: `widgets` +4. Create a **Proposed Change** + - Source branch: `add-widget-3` + - Destination branch: `main` + - Name: `Add widget3` + +### Check generator results + +1. Navigate to the **Checks** tab of your proposed change +2. Wait for the **generator** CI check to complete +![Generator CI Check](../media/guides/generator_pc_1.png) +3. Navigate to the **Data** tab +4. Click the **Refresh diff** button to see the resources created by your generator +![Data Refresh showing generated resources](../media/guides/generator_pc_2.png) + +:::success + +You should see three new resources (`widget3-1`, `widget3-2`, and `widget3-3`) automatically created by your generator. + +::: + +## Next steps + +Now that you've created a basic generator, you can: -## 7. Generator in the CI pipeline +- Create generators that handle a full object lifecycle, including creation, updates, and deletions +- Build idempotent generators that can be run multiple times without creating duplicate data +- Add unit tests to your generator to ensure it behaves as expected +- Harden your generator by adding error handling and logging -1. Navigate to [widgets](http://localhost:8000/objects/TestWidget) in the Infrahub UI -2. Create a new branch named **widget-3** -3. Create a new widget with the name of **widget3** and count of **3** -4. Add **widget3** to the **widgets** group -5. Create a [Proposed Change](http://localhost:8000/proposed-changes?branch=widget-3) -6. Navigate to the **Checks** tab -7. Once the **generator** CI check has completed, navigate to the **Data** tab -![generator ci/cd](../media/guides/generator_pc_1.png) -8. Click the **Refresh diff** button to see the **widget3** resources created by the generator -![data refresh](../media/guides/generator_pc_2.png) +:::success Real-World Example -:::success Examples +Want to see how Generators can be used in production? Read our blog post on [How to Turn Your Source of Truth into a Service Factory](https://www.opsmill.com/how-to-turn-your-source-of-truth-into-a-service-factory/). -- Want to read how Generators can be used to create a service catalog? See our blog post on [How to Turn Your Source of Truth into a Service Factory](https://www.opsmill.com/how-to-turn-your-source-of-truth-into-a-service-factory/). ::: From a0f26c63ebac61bc8f21f8504b1bff7ceb3a4444 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Thu, 24 Jul 2025 15:32:39 +0200 Subject: [PATCH 2/6] Vale please pass --- .vale/styles/spelling-exceptions.txt | 3 +++ docs/docs/guides/generator.mdx | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/.vale/styles/spelling-exceptions.txt b/.vale/styles/spelling-exceptions.txt index 5362414f0c..1531e7fc11 100644 --- a/.vale/styles/spelling-exceptions.txt +++ b/.vale/styles/spelling-exceptions.txt @@ -161,3 +161,6 @@ VSCode Slurp'it walkthrough Otternet +Widget's +IDE +AI diff --git a/docs/docs/guides/generator.mdx b/docs/docs/guides/generator.mdx index af0f6b9864..12753b74c1 100644 --- a/docs/docs/guides/generator.mdx +++ b/docs/docs/guides/generator.mdx @@ -18,7 +18,7 @@ By the end, you'll be able to: ## What you'll build -You'll build a simple generator that: +You'll build a basic generator that: 1. Reads information about Widget objects from the database 2. Creates a number of Resource objects based on each Widget's count property @@ -47,7 +47,7 @@ Before you begin, you'll need: ## Step 1: Setting up the data model -Since the default Infrahub schema doesn't include the object types we need for this example, we'll first create a simple schema with Widget and Resource nodes. +In this step, you'll create the data structures needed for your generator, including a schema, group, and sample objects that the generator will process. ### Create a new branch From 81db2e6f6add15e9a8cff818155ecd2c79cebc98 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Thu, 24 Jul 2025 15:34:37 +0200 Subject: [PATCH 3/6] Fine tune vale --- .vale/styles/Infrahub/sentence-case.yml | 2 ++ .vale/styles/spelling-exceptions.txt | 3 +-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.vale/styles/Infrahub/sentence-case.yml b/.vale/styles/Infrahub/sentence-case.yml index c878440854..9f0f1a3103 100644 --- a/.vale/styles/Infrahub/sentence-case.yml +++ b/.vale/styles/Infrahub/sentence-case.yml @@ -83,3 +83,5 @@ exceptions: - VS Code extensions - WebUI - Slurp'it + - IDE + - AI diff --git a/.vale/styles/spelling-exceptions.txt b/.vale/styles/spelling-exceptions.txt index 1531e7fc11..e8fb4f83b2 100644 --- a/.vale/styles/spelling-exceptions.txt +++ b/.vale/styles/spelling-exceptions.txt @@ -162,5 +162,4 @@ Slurp'it walkthrough Otternet Widget's -IDE -AI +widget's From cdbe288f06dd6af7918a7ead29906042dc094714 Mon Sep 17 00:00:00 2001 From: Baptiste Date: Fri, 25 Jul 2025 10:16:57 +0200 Subject: [PATCH 4/6] Remove redundant section --- docs/docs/guides/generator.mdx | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/docs/guides/generator.mdx b/docs/docs/guides/generator.mdx index 12753b74c1..576d32fe61 100644 --- a/docs/docs/guides/generator.mdx +++ b/docs/docs/guides/generator.mdx @@ -10,12 +10,6 @@ Generators in Infrahub allow you to automatically create or modify data based on In this guide, you'll learn how to develop a generator and integrate it into Infrahub. -By the end, you'll be able to: - -- Create a generator that reads data from Infrahub using GraphQL -- Generate new objects based on that data -- Run the generator both locally and through Infrahub's CI pipeline - ## What you'll build You'll build a basic generator that: From c164b85d37a111993aed768d650c542ab569ca1e Mon Sep 17 00:00:00 2001 From: Baptiste Date: Fri, 25 Jul 2025 17:07:44 +0200 Subject: [PATCH 5/6] Small issue in flow --- docs/docs/guides/generator copy.mdx | 534 ++++++++++++++++++++++++++++ 1 file changed, 534 insertions(+) create mode 100644 docs/docs/guides/generator copy.mdx diff --git a/docs/docs/guides/generator copy.mdx b/docs/docs/guides/generator copy.mdx new file mode 100644 index 0000000000..5b18306721 --- /dev/null +++ b/docs/docs/guides/generator copy.mdx @@ -0,0 +1,534 @@ +--- +title: How to Create a Generator +--- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# How to create a generator in Infrahub + +Generators in Infrahub allow you to automatically create or modify data based on existing information in your database. They are defined in [external repositories](../topics/repository) and can be developed and tested locally using [infrahubctl generator]($(base_url)infrahubctl/infrahubctl-generator). + +In this guide, you'll learn how to develop a generator and integrate it into Infrahub. + +By the end, you'll be able to: + +- Create a generator that reads data from Infrahub using GraphQL +- Generate new objects based on that data +- Run the generator both locally and through Infrahub's CI pipeline + +## What you'll build + +You'll build a basic generator that: + +1. Reads information about Widget objects from the database +2. Creates a number of Resource objects based on each Widget's count property +3. Automatically runs when changes to Widget objects are proposed + +## Steps overview + +1. Prepare the data model by creating a schema and objects +2. Create a GraphQL query to fetch data from Infrahub +3. Implement a Python generator that creates the data +4. Tie everything together in an `.infrahub.yml` file +5. Test the generator locally with infrahubctl +6. Deploy the generator to Infrahub +7. Validate that the generator works in Infrahub's CI pipeline + +## Prerequisites + +Before you begin, you'll need: + +- Basic knowledge of Infrahub, Python, GraphQL, YAML, and Git +- Familiarity with Generators and how they work in Infrahub (see [Generators topic](../topics/generator)) +- An Infrahub instance running locally or remotely +- A Git repository connected to Infrahub (see [Adding an External Repository guide](./repository)) +- [infrahubctl]($(base_url)infrahubctl/infrahubctl) installed and configured locally +- The repository cloned locally where you'll develop the generator + +## Step 1: Setting up the data model + +In this step, you'll create the data structures needed for your generator, including a schema, group, and sample objects that the generator will process. + +### Create a new branch + +:::warning Version Control & Branching + +We recommend creating a new branch in your Git repository for your generator development to avoid affecting the main branch. In this guide, we'll use a branch named `create-widget-generator`. + +```shell +git checkout -b create-widget-generator +``` + +::: + +### Create group + +:::info Object file or UI + +If you prefer a "as code" approach, use the object file method. Otherwise, you can create the group directly in the Infrahub UI. + +::: + + + + 1. Create a file named `groups.yml` in the `objects` directory of your repository: + + ```yaml title="objects/groups.yml" + --- + apiVersion: infrahub.app/v1 + kind: Object + spec: + kind: CoreStandardGroup + data: + - name: "widgets" + ``` + + 2. Update the `.infrahub.yml` file to include the new object file: + + ```yaml title=".infrahub.yml" + --- + objects: + - file_path: objects/groups.yml + ``` + + 3. Commit the changes to your repository: + + ```shell + git add objects/groups.yml .infrahub.yml + git commit -m "Add widgets group" + git push + ``` + + + + 1. Open the **Infrahub UI** in your browser + 2. Select the branch `create-widget-generator` + 3. Navigate to the **Groups** view (Object Management > Groups) + 4. Create a **Standard group** named `widgets` + + + +:::success + +To verify the group was created successfully, navigate to the Groups view in the Infrahub UI and check that the `widgets` group appears in the list. + +::: + +### Create and load schema + +:::warning Schema Creation + +This step is only necessary if you don't already have the schemas affected by the generator in your instance. + +::: + +1. Create a `widgets.yml` file in the `schemas` directory of your repository: + +```yaml title="schemas/widgets.yml" +# yaml-language-server: $schema=https://schema.infrahub.app/infrahub/schema/latest.json +--- +version: '1.0' + +nodes: + - name: Widget + namespace: Test + label: Widget + human_friendly_id: + - name__value + display_labels: + - name__value + attributes: + - name: name + kind: Text + unique: true + - name: count + kind: Number + + - name: Resource + namespace: Test + label: Resource + human_friendly_id: + - name__value + display_labels: + - name__value + attributes: + - name: name + kind: Text + unique: true + +``` + +:::info Schema Loading + +Choose the method you already use to manage schema changes in your Infrahub deployment. + +::: + + + + 2. Update the `.infrahub.yml` file to include the new schema file: + + ```yaml title=".infrahub.yml" + --- + schemas: + - schemas/widgets.yml + ``` + + 3. Commit and push the changes to your repository + + + + 2. Load the following schema using the [infrahubctl schema]($(base_url)infrahubctl/infrahubctl-schema) command. + + ```shell + infrahubctl schema load schemas/widgets.yml --branch=create-widget-generator + ``` + + ```shell + schema 'widgets.yml' loaded successfully + 1 schema processed in 8.453 seconds. + ``` + + + + +:::success + +You should see new entries for `Widget` and `Resource` in the left-hand side menu of the Infrahub UI. + +::: + +### Create widget objects + +Now that you have the schema loaded, you need to create objects that will be used by the generator: + +:::warning Object Creation + +This step is only necessary if you don't already have the objects affected by the generator in your instance. + +::: + +:::note Object Creation Method + +You can also use an object file to create the widgets, though it's less relevant here than for the group creation. + +::: + +1. Open the **Infrahub UI** in your browser +2. Select the branch `create-widget-generator` +3. Navigate to the **Widget** view +4. Create a new widget: + - Name: `widget1` + - Count: `1` + - Member of groups: `widgets` +5. Create a second widget: + - Name: `widget2` + - Count: `2` + - Member of groups: `widgets` + +## Step 2: Create the GraphQL query + +Next, create a GraphQL query that fetches the data your generator needs to process. + +### Create a query file + +Create a `widget_query.gql` file in the `queries` directory of your repository: + +```graphql title="queries/widget_query.gql" +query Widgets($name: String!) { + TestWidget(name__value: $name) { + edges { + node { + __typename + id + name { + value + } + count { + value + } + } + } + } +} +``` + +### Test the query + +To test the query you can use **Infrahub's GraphQL Sandbox**. + +:::note GraphQL Sandbox + +Access the sandbox by clicking your user icon in the bottom left corner and selecting **GraphQL Sandbox**. This tool allows you to run GraphQL queries and provides an interactive way to explore the schema. + +::: + +1. Copy the above GraphQL query in the main section + +2. In the **variables** section, add: + +```json +{ + "name": "widget1" +} +``` + +3. Click the **Execute** button, this should return a response like: + +```json +{ + "data": { + "TestWidget": { + "edges": [ + { + "node": { + "__typename": "TestWidget", + "id": "185526eb-2114-ce20-390a-c51aac78460a", + "name": { + "value": "widget1" + }, + "count": { + "value": 1 + } + } + } + ] + } + } +} +``` + +:::success + +You now have a working GraphQL query that retrieves the necessary data from Infrahub. You might want to keep the result of the query for later reference, as it will be useful when implementing the generator. + +::: + +## Step 3: Implement the generator + +Now create a Python class that implements your generator logic. The generator creates `TestResource` objects equal to the widget's count value. + +The class must: + +- Inherit from `InfrahubGenerator` +- Implement an async `generate()` method that processes data from your GraphQL query + +:::info + +If you aren't using `convert_query_response`, you can access the widget data directly from the `data` parameter passed to the `generate()` method. +If you're working with `protocols`, you can create `TestResource` objects using strict typing. + +::: + +1. Create a file named `widget_generator.py` in your `generators` directory: + +```python title="generators/widget_generator.py" +from infrahub_sdk.generator import InfrahubGenerator + +class WidgetGenerator(InfrahubGenerator): + async def generate(self, data: dict) -> None: + # Access the widget as an SDK object + widget = self.nodes[0] # or self.store.get(data["TestWidget"]["edges"][0]["node"]["id"]) + widget_name: str = widget.name.value + widget_count: int = widget.count.value + + # Create resources based on widget count + for count in range(1, widget_count + 1): + payload = { + "name": f"{widget_name.lower()}-{count}", + } + obj = await self.client.create(kind="TestResource", data=payload) + await obj.save(allow_upsert=True) + +``` + +## Step 4: Tie everything together in the .infrahub.yml file + +Now that you have your GraphQL query and Python generator, edit your [.infrahub.yml](../topics/infrahub-yml) file to tie everything together. + +1. Add the following configuration to the file `.infrahub.yml`: + +```yaml title=".infrahub.yml" +--- +generator_definitions: + - name: widget_generator + file_path: "generators/widget_generator.py" + targets: widgets + query: widget_query + convert_query_response: true + class_name: WidgetGenerator + parameters: + name: "name__value" + +queries: + - name: widget_query + file_path: "queries/widget_query.gql" +``` + +For a complete explanation of the `.infrahub.yml` file format, see the [infrahub.yml topic](../topics/infrahub-yml). + +2. Verify the configuration + +Check that your `.infrahub.yml` file is correctly formatted by listing available generators: + +```shell +infrahubctl generator --list +``` + +:::success + +If successful, you'll see output like: + +```shell +Generators defined in repository: 1 +widget_generator (generators/widget_generator.py::WidgetGenerator) Target: widgets +``` + +::: + +## Step 5: Test the generator locally + +Before deploying your generator to Infrahub, test it locally using the `infrahubctl` command-line tool. + +### Run the generator + +Run the generator on both of your widget objects: + +```shell +infrahubctl generator widget_generator --branch=create-widget-generator name=widget1 +infrahubctl generator widget_generator --branch=create-widget-generator name=widget2 +``` + +### Verify the results + +1. Select the branch `create-widget-generator` from the branch selector +2. Navigate to **Resource** objects +3. You should see: + - One resource object for `widget1` (named `widget1-1`) + - Two resource objects for `widget2` (named `widget2-1` and `widget2-2`) + +![Generated resource objects](../media/guides/generator_pc_3.png) + +:::warning + +You might want to cleanup this development data before merging your changes into the main branch. + +::: + +:::success + +Your generator worked as expected, it's now time to deploy it to Infrahub. + +::: + +## Step 6: Deploy to Infrahub + +Now that you've tested your generator, deploy it to Infrahub. + +### Verify repository structure + +Ensure your repository has the following structure: + +:::note Schemas & Objects + +Depending on your organization, you might also have `schemas` and `objects` directories in your repository. + +::: + +```shell +your-repository/ +├── .infrahub.yml +├── generators/ +│ └── widget_generator.py +└── queries/ + └── widget_query.gql +``` + +### Commit and push your code + +Upload your generator code to the repository: + +```shell +git add . +git commit -m "Add widget generator" +git push +``` + +### Confirm the generator is imported + +After merging your changes, confirm that the generator is imported correctly by checking the Infrahub UI. + +1. Open the **Infrahub UI** in your browser +2. Select the `main` branch +3. Navigate to the **Generators Definition view** (Actions > Generator Definitions) +4. You should see your `widget_generator` listed there + +If you don't see it verify the status of the repository in the **Repository view** and ensure the sync status is `synced`. + +### Proposed changes and merge + +Now merge your development branch `create-widget-generator` into `main`. + +:::note Merging Options + +We recommend using Infrahub's **Proposed Changes** feature to merge your changes. This allows you to review both data and code changes before merging. + +::: + +1. Create a **Proposed Change** + - Source branch: `create-widget-generator` + - Destination branch: `main` + - Name: `Add widget generator` +2. Navigate to the **Data** and **Files** tabs to review the changes +3. Back to the **Overview** tab, click the **Merge** button + +:::success + +Your generator is now deployed to Infrahub in the `main` branch and ready to be used. + +::: + +## Step 7: Run the generator in Infrahub's CI pipeline + +Now let's verify that your generator automatically runs when new widgets are created: + +1. Create a new branch named `add-widget-3` +2. Navigate to the **Widget** objects in the Infrahub UI +3. Create a new widget: + - Name: `widget3` + - Count: `3` + - Member of groups: `widgets` +4. Create a **Proposed Change** + - Source branch: `add-widget-3` + - Destination branch: `main` + - Name: `Add widget3` + +### Check generator results + +1. Navigate to the **Checks** tab of your proposed change +2. Wait for the **generator** CI check to complete +![Generator CI Check](../media/guides/generator_pc_1.png) +3. Navigate to the **Data** tab +4. Click the **Refresh diff** button to see the resources created by your generator +![Data Refresh showing generated resources](../media/guides/generator_pc_2.png) + +:::success + +You should see three new resources (`widget3-1`, `widget3-2`, and `widget3-3`) automatically created by your generator. + +::: + +## Next steps + +Now that you've created a basic generator, you can: + +- Create generators that handle a full object lifecycle, including creation, updates, and deletions +- Build idempotent generators that can be run multiple times without creating duplicate data +- Add unit tests to your generator to ensure it behaves as expected +- Harden your generator by adding error handling and logging + +:::success Real-World Example + +Want to see how Generators can be used in production? Read our blog post on [How to Turn Your Source of Truth into a Service Factory](https://www.opsmill.com/how-to-turn-your-source-of-truth-into-a-service-factory/). + +::: \ No newline at end of file From 711d46264872a71bf4c6eab1622dc7374f6d6eba Mon Sep 17 00:00:00 2001 From: Baptiste Date: Fri, 25 Jul 2025 17:15:44 +0200 Subject: [PATCH 6/6] Pushing the right file --- docs/docs/guides/generator copy.mdx | 534 ---------------------------- docs/docs/guides/generator.mdx | 4 +- 2 files changed, 2 insertions(+), 536 deletions(-) delete mode 100644 docs/docs/guides/generator copy.mdx diff --git a/docs/docs/guides/generator copy.mdx b/docs/docs/guides/generator copy.mdx deleted file mode 100644 index 5b18306721..0000000000 --- a/docs/docs/guides/generator copy.mdx +++ /dev/null @@ -1,534 +0,0 @@ ---- -title: How to Create a Generator ---- -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - -# How to create a generator in Infrahub - -Generators in Infrahub allow you to automatically create or modify data based on existing information in your database. They are defined in [external repositories](../topics/repository) and can be developed and tested locally using [infrahubctl generator]($(base_url)infrahubctl/infrahubctl-generator). - -In this guide, you'll learn how to develop a generator and integrate it into Infrahub. - -By the end, you'll be able to: - -- Create a generator that reads data from Infrahub using GraphQL -- Generate new objects based on that data -- Run the generator both locally and through Infrahub's CI pipeline - -## What you'll build - -You'll build a basic generator that: - -1. Reads information about Widget objects from the database -2. Creates a number of Resource objects based on each Widget's count property -3. Automatically runs when changes to Widget objects are proposed - -## Steps overview - -1. Prepare the data model by creating a schema and objects -2. Create a GraphQL query to fetch data from Infrahub -3. Implement a Python generator that creates the data -4. Tie everything together in an `.infrahub.yml` file -5. Test the generator locally with infrahubctl -6. Deploy the generator to Infrahub -7. Validate that the generator works in Infrahub's CI pipeline - -## Prerequisites - -Before you begin, you'll need: - -- Basic knowledge of Infrahub, Python, GraphQL, YAML, and Git -- Familiarity with Generators and how they work in Infrahub (see [Generators topic](../topics/generator)) -- An Infrahub instance running locally or remotely -- A Git repository connected to Infrahub (see [Adding an External Repository guide](./repository)) -- [infrahubctl]($(base_url)infrahubctl/infrahubctl) installed and configured locally -- The repository cloned locally where you'll develop the generator - -## Step 1: Setting up the data model - -In this step, you'll create the data structures needed for your generator, including a schema, group, and sample objects that the generator will process. - -### Create a new branch - -:::warning Version Control & Branching - -We recommend creating a new branch in your Git repository for your generator development to avoid affecting the main branch. In this guide, we'll use a branch named `create-widget-generator`. - -```shell -git checkout -b create-widget-generator -``` - -::: - -### Create group - -:::info Object file or UI - -If you prefer a "as code" approach, use the object file method. Otherwise, you can create the group directly in the Infrahub UI. - -::: - - - - 1. Create a file named `groups.yml` in the `objects` directory of your repository: - - ```yaml title="objects/groups.yml" - --- - apiVersion: infrahub.app/v1 - kind: Object - spec: - kind: CoreStandardGroup - data: - - name: "widgets" - ``` - - 2. Update the `.infrahub.yml` file to include the new object file: - - ```yaml title=".infrahub.yml" - --- - objects: - - file_path: objects/groups.yml - ``` - - 3. Commit the changes to your repository: - - ```shell - git add objects/groups.yml .infrahub.yml - git commit -m "Add widgets group" - git push - ``` - - - - 1. Open the **Infrahub UI** in your browser - 2. Select the branch `create-widget-generator` - 3. Navigate to the **Groups** view (Object Management > Groups) - 4. Create a **Standard group** named `widgets` - - - -:::success - -To verify the group was created successfully, navigate to the Groups view in the Infrahub UI and check that the `widgets` group appears in the list. - -::: - -### Create and load schema - -:::warning Schema Creation - -This step is only necessary if you don't already have the schemas affected by the generator in your instance. - -::: - -1. Create a `widgets.yml` file in the `schemas` directory of your repository: - -```yaml title="schemas/widgets.yml" -# yaml-language-server: $schema=https://schema.infrahub.app/infrahub/schema/latest.json ---- -version: '1.0' - -nodes: - - name: Widget - namespace: Test - label: Widget - human_friendly_id: - - name__value - display_labels: - - name__value - attributes: - - name: name - kind: Text - unique: true - - name: count - kind: Number - - - name: Resource - namespace: Test - label: Resource - human_friendly_id: - - name__value - display_labels: - - name__value - attributes: - - name: name - kind: Text - unique: true - -``` - -:::info Schema Loading - -Choose the method you already use to manage schema changes in your Infrahub deployment. - -::: - - - - 2. Update the `.infrahub.yml` file to include the new schema file: - - ```yaml title=".infrahub.yml" - --- - schemas: - - schemas/widgets.yml - ``` - - 3. Commit and push the changes to your repository - - - - 2. Load the following schema using the [infrahubctl schema]($(base_url)infrahubctl/infrahubctl-schema) command. - - ```shell - infrahubctl schema load schemas/widgets.yml --branch=create-widget-generator - ``` - - ```shell - schema 'widgets.yml' loaded successfully - 1 schema processed in 8.453 seconds. - ``` - - - - -:::success - -You should see new entries for `Widget` and `Resource` in the left-hand side menu of the Infrahub UI. - -::: - -### Create widget objects - -Now that you have the schema loaded, you need to create objects that will be used by the generator: - -:::warning Object Creation - -This step is only necessary if you don't already have the objects affected by the generator in your instance. - -::: - -:::note Object Creation Method - -You can also use an object file to create the widgets, though it's less relevant here than for the group creation. - -::: - -1. Open the **Infrahub UI** in your browser -2. Select the branch `create-widget-generator` -3. Navigate to the **Widget** view -4. Create a new widget: - - Name: `widget1` - - Count: `1` - - Member of groups: `widgets` -5. Create a second widget: - - Name: `widget2` - - Count: `2` - - Member of groups: `widgets` - -## Step 2: Create the GraphQL query - -Next, create a GraphQL query that fetches the data your generator needs to process. - -### Create a query file - -Create a `widget_query.gql` file in the `queries` directory of your repository: - -```graphql title="queries/widget_query.gql" -query Widgets($name: String!) { - TestWidget(name__value: $name) { - edges { - node { - __typename - id - name { - value - } - count { - value - } - } - } - } -} -``` - -### Test the query - -To test the query you can use **Infrahub's GraphQL Sandbox**. - -:::note GraphQL Sandbox - -Access the sandbox by clicking your user icon in the bottom left corner and selecting **GraphQL Sandbox**. This tool allows you to run GraphQL queries and provides an interactive way to explore the schema. - -::: - -1. Copy the above GraphQL query in the main section - -2. In the **variables** section, add: - -```json -{ - "name": "widget1" -} -``` - -3. Click the **Execute** button, this should return a response like: - -```json -{ - "data": { - "TestWidget": { - "edges": [ - { - "node": { - "__typename": "TestWidget", - "id": "185526eb-2114-ce20-390a-c51aac78460a", - "name": { - "value": "widget1" - }, - "count": { - "value": 1 - } - } - } - ] - } - } -} -``` - -:::success - -You now have a working GraphQL query that retrieves the necessary data from Infrahub. You might want to keep the result of the query for later reference, as it will be useful when implementing the generator. - -::: - -## Step 3: Implement the generator - -Now create a Python class that implements your generator logic. The generator creates `TestResource` objects equal to the widget's count value. - -The class must: - -- Inherit from `InfrahubGenerator` -- Implement an async `generate()` method that processes data from your GraphQL query - -:::info - -If you aren't using `convert_query_response`, you can access the widget data directly from the `data` parameter passed to the `generate()` method. -If you're working with `protocols`, you can create `TestResource` objects using strict typing. - -::: - -1. Create a file named `widget_generator.py` in your `generators` directory: - -```python title="generators/widget_generator.py" -from infrahub_sdk.generator import InfrahubGenerator - -class WidgetGenerator(InfrahubGenerator): - async def generate(self, data: dict) -> None: - # Access the widget as an SDK object - widget = self.nodes[0] # or self.store.get(data["TestWidget"]["edges"][0]["node"]["id"]) - widget_name: str = widget.name.value - widget_count: int = widget.count.value - - # Create resources based on widget count - for count in range(1, widget_count + 1): - payload = { - "name": f"{widget_name.lower()}-{count}", - } - obj = await self.client.create(kind="TestResource", data=payload) - await obj.save(allow_upsert=True) - -``` - -## Step 4: Tie everything together in the .infrahub.yml file - -Now that you have your GraphQL query and Python generator, edit your [.infrahub.yml](../topics/infrahub-yml) file to tie everything together. - -1. Add the following configuration to the file `.infrahub.yml`: - -```yaml title=".infrahub.yml" ---- -generator_definitions: - - name: widget_generator - file_path: "generators/widget_generator.py" - targets: widgets - query: widget_query - convert_query_response: true - class_name: WidgetGenerator - parameters: - name: "name__value" - -queries: - - name: widget_query - file_path: "queries/widget_query.gql" -``` - -For a complete explanation of the `.infrahub.yml` file format, see the [infrahub.yml topic](../topics/infrahub-yml). - -2. Verify the configuration - -Check that your `.infrahub.yml` file is correctly formatted by listing available generators: - -```shell -infrahubctl generator --list -``` - -:::success - -If successful, you'll see output like: - -```shell -Generators defined in repository: 1 -widget_generator (generators/widget_generator.py::WidgetGenerator) Target: widgets -``` - -::: - -## Step 5: Test the generator locally - -Before deploying your generator to Infrahub, test it locally using the `infrahubctl` command-line tool. - -### Run the generator - -Run the generator on both of your widget objects: - -```shell -infrahubctl generator widget_generator --branch=create-widget-generator name=widget1 -infrahubctl generator widget_generator --branch=create-widget-generator name=widget2 -``` - -### Verify the results - -1. Select the branch `create-widget-generator` from the branch selector -2. Navigate to **Resource** objects -3. You should see: - - One resource object for `widget1` (named `widget1-1`) - - Two resource objects for `widget2` (named `widget2-1` and `widget2-2`) - -![Generated resource objects](../media/guides/generator_pc_3.png) - -:::warning - -You might want to cleanup this development data before merging your changes into the main branch. - -::: - -:::success - -Your generator worked as expected, it's now time to deploy it to Infrahub. - -::: - -## Step 6: Deploy to Infrahub - -Now that you've tested your generator, deploy it to Infrahub. - -### Verify repository structure - -Ensure your repository has the following structure: - -:::note Schemas & Objects - -Depending on your organization, you might also have `schemas` and `objects` directories in your repository. - -::: - -```shell -your-repository/ -├── .infrahub.yml -├── generators/ -│ └── widget_generator.py -└── queries/ - └── widget_query.gql -``` - -### Commit and push your code - -Upload your generator code to the repository: - -```shell -git add . -git commit -m "Add widget generator" -git push -``` - -### Confirm the generator is imported - -After merging your changes, confirm that the generator is imported correctly by checking the Infrahub UI. - -1. Open the **Infrahub UI** in your browser -2. Select the `main` branch -3. Navigate to the **Generators Definition view** (Actions > Generator Definitions) -4. You should see your `widget_generator` listed there - -If you don't see it verify the status of the repository in the **Repository view** and ensure the sync status is `synced`. - -### Proposed changes and merge - -Now merge your development branch `create-widget-generator` into `main`. - -:::note Merging Options - -We recommend using Infrahub's **Proposed Changes** feature to merge your changes. This allows you to review both data and code changes before merging. - -::: - -1. Create a **Proposed Change** - - Source branch: `create-widget-generator` - - Destination branch: `main` - - Name: `Add widget generator` -2. Navigate to the **Data** and **Files** tabs to review the changes -3. Back to the **Overview** tab, click the **Merge** button - -:::success - -Your generator is now deployed to Infrahub in the `main` branch and ready to be used. - -::: - -## Step 7: Run the generator in Infrahub's CI pipeline - -Now let's verify that your generator automatically runs when new widgets are created: - -1. Create a new branch named `add-widget-3` -2. Navigate to the **Widget** objects in the Infrahub UI -3. Create a new widget: - - Name: `widget3` - - Count: `3` - - Member of groups: `widgets` -4. Create a **Proposed Change** - - Source branch: `add-widget-3` - - Destination branch: `main` - - Name: `Add widget3` - -### Check generator results - -1. Navigate to the **Checks** tab of your proposed change -2. Wait for the **generator** CI check to complete -![Generator CI Check](../media/guides/generator_pc_1.png) -3. Navigate to the **Data** tab -4. Click the **Refresh diff** button to see the resources created by your generator -![Data Refresh showing generated resources](../media/guides/generator_pc_2.png) - -:::success - -You should see three new resources (`widget3-1`, `widget3-2`, and `widget3-3`) automatically created by your generator. - -::: - -## Next steps - -Now that you've created a basic generator, you can: - -- Create generators that handle a full object lifecycle, including creation, updates, and deletions -- Build idempotent generators that can be run multiple times without creating duplicate data -- Add unit tests to your generator to ensure it behaves as expected -- Harden your generator by adding error handling and logging - -:::success Real-World Example - -Want to see how Generators can be used in production? Read our blog post on [How to Turn Your Source of Truth into a Service Factory](https://www.opsmill.com/how-to-turn-your-source-of-truth-into-a-service-factory/). - -::: \ No newline at end of file diff --git a/docs/docs/guides/generator.mdx b/docs/docs/guides/generator.mdx index 576d32fe61..4715e61254 100644 --- a/docs/docs/guides/generator.mdx +++ b/docs/docs/guides/generator.mdx @@ -450,10 +450,10 @@ git push ### Confirm the generator is imported -After merging your changes, confirm that the generator is imported correctly by checking the Infrahub UI. +After pushing your changes, confirm that the generator is imported correctly by checking the Infrahub UI. 1. Open the **Infrahub UI** in your browser -2. Select the `main` branch +2. Select the `create-widget-generator` branch 3. Navigate to the **Generators Definition view** (Actions > Generator Definitions) 4. You should see your `widget_generator` listed there