-
Couldn't load subscription status.
- Fork 25
Docs: Data Contracts Section fixed and Added in 1.10 and 1.11 #510
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
96f8957
Docs: Note Addition for Connection Details
b6772ca
Docs: Data Contract Fixes
99554ed
Update content/v1.10.x/how-to-guides/data-contracts/create.md
RounakDhillon cb770a7
Update create.md
RounakDhillon 79d03e4
Update content/v1.11.x-SNAPSHOT/how-to-guides/data-contracts/spec.md
RounakDhillon 4887d94
Update spec.md
RounakDhillon 23b3b10
Update content/v1.11.x-SNAPSHOT/how-to-guides/data-contracts/spec.md
RounakDhillon 09a5bf9
Update title in data contracts specification guide
RounakDhillon 64e24c8
Fix title formatting in data contracts guide
RounakDhillon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| --- | ||
| title: Creating Data Contracts | OpenMetadata Data Contracts Guide | ||
| description: Follow this guide to create data contracts. Define schema, add business rules, and run quality tests to ensure consistent, reliable table data. | ||
| slug: /how-to-guides/data-contracts/create | ||
| --- | ||
|
|
||
| Data contracts are currently available for tables ingested into OpenMetadata. The following guide shows how a data contract was created for a snowflake.DEMO_STAGE.JAFFLE_SHOP.CUSTOMERS table. | ||
|
|
||
| To create a Data Contract for a Table in OpenMetadata: | ||
| - Go to the Table's page, select **Contract**, then **+ Add Contract** | ||
|
|
||
| {% image | ||
| src="/images/v1.10/how-to-guides/contracts/create1.png" | ||
| alt="Adding a Data Contract to an OpenMetadata Table" | ||
| caption="Adding a Data Contract to an OpenMetadata Table" | ||
| /%} | ||
|
|
||
| - In *Contract Details*, be sure to give your new data contract a name. Optionally, you can assign Owners and provide a description of your data contract. Then select **Schema**. | ||
|
|
||
| {% image | ||
| src="/images/v1.10/how-to-guides/contracts/create2.png" | ||
| alt="Adding Contract Details" | ||
| caption="Adding a Contract Details to an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Select the columns of your table that you would like to add to your new data contract, or select all columns by checking the box at the top right. Then select **Semantics** | ||
|
|
||
| {% image | ||
| src="/images/v1.10/how-to-guides/contracts/create3.png" | ||
| alt="Selecting a schema for an OpenMetadata Data Contract" | ||
| caption="Selecting a schema for an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Add the business rules you would like to enforce in *Semantics*. For OpenMetadata Tables, rules can be created for: | ||
| - Service | ||
| - Owners | ||
| - Display Name | ||
| - Name | ||
| - Description | ||
| - Tags | ||
| - Domain | ||
| - Data Product | ||
| - Tier | ||
|
|
||
| {% image | ||
| src="/images/v1.10/how-to-guides/contracts/create4.png" | ||
| alt="Adding rules to an OpenMetadata Data Contract" | ||
| caption="Adding rules to an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Once a rule is created, you can **+ Add New Rule** to create more, or select **Quality** | ||
|
|
||
| {% image | ||
| src="/images/v1.10/how-to-guides/contracts/create5.png" | ||
| alt="Adding more rules to an OpenMetadata Data Contract" | ||
| caption="Adding more rules to an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Select **+ Add Test** to add a [Data Quality Test](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/test) or tests to your new contract, then select **Save** | ||
|
|
||
| {% image | ||
| src="/images/v1.10/how-to-guides/contracts/create6.png" | ||
| alt="Adding data quality tests to an OpenMetadata Data Contract" | ||
| caption="Adding data quality tests to an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Once your new data contract has been created successfully, you can run it by selecting **> Run now** | ||
|
|
||
| {% image | ||
| src="/images/v1.10/how-to-guides/contracts/create7.png" | ||
| alt="Running an OpenMetadata Data Contract" | ||
| caption="Running an OpenMetadata Data Contract" | ||
| /%} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| --- | ||
| title: Data Contracts | OpenMetadata Data Contracts | ||
| description: Step-by-step guidance for data contracts using OpenMetadata. Learn key actions, features, and best practices. | ||
| slug: /how-to-guides/data-contracts | ||
| --- | ||
|
|
||
| # Overview of Data Contracts | ||
|
|
||
| OpenMetadata is a centralized, active metadata repository where all your data resides. In any organization, it is important to get the consumers utilizing data assets aligned with asset producers. As data platform adoption increases in an organization, a data mesh architecture with OpenMetadata Data Contracts can produce automated schema, semantic, and data quality validations that enforce data quality and SLAs across departments and domains. | ||
|
|
||
| Contracts in OpenMetadata are now available for tables. With OpenMetadata Data Contracts, you can set the expected schema structure of an asset, define its semantics (like ensuring that a table has a description field or an [owner assigned](https://docs.open-metadata.org/latest/how-to-guides/guide-for-data-users/data-ownership)), assign data quality tests, and track a data contract's execution history over time. | ||
|
|
||
| Watch the video to learn more about OpenMetadata Data Contracts. | ||
|
|
||
| {% ossContent %} | ||
|
|
||
| {% youtube videoId="9CAy_kNvenA" start="0:00" end="4:24" width="800px" height="450px" /%} | ||
|
|
||
| {% /ossContent %} | ||
|
|
||
| {% collateContent %} | ||
|
|
||
| {% youtube videoId="thLYeMx2sxs" start="0:00" end="4:24" width="800px" height="450px" /%} | ||
|
|
||
| {% /collateContent %} | ||
|
|
||
| Watch a demo of Data Contracts from our [OpenMetadata Community Meetup](https://www.meetup.com/openmetadata-meetup-group/) | ||
|
|
||
| {% ossContent %} | ||
|
|
||
| {% youtube videoId="sz2XUEQHp08" start="0:00" end="29:10" width="800px" height="450px" /%} | ||
|
|
||
| {% /ossContent %} | ||
|
|
||
| {% collateContent %} | ||
|
|
||
| {% youtube videoId="H4z6exr45X4" start="0:00" end="02:08" width="800px" height="450px" /%} | ||
|
|
||
| {% /collateContent %} | ||
|
|
||
| {%inlineCalloutContainer%} | ||
| {%inlineCallout | ||
| color="violet-70" | ||
| bold="OpenMetadata Data Contract Specification" | ||
| icon="puzzle" | ||
| href="/how-to-guides/data-contracts/spec"%} | ||
| Overview of JSON Schema for DataContract entity | ||
| {%/inlineCallout%} | ||
| {%inlineCallout | ||
| color="violet-70" | ||
| bold="Creating a Data Contract" | ||
| icon="MdConnectWithoutContact" | ||
| href="/how-to-guides/data-contracts/create"%} | ||
| Create data contracts directly from the OpenMetadata UI! | ||
| {%/inlineCallout%} | ||
| {%/inlineCalloutContainer%} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| --- | ||
| title: Data Contract Specification | OpenMetadata Data Contracts Guide | ||
| description: Create open-source data contracts directly in the OpenMetadata UI | ||
| slug: /how-to-guides/data-contracts/spec | ||
| --- | ||
|
|
||
| # Introduction | ||
|
|
||
| Data contracts formalize an agreement between data producers and consumers about what to expect from a data asset’s data. They capture the structure, semantics, quality, and SLAs of data in a machine-readable way, similar to an API contract but for data. In essence, a Data Contract is enforceable in the data ecosystem to bring standardization, control, and reliability. | ||
|
|
||
| OpenMetadata, as a metadata management platform, integrates this concept by introducing a Data Contract entity defined via JSON Schema. This allows OpenMetadata admins and [data product](https://docs.open-metadata.org/latest/how-to-guides/data-governance/domains-&-data-products#data-products) owners to attach a contract to tables in OpenMetadata, codifying expectations in a structured format. The contract can then be enforced or validated using OpenMetadata’s existing metadata and data quality frameworks. The goal is to have contextually rich, high-quality, well-governed data that is trustworthy. Data contracts achieve this by making data expectations explicit and automating their enforcement. | ||
|
|
||
| # Data Contract Entity Schema Design | ||
|
|
||
| The JSON Schema definition for Data Contract entities in OpenMetadata defines the contract’s structure and allowed fields. The contract covers seven main categories of expectations: | ||
|
|
||
| 1. [Schema](#schema) | ||
| 2. [Semantics](#semantics) | ||
| 3. [Security](#security) | ||
| 4. [Business Assertions (data quality)](#quality) | ||
| 5. [SLA](#sla) | ||
| 6. [Terms of Use](#terms-of-use) | ||
| 7. [Status](#status) | ||
|
|
||
| We also include an SLA section for service-level agreements and an Ownership field for accountability. Each Data Contract is designed to represent one single data asset (dataset, topic, model, etc.) in a well-structured, templated format. Data contracts are currently available for Table asset types. | ||
|
|
||
| The JSON Schema for the Data Contract entity can be found [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/dataContract.json) | ||
|
|
||
| # Data Contract Sections | ||
| ## Schema | ||
|
|
||
| This is where the expected structural schema of the data asset is defined. It includes a list of fields (for a table, these are columns) each with name and data type. This captures the contractual schema that producers and consumers agreed on, which usually is a subset of the available fields on an asset. | ||
|
|
||
| ## Semantics | ||
|
|
||
| Business meaning and documentation requirements are defined in a contract's Semantics section. For example, one can enforce that a data asset must have a description, owner or domain. | ||
|
|
||
| These rules complement the formal tests in the quality section, acting as documentation of business expectations. This section ensures the contract isn’t just about technical schema, but also carries business context. | ||
|
|
||
| ## Security | ||
|
|
||
| Data security and access expectations are defined in this section. This can reference an access policy ID or name that should govern this data, or a required classification label. | ||
|
|
||
| In practice, this means the contract might require the data asset to be tagged as `PII` or `Confidential` if appropriate, and that only certain roles can access it (through an associated policy). | ||
|
|
||
| ## Quality (Assertions) | ||
|
|
||
| Data quality tests and assertions required by the contract are here. | ||
|
|
||
| Built on top of the native Data Quality features in OpenMetadata, this section allows defining specific tests that must pass for the data to be considered compliant with the contract. Tests can be at the column level (e.g., a column must be non-null) or table level (e.g., row count must be above a threshold), and can be managed from the Data Contract UI itself. | ||
|
|
||
| ## SLA | ||
|
|
||
| Service-Level Agreements related to the data’s timeliness and lifecycle are captured in this section. | ||
| This includes: | ||
| Refresh Frequency: how often the data is expected to be updated or refreshed (e.g., daily, weekly,...). | ||
| Max Latency: the maximum allowed delay between data generation and when it’s available to consumers (e.g., data may be up to 4 hours old at most, or one day for typical daily batch ETLs). | ||
| Availability Time: the time by which daily or periodic data should be available (e.g., “09:00 UTC” daily data drop). | ||
| Retention: how long the data is kept accessible (if applicable). | ||
|
|
||
| Including them in the contract means producers commit to certain delivery timelines, and consumers know what availability to expect. | ||
|
|
||
| ## Terms of Use | ||
|
|
||
| This section captures the allowed and disallowed uses of the data asset, as well as any compliance or regulatory requirements. This can include: | ||
| - Allowed Uses: Describes what the data can be used for (e.g., internal analytics, reporting). | ||
| - Disallowed Uses: Specifies prohibited uses (e.g., no sharing with third parties, or no training AI models). | ||
| - Compliance Requirements: Any legal or regulatory obligations (e.g., GDPR, HIPAA). | ||
|
|
||
| ## Status | ||
|
|
||
| A status field indicates whether the contract is active, draft, or currently violated. For instance, when first created, a contract is in `DRAFT` when it is not yet enforced or not fully implemented by the data producer. Once a data contract is published, it becomes `ACTIVE`. If a violation occurs (e.g., a test fails or schema deviates), the contract's status is `VIOLATED`. | ||
|
|
||
| Data Contracts have approval workflows when changes are made, similar to Glossaries. This ensures that any modifications to the contract (like adding new quality tests or changing schema expectations) go through a review and approval process. | ||
|
|
||
| # Applying Contracts to Tables | ||
|
|
||
| Below is an example of a data contract for a warehouse.sales.orders table. | ||
|
|
||
| ## Data Contract YAML Example | ||
|
|
||
| This is an example YAML of a Data Contract applied to a table in OpenMetadata. Note that while OpenMetadata brings full UI support for creating and managing Data Contracts, you can still use the API to manage them programmatically. | ||
|
|
||
| ``` | ||
| name: Customers DC | ||
| status: Active | ||
| entity: | ||
| id: 8beb4301-8302-4791-9944-2897e7614a1a | ||
| type: table | ||
| href: https://example.com/v1/tables/8beb4301-8302-4791-9944-2897e7614a1a | ||
| schema: | ||
| - name: customer_id | ||
| dataType: INT | ||
| dataLength: 1 | ||
| dataTypeDisplay: integer | ||
| description: New ID from Collate UI | ||
| fullyQualifiedName: red.dev.dbt_jaffle.customers.customer_id | ||
| tags: [] | ||
| constraint: 'NULL' | ||
| children: [] | ||
| - name: first_name | ||
| dataType: VARCHAR | ||
| dataLength: 20 | ||
| dataTypeDisplay: character varying(20) | ||
| fullyQualifiedName: red.dev.dbt_jaffle.customers.first_name | ||
| tags: | ||
| - tagFQN: General.Person | ||
| name: Person | ||
| description: >- | ||
| A full person name, which can include first names, middle names or | ||
| initials, and last names. | ||
| source: Classification | ||
| labelType: Generated | ||
| state: Suggested | ||
| - tagFQN: PII.Sensitive | ||
| name: Sensitive | ||
| description: >- | ||
| PII which if lost, compromised, or disclosed without authorization, | ||
| could result in substantial harm, embarrassment, inconvenience, or | ||
| unfairness to an individual. | ||
| source: Classification | ||
| labelType: Generated | ||
| state: Suggested | ||
| constraint: 'NULL' | ||
| children: [] | ||
| semantics: | ||
| - name: Owners is set | ||
| description: Ownership is mandatory | ||
| rule: >- | ||
| {"and":[{"some":[{"var":"owners"},{"!=":[{"var":"fullyQualifiedName"},null]}]}]} | ||
| qualityExpectations: | ||
| - id: 1efbda53-063d-4611-8f69-402f4490a503 | ||
| type: testCase | ||
| name: customer rows | ||
| - id: 707a43f9-d1d1-4fb8-96da-7bb428429f87 | ||
| type: testCase | ||
| name: relationships_orders_customer_id__customer_id__ref_customers_ | ||
| description: '' | ||
| owners: [] | ||
| reviewers: [] | ||
| ``` | ||
|
|
||
| {%inlineCallout | ||
| color="violet-70" | ||
| bold="Creating Data Contracts" | ||
| icon="MdArrowForward" | ||
| href="/how-to-guides/data-contracts/create"%} | ||
| Create Data Contracts in the OpenMetadata UI. | ||
| {%/inlineCallout%} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
73 changes: 73 additions & 0 deletions
73
content/v1.11.x-SNAPSHOT/how-to-guides/data-contracts/create.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| --- | ||
| title: Creating Data Contracts | OpenMetadata Data Contracts Guide | ||
| description: Follow this guide to create data contracts. Define schema, add business rules, and run quality tests to ensure consistent, reliable table data. | ||
| slug: /how-to-guides/data-contracts/create | ||
| --- | ||
|
|
||
| Data contracts are currently available for tables ingested into OpenMetadata. The following guide shows how a data contract was created for a snowflake.DEMO_STAGE.JAFFLE_SHOP.CUSTOMERS table. | ||
|
|
||
| To create a Data Contract for a Table in OpenMetadata: | ||
| - Go to the Table's page, select **Contract**, then **+ Add Contract** | ||
|
|
||
| {% image | ||
| src="/images/v1.11/how-to-guides/contracts/create1.png" | ||
| alt="Adding a Data Contract to an OpenMetadata Table" | ||
| caption="Adding a Data Contract to an OpenMetadata Table" | ||
| /%} | ||
|
|
||
| - In *Contract Details*, be sure to give your new data contract a name. Optionally, you can assign Owners and provide a description of your data contract. Then select **Schema**. | ||
|
|
||
| {% image | ||
| src="/images/v1.11/how-to-guides/contracts/create2.png" | ||
| alt="Adding Contract Details" | ||
| caption="Adding a Contract Details to an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Select the columns of your table that you would like to add to your new data contract, or select all columns by checking the box at the top right. Then select **Semantics** | ||
|
|
||
| {% image | ||
| src="/images/v1.11/how-to-guides/contracts/create3.png" | ||
| alt="Selecting a schema for an OpenMetadata Data Contract" | ||
| caption="Selecting a schema for an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Add the business rules you would like to enforce in *Semantics*. For OpenMetadata Tables, rules can be created for: | ||
| - Service | ||
| - Owners | ||
| - Display Name | ||
| - Name | ||
| - Description | ||
| - Tags | ||
| - Domain | ||
| - Data Product | ||
| - Tier | ||
|
|
||
| {% image | ||
| src="/images/v1.11/how-to-guides/contracts/create4.png" | ||
| alt="Adding rules to an OpenMetadata Data Contract" | ||
| caption="Adding rules to an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Once a rule is created, you can **+ Add New Rule** to create more, or select **Quality** | ||
|
|
||
| {% image | ||
| src="/images/v1.11/how-to-guides/contracts/create5.png" | ||
| alt="Adding more rules to an OpenMetadata Data Contract" | ||
| caption="Adding more rules to an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Select **+ Add Test** to add a [Data Quality Test](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/test) or tests to your new contract, then select **Save** | ||
|
|
||
| {% image | ||
| src="/images/v1.11/how-to-guides/contracts/create6.png" | ||
| alt="Adding data quality tests to an OpenMetadata Data Contract" | ||
| caption="Adding data quality tests to an OpenMetadata Data Contract" | ||
| /%} | ||
|
|
||
| - Once your new data contract has been created successfully, you can run it by selecting **> Run now** | ||
|
|
||
| {% image | ||
| src="/images/v1.11/how-to-guides/contracts/create7.png" | ||
| alt="Running an OpenMetadata Data Contract" | ||
| caption="Running an OpenMetadata Data Contract" | ||
| /%} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.