Skip to content

Commit dfad586

Browse files
RounakDhillon“Rounakayush-shahCopilot
authored
Docs: Data Contracts Section fixed and Added in 1.10 and 1.11 (#510)
* Docs: Note Addition for Connection Details * Docs: Data Contract Fixes * Update content/v1.10.x/how-to-guides/data-contracts/create.md Co-authored-by: Ayush Shah <[email protected]> * Update create.md * Update content/v1.11.x-SNAPSHOT/how-to-guides/data-contracts/spec.md Co-authored-by: Copilot <[email protected]> * Update spec.md * Update content/v1.11.x-SNAPSHOT/how-to-guides/data-contracts/spec.md Co-authored-by: Copilot <[email protected]> * Update title in data contracts specification guide * Fix title formatting in data contracts guide --------- Co-authored-by: “Rounak <“[email protected]”> Co-authored-by: Ayush Shah <[email protected]> Co-authored-by: Copilot <[email protected]>
1 parent c4b9ff6 commit dfad586

File tree

25 files changed

+592
-1
lines changed

25 files changed

+592
-1
lines changed

‎content/v1.10.x/how-to-guides/data-contracts/create.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
title: Creating Data Contracts | OpenMetadata Data Contracts Guide
3+
description: Follow this guide to create data contracts. Define schema, add business rules, and run quality tests to ensure consistent, reliable table data.
4+
slug: /how-to-guides/data-contracts/create
5+
---
6+
7+
Data contracts are currently available for tables ingested into OpenMetadata. The following guide shows how a data contract was created for a snowflake.DEMO_STAGE.JAFFLE_SHOP.CUSTOMERS table.
8+
9+
To create a Data Contract for a Table in OpenMetadata:
10+
- Go to the Table's page, select **Contract**, then **+ Add Contract**
11+
12+
{% image
13+
src="/images/v1.10/how-to-guides/contracts/create1.png"
14+
alt="Adding a Data Contract to an OpenMetadata Table"
15+
caption="Adding a Data Contract to an OpenMetadata Table"
16+
/%}
17+
18+
- In *Contract Details*, be sure to give your new data contract a name. Optionally, you can assign Owners and provide a description of your data contract. Then select **Schema**.
19+
20+
{% image
21+
src="/images/v1.10/how-to-guides/contracts/create2.png"
22+
alt="Adding Contract Details"
23+
caption="Adding a Contract Details to an OpenMetadata Data Contract"
24+
/%}
25+
26+
- Select the columns of your table that you would like to add to your new data contract, or select all columns by checking the box at the top right. Then select **Semantics**
27+
28+
{% image
29+
src="/images/v1.10/how-to-guides/contracts/create3.png"
30+
alt="Selecting a schema for an OpenMetadata Data Contract"
31+
caption="Selecting a schema for an OpenMetadata Data Contract"
32+
/%}
33+
34+
- Add the business rules you would like to enforce in *Semantics*. For OpenMetadata Tables, rules can be created for:
35+
- Service
36+
- Owners
37+
- Display Name
38+
- Name
39+
- Description
40+
- Tags
41+
- Domain
42+
- Data Product
43+
- Tier
44+
45+
{% image
46+
src="/images/v1.10/how-to-guides/contracts/create4.png"
47+
alt="Adding rules to an OpenMetadata Data Contract"
48+
caption="Adding rules to an OpenMetadata Data Contract"
49+
/%}
50+
51+
- Once a rule is created, you can **+ Add New Rule** to create more, or select **Quality**
52+
53+
{% image
54+
src="/images/v1.10/how-to-guides/contracts/create5.png"
55+
alt="Adding more rules to an OpenMetadata Data Contract"
56+
caption="Adding more rules to an OpenMetadata Data Contract"
57+
/%}
58+
59+
- Select **+ Add Test** to add a [Data Quality Test](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/test) or tests to your new contract, then select **Save**
60+
61+
{% image
62+
src="/images/v1.10/how-to-guides/contracts/create6.png"
63+
alt="Adding data quality tests to an OpenMetadata Data Contract"
64+
caption="Adding data quality tests to an OpenMetadata Data Contract"
65+
/%}
66+
67+
- Once your new data contract has been created successfully, you can run it by selecting **> Run now**
68+
69+
{% image
70+
src="/images/v1.10/how-to-guides/contracts/create7.png"
71+
alt="Running an OpenMetadata Data Contract"
72+
caption="Running an OpenMetadata Data Contract"
73+
/%}

‎content/v1.10.x/how-to-guides/data-contracts/index.md‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
---
2+
title: Data Contracts | OpenMetadata Data Contracts
3+
description: Step-by-step guidance for data contracts using OpenMetadata. Learn key actions, features, and best practices.
4+
slug: /how-to-guides/data-contracts
5+
---
6+
7+
# Overview of Data Contracts
8+
9+
OpenMetadata is a centralized, active metadata repository where all your data resides. In any organization, it is important to get the consumers utilizing data assets aligned with asset producers. As data platform adoption increases in an organization, a data mesh architecture with OpenMetadata Data Contracts can produce automated schema, semantic, and data quality validations that enforce data quality and SLAs across departments and domains.
10+
11+
Contracts in OpenMetadata are now available for tables. With OpenMetadata Data Contracts, you can set the expected schema structure of an asset, define its semantics (like ensuring that a table has a description field or an [owner assigned](https://docs.open-metadata.org/latest/how-to-guides/guide-for-data-users/data-ownership)), assign data quality tests, and track a data contract's execution history over time.
12+
13+
Watch the video to learn more about OpenMetadata Data Contracts.
14+
15+
{% ossContent %}
16+
17+
{% youtube videoId="9CAy_kNvenA" start="0:00" end="4:24" width="800px" height="450px" /%}
18+
19+
{% /ossContent %}
20+
21+
{% collateContent %}
22+
23+
{% youtube videoId="thLYeMx2sxs" start="0:00" end="4:24" width="800px" height="450px" /%}
24+
25+
{% /collateContent %}
26+
27+
Watch a demo of Data Contracts from our [OpenMetadata Community Meetup](https://www.meetup.com/openmetadata-meetup-group/)
28+
29+
{% ossContent %}
30+
31+
{% youtube videoId="sz2XUEQHp08" start="0:00" end="29:10" width="800px" height="450px" /%}
32+
33+
{% /ossContent %}
34+
35+
{% collateContent %}
36+
37+
{% youtube videoId="H4z6exr45X4" start="0:00" end="02:08" width="800px" height="450px" /%}
38+
39+
{% /collateContent %}
40+
41+
{%inlineCalloutContainer%}
42+
{%inlineCallout
43+
color="violet-70"
44+
bold="OpenMetadata Data Contract Specification"
45+
icon="puzzle"
46+
href="/how-to-guides/data-contracts/spec"%}
47+
Overview of JSON Schema for DataContract entity
48+
{%/inlineCallout%}
49+
{%inlineCallout
50+
color="violet-70"
51+
bold="Creating a Data Contract"
52+
icon="MdConnectWithoutContact"
53+
href="/how-to-guides/data-contracts/create"%}
54+
Create data contracts directly from the OpenMetadata UI!
55+
{%/inlineCallout%}
56+
{%/inlineCalloutContainer%}

‎content/v1.10.x/how-to-guides/data-contracts/spec.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
---
2+
title: Data Contract Specification | OpenMetadata Data Contracts Guide
3+
description: Create open-source data contracts directly in the OpenMetadata UI
4+
slug: /how-to-guides/data-contracts/spec
5+
---
6+
7+
# Introduction
8+
9+
Data contracts formalize an agreement between data producers and consumers about what to expect from a data asset’s data. They capture the structure, semantics, quality, and SLAs of data in a machine-readable way, similar to an API contract but for data. In essence, a Data Contract is enforceable in the data ecosystem to bring standardization, control, and reliability.
10+
11+
OpenMetadata, as a metadata management platform, integrates this concept by introducing a Data Contract entity defined via JSON Schema. This allows OpenMetadata admins and [data product](https://docs.open-metadata.org/latest/how-to-guides/data-governance/domains-&-data-products#data-products) owners to attach a contract to tables in OpenMetadata, codifying expectations in a structured format. The contract can then be enforced or validated using OpenMetadata’s existing metadata and data quality frameworks. The goal is to have contextually rich, high-quality, well-governed data that is trustworthy. Data contracts achieve this by making data expectations explicit and automating their enforcement.
12+
13+
# Data Contract Entity Schema Design
14+
15+
The JSON Schema definition for Data Contract entities in OpenMetadata defines the contract’s structure and allowed fields. The contract covers seven main categories of expectations:
16+
17+
1. [Schema](#schema)
18+
2. [Semantics](#semantics)
19+
3. [Security](#security)
20+
4. [Business Assertions (data quality)](#quality)
21+
5. [SLA](#sla)
22+
6. [Terms of Use](#terms-of-use)
23+
7. [Status](#status)
24+
25+
We also include an SLA section for service-level agreements and an Ownership field for accountability. Each Data Contract is designed to represent one single data asset (dataset, topic, model, etc.) in a well-structured, templated format. Data contracts are currently available for Table asset types.
26+
27+
The JSON Schema for the Data Contract entity can be found [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/data/dataContract.json)
28+
29+
# Data Contract Sections
30+
## Schema
31+
32+
This is where the expected structural schema of the data asset is defined. It includes a list of fields (for a table, these are columns) each with name and data type. This captures the contractual schema that producers and consumers agreed on, which usually is a subset of the available fields on an asset.
33+
34+
## Semantics
35+
36+
Business meaning and documentation requirements are defined in a contract's Semantics section. For example, one can enforce that a data asset must have a description, owner or domain.
37+
38+
These rules complement the formal tests in the quality section, acting as documentation of business expectations. This section ensures the contract isn’t just about technical schema, but also carries business context.
39+
40+
## Security
41+
42+
Data security and access expectations are defined in this section. This can reference an access policy ID or name that should govern this data, or a required classification label.
43+
44+
In practice, this means the contract might require the data asset to be tagged as `PII` or `Confidential` if appropriate, and that only certain roles can access it (through an associated policy).
45+
46+
## Quality (Assertions)
47+
48+
Data quality tests and assertions required by the contract are here.
49+
50+
Built on top of the native Data Quality features in OpenMetadata, this section allows defining specific tests that must pass for the data to be considered compliant with the contract. Tests can be at the column level (e.g., a column must be non-null) or table level (e.g., row count must be above a threshold), and can be managed from the Data Contract UI itself.
51+
52+
## SLA
53+
54+
Service-Level Agreements related to the data’s timeliness and lifecycle are captured in this section.
55+
This includes:
56+
Refresh Frequency: how often the data is expected to be updated or refreshed (e.g., daily, weekly,...).
57+
Max Latency: the maximum allowed delay between data generation and when it’s available to consumers (e.g., data may be up to 4 hours old at most, or one day for typical daily batch ETLs).
58+
Availability Time: the time by which daily or periodic data should be available (e.g., “09:00 UTC” daily data drop).
59+
Retention: how long the data is kept accessible (if applicable).
60+
61+
Including them in the contract means producers commit to certain delivery timelines, and consumers know what availability to expect.
62+
63+
## Terms of Use
64+
65+
This section captures the allowed and disallowed uses of the data asset, as well as any compliance or regulatory requirements. This can include:
66+
- Allowed Uses: Describes what the data can be used for (e.g., internal analytics, reporting).
67+
- Disallowed Uses: Specifies prohibited uses (e.g., no sharing with third parties, or no training AI models).
68+
- Compliance Requirements: Any legal or regulatory obligations (e.g., GDPR, HIPAA).
69+
70+
## Status
71+
72+
A status field indicates whether the contract is active, draft, or currently violated. For instance, when first created, a contract is in `DRAFT` when it is not yet enforced or not fully implemented by the data producer. Once a data contract is published, it becomes `ACTIVE`. If a violation occurs (e.g., a test fails or schema deviates), the contract's status is `VIOLATED`.
73+
74+
Data Contracts have approval workflows when changes are made, similar to Glossaries. This ensures that any modifications to the contract (like adding new quality tests or changing schema expectations) go through a review and approval process.
75+
76+
# Applying Contracts to Tables
77+
78+
Below is an example of a data contract for a warehouse.sales.orders table.
79+
80+
## Data Contract YAML Example
81+
82+
This is an example YAML of a Data Contract applied to a table in OpenMetadata. Note that while OpenMetadata brings full UI support for creating and managing Data Contracts, you can still use the API to manage them programmatically.
83+
84+
```
85+
name: Customers DC
86+
status: Active
87+
entity:
88+
id: 8beb4301-8302-4791-9944-2897e7614a1a
89+
type: table
90+
href: https://example.com/v1/tables/8beb4301-8302-4791-9944-2897e7614a1a
91+
schema:
92+
- name: customer_id
93+
dataType: INT
94+
dataLength: 1
95+
dataTypeDisplay: integer
96+
description: New ID from Collate UI
97+
fullyQualifiedName: red.dev.dbt_jaffle.customers.customer_id
98+
tags: []
99+
constraint: 'NULL'
100+
children: []
101+
- name: first_name
102+
dataType: VARCHAR
103+
dataLength: 20
104+
dataTypeDisplay: character varying(20)
105+
fullyQualifiedName: red.dev.dbt_jaffle.customers.first_name
106+
tags:
107+
- tagFQN: General.Person
108+
name: Person
109+
description: >-
110+
A full person name, which can include first names, middle names or
111+
initials, and last names.
112+
source: Classification
113+
labelType: Generated
114+
state: Suggested
115+
- tagFQN: PII.Sensitive
116+
name: Sensitive
117+
description: >-
118+
PII which if lost, compromised, or disclosed without authorization,
119+
could result in substantial harm, embarrassment, inconvenience, or
120+
unfairness to an individual.
121+
source: Classification
122+
labelType: Generated
123+
state: Suggested
124+
constraint: 'NULL'
125+
children: []
126+
semantics:
127+
- name: Owners is set
128+
description: Ownership is mandatory
129+
rule: >-
130+
{"and":[{"some":[{"var":"owners"},{"!=":[{"var":"fullyQualifiedName"},null]}]}]}
131+
qualityExpectations:
132+
- id: 1efbda53-063d-4611-8f69-402f4490a503
133+
type: testCase
134+
name: customer rows
135+
- id: 707a43f9-d1d1-4fb8-96da-7bb428429f87
136+
type: testCase
137+
name: relationships_orders_customer_id__customer_id__ref_customers_
138+
description: ''
139+
owners: []
140+
reviewers: []
141+
```
142+
143+
{%inlineCallout
144+
color="violet-70"
145+
bold="Creating Data Contracts"
146+
icon="MdArrowForward"
147+
href="/how-to-guides/data-contracts/create"%}
148+
Create Data Contracts in the OpenMetadata UI.
149+
{%/inlineCallout%}

‎content/v1.10.x/menu.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1008,6 +1008,11 @@ site_menu:
10081008
- category: How-to Guides / Data Discovery / OpenMetadata Chrome Extension / Viewing tasks & add comment directly from the extension
10091009
url: /how-to-guides/data-discovery/openmetadata-extension/viewing-tasks
10101010

1011+
- category: How-to Guides / Data Contracts
1012+
url: /how-to-guides/data-contracts
1013+
- category: How-to Guides / Data Contracts / Creating Data Contracts
1014+
url: /how-to-guides/data-contracts/create
1015+
10111016
- category: How-to Guides / Data Collaboration
10121017
url: /how-to-guides/data-collaboration
10131018
- category: How-to Guides / Data Collaboration / Understanding Activity Feeds

‎content/v1.11.x-SNAPSHOT/how-to-guides/data-contracts/create.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
title: Creating Data Contracts | OpenMetadata Data Contracts Guide
3+
description: Follow this guide to create data contracts. Define schema, add business rules, and run quality tests to ensure consistent, reliable table data.
4+
slug: /how-to-guides/data-contracts/create
5+
---
6+
7+
Data contracts are currently available for tables ingested into OpenMetadata. The following guide shows how a data contract was created for a snowflake.DEMO_STAGE.JAFFLE_SHOP.CUSTOMERS table.
8+
9+
To create a Data Contract for a Table in OpenMetadata:
10+
- Go to the Table's page, select **Contract**, then **+ Add Contract**
11+
12+
{% image
13+
src="/images/v1.11/how-to-guides/contracts/create1.png"
14+
alt="Adding a Data Contract to an OpenMetadata Table"
15+
caption="Adding a Data Contract to an OpenMetadata Table"
16+
/%}
17+
18+
- In *Contract Details*, be sure to give your new data contract a name. Optionally, you can assign Owners and provide a description of your data contract. Then select **Schema**.
19+
20+
{% image
21+
src="/images/v1.11/how-to-guides/contracts/create2.png"
22+
alt="Adding Contract Details"
23+
caption="Adding a Contract Details to an OpenMetadata Data Contract"
24+
/%}
25+
26+
- Select the columns of your table that you would like to add to your new data contract, or select all columns by checking the box at the top right. Then select **Semantics**
27+
28+
{% image
29+
src="/images/v1.11/how-to-guides/contracts/create3.png"
30+
alt="Selecting a schema for an OpenMetadata Data Contract"
31+
caption="Selecting a schema for an OpenMetadata Data Contract"
32+
/%}
33+
34+
- Add the business rules you would like to enforce in *Semantics*. For OpenMetadata Tables, rules can be created for:
35+
- Service
36+
- Owners
37+
- Display Name
38+
- Name
39+
- Description
40+
- Tags
41+
- Domain
42+
- Data Product
43+
- Tier
44+
45+
{% image
46+
src="/images/v1.11/how-to-guides/contracts/create4.png"
47+
alt="Adding rules to an OpenMetadata Data Contract"
48+
caption="Adding rules to an OpenMetadata Data Contract"
49+
/%}
50+
51+
- Once a rule is created, you can **+ Add New Rule** to create more, or select **Quality**
52+
53+
{% image
54+
src="/images/v1.11/how-to-guides/contracts/create5.png"
55+
alt="Adding more rules to an OpenMetadata Data Contract"
56+
caption="Adding more rules to an OpenMetadata Data Contract"
57+
/%}
58+
59+
- Select **+ Add Test** to add a [Data Quality Test](https://docs.open-metadata.org/latest/how-to-guides/data-quality-observability/quality/test) or tests to your new contract, then select **Save**
60+
61+
{% image
62+
src="/images/v1.11/how-to-guides/contracts/create6.png"
63+
alt="Adding data quality tests to an OpenMetadata Data Contract"
64+
caption="Adding data quality tests to an OpenMetadata Data Contract"
65+
/%}
66+
67+
- Once your new data contract has been created successfully, you can run it by selecting **> Run now**
68+
69+
{% image
70+
src="/images/v1.11/how-to-guides/contracts/create7.png"
71+
alt="Running an OpenMetadata Data Contract"
72+
caption="Running an OpenMetadata Data Contract"
73+
/%}

0 commit comments

Comments
 (0)