Purview Multi-Criteria Data Quality Accelerator

Deploy to: Microsoft Fabric Lakehouse + Microsoft Purview Data Governance

---

Overview

The problem

Microsoft Purview Data Quality lets you define rules on individual data assets — check if a column is null, validate a format, or verify a value range. But real-world data quality requirements are rarely that simple. Consider an insurance company that needs to enforce:

• A claim amount limit of 50,000 — but only for commercial auto policies in the EU region
• A premium rate cap of 8% — but only for accounts in the high-risk tier
• A policy number format check — but only for EU-region claims

Purview DQ rules operate on a single table at a time. They don't support multi-criteria matching across dimensions like policy type, region, product line, and risk tier. There is no native way to say "apply this rule only when account_type = Commercial AND region = EU AND product = Auto." You'd need to create separate filtered views or hardcode each combination — an approach that breaks down as the number of segments grows.

What this accelerator does

This project fills that gap. It is a metadata-driven DQ rules engine that runs on Microsoft Fabric and feeds structured results into Microsoft Purview for monitoring, scoring, and alerting. Rules are defined as data in a config table — not hardcoded — so business and compliance teams can add, edit, or disable rules without touching code.

The engine joins source data with reference data, matches each record against applicable rules using multi-criteria filters with wildcard support, evaluates Spark SQL expressions dynamically, and writes detailed pass/fail results to Delta tables. Purview then monitors those output tables, tracks quality scores over time, and sends alerts when thresholds are breached.

Key capabilities

• Multi-criteria rule matching with wildcard support — target rules to any combination of dimensions (e.g., policy type + region + product + risk tier + channel)
• Dynamic rule evaluation — Spark SQL expressions stored in a config table and evaluated at runtime
• Severity classification — Critical, High, Medium, Low — with separate thresholds and alerting
• Purview integration — output tables feed directly into Purview DQ monitoring, quality scoring, trending, and failed-row drill-down
• Zero business logic in code — the engine is generic; all domain knowledge lives in the rules config table
• No-code rule management — business users add or modify rules by editing a table, not by deploying code

Industry example

This accelerator ships with a complete insurance industry example covering claims processing data quality. The same engine works for any industry — banking, healthcare, retail, manufacturing — swap the sample data and rules config for your own domain.

---

Architecture

flowchart TD
    Business["👤 Business Users<br/><i>Write rules as data — no code</i>"]

    subgraph Fabric["<b>Microsoft Fabric</b>"]
        direction TB
        Claims[("🗂️ Claims<br/>Source data")]
        Ref[("🗂️ Policy Criteria<br/>Reference data")]
        Rules[("📋 Rules Config<br/>Metadata-driven")]
        Engine["⚙️ <b>Rules Engine</b><br/>Join · Match · Evaluate"]
        Results[("✅ DQ Results<br/>Pass / Fail per record")]
        Summary[("📊 DQ Summary<br/>Quality scorecard")]
    end

    subgraph Purview["<b>Microsoft Purview Data Governance</b>"]
        direction TB
        Monitor["🛡️ Quality Monitoring<br/>Scoring · Trending"]
        Alerts["🔔 Alerts<br/>Threshold notifications"]
        DrillDown["🔍 Failed Rows<br/>Drill-down details"]
    end

    Stewards["🔧 Data Stewards"]
    Compliance["📜 Compliance &amp; Regulators"]
    Management["📈 Management"]

    Business -- define rules --> Rules
    Claims --> Engine
    Ref --> Engine
    Rules --> Engine
    Engine --> Results
    Engine --> Summary
    Results --> Monitor
    Summary --> Monitor
    Monitor --> Alerts
    Monitor --> DrillDown
    Alerts --> Stewards
    DrillDown --> Stewards
    Monitor --> Compliance
    Monitor --> Management

    style Fabric fill:#e6f3ff,stroke:#0078d4,stroke-width:2px,color:#333
    style Purview fill:#fff0f5,stroke:#cc0066,stroke-width:2px,color:#333
    style Business fill:#fff3e6,stroke:#ff8c00,stroke-width:2px,color:#333
    style Claims fill:#dce8f5,stroke:#0078d4,color:#333
    style Ref fill:#dce8f5,stroke:#0078d4,color:#333
    style Rules fill:#dce8f5,stroke:#0078d4,color:#333
    style Engine fill:#c5d5e8,stroke:#0058a3,stroke-width:3px,color:#333
    style Results fill:#d5ecd5,stroke:#2e8b57,color:#333
    style Summary fill:#d5ecd5,stroke:#2e8b57,color:#333
    style Monitor fill:#f5d5e0,stroke:#cc0066,color:#333
    style Alerts fill:#f5d5e0,stroke:#cc0066,color:#333
    style DrillDown fill:#f5d5e0,stroke:#cc0066,color:#333
    style Stewards fill:#f0f0f0,stroke:#666,color:#333
    style Compliance fill:#f0f0f0,stroke:#666,color:#333
    style Management fill:#f0f0f0,stroke:#666,color:#333

Loading

---

Folder Contents

File	Purpose	Deploy as
01_generate_sample_data.py	Creates sample insurance claims and policy criteria tables	Fabric Notebook
02_setup_rules_config.py	Creates the DQ rules config table with initial insurance rules	Fabric Notebook
03_dq_rules_engine.py	The core engine — matches records to rules, evaluates, writes results	Fabric Notebook
04_dq_summary_report.py	Generates summary metrics and a report on the DQ results	Fabric Notebook
pipeline_config.json	Pipeline configuration for Fabric Data Factory pipeline	Reference
dq_flow_diagram.mmd	Mermaid architecture diagram	Reference
docs/purview_dq_playbook.md	Complete Purview DQ setup guide with exact syntax and troubleshooting	Reference

---

Quick Start

1. Create Fabric Workspace and Lakehouse

1. Open Microsoft Fabric
2. Create a new Workspace (e.g., DQ_Accelerator)
3. Inside the workspace, create a new Lakehouse (e.g., dq_lakehouse)

2. Deploy Notebooks

For each .py file:

1. In the workspace, click + New → Notebook
2. Name the notebook matching the file (e.g., 01_generate_sample_data)
3. Copy the .py file content into the notebook
4. Attach the notebook to your Lakehouse

3. Run in Order

01_generate_sample_data   →  Creates insurance claims + policy reference tables
02_setup_rules_config     →  Creates 10 metadata-driven DQ rules
03_dq_rules_engine        →  Evaluates all active rules against enriched data
04_dq_summary_report      →  Produces quality scorecard for Purview

4. Create Fabric Pipeline (Optional)

1. + New → Data Pipeline
2. Add 4 Notebook activities in sequence, pointing to each notebook
3. Schedule daily after your ETL completes

5. Configure Purview Data Governance

See docs/purview_dq_playbook.md for the full step-by-step guide.

---

Insurance Example — What Gets Created

The scenario

An insurance company processes claims across multiple product lines (auto, home, health, life), regions (EU, US), and policy types (commercial, individual). Each combination has different regulatory and business rules — a commercial auto policy in the EU has a different claim limit than an individual health policy in the US. High-risk accounts face stricter premium rate caps than standard accounts.

The sample data simulates 15 claims filed against 6 policies. Some claims are intentionally designed to fail specific rules, demonstrating how the engine catches violations that depend on multiple criteria.

Source tables

claims — 15 insurance claims (the data being validated)

Each claim has an amount, type (Medical, Property, Collision, etc.), premium rate, policy number, and a date. Claims span commercial fleet accidents, individual health procedures, property damage, and edge cases like missing policy numbers or negative amounts.

policy_criteria — 6 policy reference records (the context that determines which rules apply)

Each policy has a type (Commercial / Individual), region (EU / US), product (Auto / Health / Home / Life / Comprehensive), risk tier (Low / Medium / High), and channel (Broker / Agent / Online). The engine joins claims to their policy to enrich each record with these dimensions before evaluating rules.

How multi-criteria matching works

Each rule in the config table specifies which segments it targets using 5 criteria columns. A * means "match all" (wildcard):

Rule ID	Name	Type	Region	Product	Risk	Channel	What it checks
R001	Commercial Auto Claim Limit	Commercial	EU	Auto	*	*	claim_amount <= 50000
R002	Commercial EU Premium Rate Cap	Commercial	EU	*	*	*	premium_rate <= 0.12
R003	Individual Health Claim Limit	Individual	US	Health	*	*	claim_amount <= 10000
R004	Allowed Claim Types	*	*	*	*	*	claim_type IN ('Medical', 'Property', 'Liability', 'Collision', 'Comprehensive')
R005	High-Risk Premium Rate Cap	*	*	*	High	*	premium_rate <= 0.08
R006	Premium Rate Not Null	*	*	*	*	*	premium_rate IS NOT NULL
R007	Positive Claim Amount	*	*	*	*	*	claim_amount > 0
R008	EU Policy Number Required	*	EU	*	*	*	policy_number IS NOT NULL
R009	Claim Within Coverage Limit	*	*	*	*	*	claim_amount <= coverage_limit
R010	EU Policy Number Format	*	EU	*	*	*	18-character format validation

R001 only fires for records where policy_type = Commercial AND region = EU AND product = Auto. R005 only fires when risk_tier = High, regardless of type, region, or product. R004 and R007 fire for every record (all wildcards). This is the multi-criteria matching that Purview DQ doesn't support natively.

Expected failures in the sample data

The sample data is designed so specific claims trigger specific rule failures:

Claim	Policy	What goes wrong	Rules that catch it
C002	P100 (Commercial/EU/Auto)	Claim amount 65,000 exceeds 50K limit	R001
C003	P100 (Commercial/EU)	Premium rate 0.15 exceeds 12% cap	R002
C005	P200 (Individual/US/Health)	Claim amount 12,000 exceeds 10K limit	R003
C006	P200 (Individual/US)	"Dental" is not an allowed claim type	R004
C007	P300 (Commercial/EU/Auto/High)	Fails R001 (>50K), R002 (>12%), R005 (>8% for high-risk)	R001, R002, R005
C011	P500 (Commercial/US/Life)	Premium rate is NULL	R006
C012	P500 (Commercial/US/Life)	Claim amount is zero	R007
C013	P600 (Individual/EU)	Policy number is NULL	R008
C014	P600 (Individual/EU)	Claim amount is negative (-1,500)	R007

Claim C007 is the best demonstration of multi-criteria matching — it fails 3 different rules because it matches multiple criteria combinations: Commercial+EU+Auto triggers R001, Commercial+EU triggers R002, and High risk tier triggers R005.

Output tables

dq_results — one row per claim per rule evaluation. Each row records the run ID, claim ID, rule ID, whether the rule passed, the actual value that was checked, and the full criteria context (policy type, region, product, risk tier, channel). This table is appended with each run, preserving full audit history.

dq_summary — one row per run with aggregated metrics: total evaluations, pass/fail counts, overall pass rate, and failure counts broken down by severity level (critical, high, medium, low). Purview monitors this table for quality score trending and alerting.

---

Customization

Adding new rules

Edit the dq_rules_config table directly — add rows with new rule definitions. No code changes needed.

Adding new criteria columns

1. Add the column to dq_rules_config
2. Add a matching entry to the CRITERIA_COLUMNS list in 03_dq_rules_engine.py

Changing target tables

Modify the parameters at the top of 03_dq_rules_engine.py to point to different source tables.

Adapting to another industry

1. Replace sample data in 01_generate_sample_data.py with your domain entities
2. Update rules in 02_setup_rules_config.py (or edit the config table directly)
3. Adjust CRITERIA_COLUMNS in 03_dq_rules_engine.py if your segmentation differs
4. Notebooks 03 and 04 require no business-logic changes — they are fully generic

---

Requirements

• Microsoft Fabric capacity (F2 or higher)
• Fabric Lakehouse
• Microsoft Purview Data Governance (for the monitoring layer)
• Spark runtime (provided by Fabric)

---

Disclaimer

This is not an official Microsoft product. This project is a personal contribution by a Microsoft employee and is not endorsed, supported, or maintained by Microsoft.

The accelerator is built on Microsoft Fabric and Microsoft Purview, which provide their own SLAs. The accelerator itself is provided as-is, with no warranties or SLAs. It is intended as a starting point and reference implementation — each organization is responsible for testing, validating, and adapting the solution to their own requirements, data governance practices, and security policies. The sample data and rules included are fictional and for demonstration purposes only.

---

Contributing

See CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License — see LICENSE for details.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. See CODE_OF_CONDUCT.md for details.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.