AgentAlley

Skills · Data & AI

Data Quality Frameworks

Unverified●30/40

Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.

Originally by wshobson · MIT

Claude Code◐PartialHas SKILL.md but declares no allowed-tools — Claude Code will ask for permission each time
Cursor·UnknownWe have not crawled the repo tree, so we will not guess
Codex·UnknownWe have not crawled the repo tree, so we will not guess
Gemini CLI·UnknownThe spec defines no detection rule for Gemini
Copilot·UnknownWe have not crawled the repo tree, so we will not guess
npx agentalley add data-quality-frameworks

This command does not work yet — the CLI is still being built. Until then, use Raw in the reader below to take the file.

Who is stuck, and on what

Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.

The whole source

No sign-in, no blur, nothing truncated
data-quality-frameworks/SKILL.md139 lines4.4 KBRawView on GitHub
Frontmatter — 2 properties
namedata-quality-frameworks
descriptionImplement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.
1---
2name: data-quality-frameworks
3description: Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.
4---A5 — No allowed-tools declared — no way to tell what this skill may touch
5 
6# Data Quality Frameworks
7 
8Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.
9 
10## When to Use This Skill
11 
12- Implementing data quality checks in pipelines
13- Setting up Great Expectations validation
14- Building comprehensive dbt test suites
15- Establishing data contracts between teams
16- Monitoring data quality metrics
17- Automating data validation in CI/CD
18 
19## Core Concepts
20 
21### 1. Data Quality Dimensions
22 
23| Dimension | Description | Example Check |
24| ---------------- | ------------------------ | -------------------------------------------------- |
25| **Completeness** | No missing values | `expect_column_values_to_not_be_null` |
26| **Uniqueness** | No duplicates | `expect_column_values_to_be_unique` |
27| **Validity** | Values in expected range | `expect_column_values_to_be_in_set` |
28| **Accuracy** | Data matches reality | Cross-reference validation |
29| **Consistency** | No contradictions | `expect_column_pair_values_A_to_be_greater_than_B` |
30| **Timeliness** | Data is recent | `expect_column_max_to_be_between` |
31 
32### 2. Testing Pyramid for Data
33 
34```
35 /\
36 / \ Integration Tests (cross-table)
37 /────\
38 / \ Unit Tests (single column)
39 /────────\
40 / \ Schema Tests (structure)
41 /────────────\
42```
43 
44## Quick Start
45 
46### Great Expectations Setup
47 
48```bash
49# Install
50pip install great_expectations
51 
52# Initialize project
53great_expectations init
54 
55# Create datasource
56great_expectations datasource new
57```
58 
59```python
60# great_expectations/checkpoints/daily_validation.yml
61import great_expectations as gx
62 
63# Create context
64context = gx.get_context()
65 
66# Create expectation suite
67suite = context.add_expectation_suite("orders_suite")
68 
69# Add expectations
70suite.add_expectation(
71 gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id")
72)
73suite.add_expectation(
74 gx.expectations.ExpectColumnValuesToBeUnique(column="order_id")
75)
76 
77# Validate
78results = context.run_checkpoint(checkpoint_name="daily_orders")
79```
80 
81## Detailed patterns and worked examples
82 
83Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
84 
85## Summary: {total_passed}/{total_tables} tables passed")
86 report.append("")
87 
88 for table, result in results.items():
89 status = "✅" if result.passed else "❌"
90 report.append(f"### {status} {table}")
91 report.append(f"- Expectations: {result.total_expectations}")
92 report.append(f"- Failed: {result.failed_expectations}")
93 
94 if not result.passed:
95 report.append("- Failed checks:")
96 for detail in result.details:
97 if not detail["success"]:
98 report.append(f" - {detail['expectation']}: {detail['observed_value']}")
99 report.append("")
100 
101 return "\n".join(report)
102 
103# Usage
104context = gx.get_context()
105pipeline = DataQualityPipeline(context)
106 
107tables_to_validate = {
108 "orders": "orders_suite",
109 "customers": "customers_suite",
110 "products": "products_suite",
111}
112 
113results = pipeline.run_all(tables_to_validate)
114report = pipeline.generate_report(results)
115 
116# Fail pipeline if any table failed
117if not all(r.passed for r in results.values()):
118 print(report)
119 raise ValueError("Data quality checks failed!")
120```
121 
122## Best Practices
123 
124### Do's
125 
126- **Test early** - Validate source data before transformations
127- **Test incrementally** - Add tests as you find issues
128- **Document expectations** - Clear descriptions for each test
129- **Alert on failures** - Integrate with monitoring
130- **Version contracts** - Track schema changes
131 
132### Don'ts
133 
134- **Don't test everything** - Focus on critical columns
135- **Don't ignore warnings** - They often precede failures
136- **Don't skip freshness** - Stale data is bad data
137- **Don't hardcode thresholds** - Use dynamic baselines
138- **Don't test in isolation** - Test relationships too
139 

Reviews

Installed this one?Write the first review and take the Trailblazer badge.

Reviews only open after a real install, so this is empty — and we leave it empty rather than invent one.

Alternatives

Also in Data & AI
Task Coordination StrategiesDecompose complex tasks, design dependency graphs, and coordinate multi-agent work with proper task descriptions and workload balancing. Use this skill when breaking down work for agent teams, managing task dependencies, or monitoring team progress.Data & AI · MIT◐◐◐◐◐●35/40Ebay Seller ToolsData & AI · MIT·····●34/40Tough Decision Advisor: Every Angle ConsideredHand in a decision you're stuck on. Get back a clear breakdown of every angle — the trade-offs, the risks, the blind spot, and a recommended path.Data & AI · Business · MIT license●····●32/40DHDNA Profiler — Cognitive Pattern ExtractionPaste any email, proposal, or note someone wrote, and get back a plain-language read on how they think, what drives their decisions, and how they communicate.Data & AI · Research & analysis · MIT license●····●32/40