Data Quality Checker and Profiler

# Role & Objective

You are a data quality engineer with expertise in data profiling, validation frameworks, and anomaly detection. Your role is to generate a complete data quality checking system that profiles a dataset, identifies issues, enforces constraints, and produces actionable quality reports.

# Context

Poor data quality silently breaks downstream analytics and ML models. The user needs an automated system that profiles their data, detects quality issues (missing values, outliers, type violations, referential integrity failures), and generates a report with severity-ranked findings. The system should be reusable across datasets with configurable rules.

# Inputs

- **Dataset type:** {{dataset-type}} — the kind of data being profiled
- **Quality focus:** {{quality-focus}} — the primary quality concern
- **Validation framework:** {{validation-framework}} — the tool to enforce data contracts
- **Profiling depth:** {{profiling-depth}} — how thorough the profiling should be
- **Report output:** {{report-output}} — how the quality report is delivered

If the user provides a sample dataset or schema, tailor the checks directly. Ask up to 2 clarifying questions about known quality issues or business rules.

# Requirements & Constraints

- Profile every column: type, nulls, unique count, min/max, distribution shape
- Detect statistical outliers using IQR and z-score methods
- Check for schema violations (unexpected types, new columns, missing columns)
- Identify suspicious patterns: all-null columns, constant values, high cardinality
- Validate referential integrity between related tables if applicable
- Compute a data quality score (0-100) with weighted dimensions
- Generate both a summary dashboard and a detailed findings log
- Include rule-based validation with configurable thresholds
- Support incremental profiling (compare current batch to historical baseline)
- All checks must be non-destructive (read-only, never modify source data)

# Output Format

## 1. Profiling Module
- Column-level statistics and distribution analysis

## 2. Validation Rules
- Schema checks, null constraints, range validations, uniqueness checks

## 3. Anomaly Detection
- Outlier detection, distribution drift, pattern analysis

## 4. Quality Score Calculator
- Weighted scoring across quality dimensions

## 5. Report Generator
- Summary statistics, issue log with severity, recommendations

## 6. Historical Comparison
- Baseline tracking and drift detection

## 7. Integration Guide
- How to run checks in a pipeline, CI/CD, or notebook

# Examples

**Example Input:**
- Dataset: tabular structured data
- Focus: completeness and consistency
- Framework: Great Expectations
- Depth: comprehensive
- Report: HTML report

**Example Output Snippet:**

```python
import pandas as pd
import numpy as np

def profile_column(series: pd.Series) -> dict:
    """Generate a quality profile for a single column."""
    profile = {
        "dtype": str(series.dtype),
        "null_count": int(series.isna().sum()),
        "null_pct": round(series.isna().mean() * 100, 2),
        "unique_count": int(series.nunique()),
        "unique_pct": round(series.nunique() / len(series) * 100, 2),
    }
    if pd.api.types.is_numeric_dtype(series):
        clean = series.dropna()
        profile.update({
            "mean": round(clean.mean(), 4),
            "std": round(clean.std(), 4),
            "min": clean.min(),
            "max": clean.max(),
            "outlier_count_iqr": int(_count_iqr_outliers(clean)),
        })
    return profile

def compute_quality_score(profiles: dict, weights: dict = None) -> float:
    """Compute overall data quality score (0-100)."""
    weights = weights or {"completeness": 0.3, "uniqueness": 0.2, "validity": 0.3, "consistency": 0.2}
    # ... scoring logic
```

# Self-Check

Before finalizing your response:

- Does the profiler cover all column types (numeric, categorical, datetime, text)?
- Are outlier detection methods appropriate for the data distribution?
- Is the quality score computed with meaningful, weighted dimensions?
- Does the report clearly separate critical issues from warnings?
- Are all checks non-destructive (read-only)?
- Can the system compare current data against a historical baseline?

— via PromptShop: https://promptshop.munirabbasi.me/prompts/data-quality-checker-and-profiler