How I Built a COREP Regulatory Reporting Pipeline in 18 Days — From Data Governance Theory to Valid XBRL

Eighteen days ago I set a constraint: build a complete COREP regulatory reporting pipeline using only open-source tools, learn EU banking governance from scratch, and write a blog post every day documenting both the theory and the implementation. Today is the retrospective.

The end product is a working pipeline that ingests synthetic bank data, transforms it through a governed dbt layer, validates it against the EBA XBRL taxonomy with Arelle, and produces a submission-ready ZIP file — all orchestrated by Apache Airflow with column-level lineage to Marquez, row/column security via Apache Ranger, and a regulatory dashboard in Apache Superset. Every component is open source. The total cost of the infrastructure is the electricity to run Docker Desktop.

This post covers: what was actually built, the six governance pillars the project was designed around, the one architecture decision that changed everything (Trino as the single query gateway), the three hardest technical problems encountered, and what a production deployment at a real bank would need on top of all this.

What Was Actually Built

The complete codebase: 17 Docker services, 8 Python modules, 4 dbt mart models, 10 Ranger policies, 4 GX expectation suites, and a pytest suite with 15 adversarial test cases.

The Six Governance Pillars

The 18-day plan was structured around six pillars that the EBA and ECB specifically audit when reviewing a bank’s COREP data quality framework. Here is how each pillar was addressed:

The Architecture Decision That Changed Everything: Trino as the Single Query Gateway

The most consequential architecture decision made on Day 4 was this: every consumer of data — dbt, GX, Superset, XbrlGenModule, all test code — connects to Trino, never directly to PostgreSQL. At the time it felt like a slightly awkward constraint. In retrospect it is the single decision that makes the entire governance stack coherent.

Here is what it actually delivered:

The cost was one additional hop in the query path (application → Trino → PostgreSQL) and the need to qualify every table as postgresql.schema.table rather than schema.table. That is a trivial cost relative to what it buys.

The Three Hardest Technical Problems

1 · XBRL Formula Validation — Parents Must Equal Children

The calculation linkbase problem was not obvious until Day 12. The EBA taxonomy specifies that c0010 (Own Funds) = c0020 (CET1) + c0030 (AT1) + c0040 (T2) via a calculation linkbase arc with weight +1. Arelle enforces this with tolerance decimals=-3, meaning a EUR 1,000 rounding difference is acceptable but a EUR 1,001 difference is an xbrlCalcs:inconsistency error that fails validation.

The root cause: the dbt mart models computed each tier independently using separate CTEs, introducing rounding divergence at the EUR thousandths position. The fix required two changes:

1. In dbt: all four COREP concepts (c0010, c0020, c0030, c0040) must be derived from the same CTE using a single SUM(amount_eur) grouped by tier, then assembled with FILTER(WHERE tier = 'CET1'). Parent = sum of same-CTE children by construction.
2. In xbrl_gen.py: rounding (ROUND_HALF_UP, decimals=-3) is applied exactly once, terminally, at the point of writing the XBRL fact. Never round in intermediate dbt models — let the mart values carry full precision and round only when writing to XML.

The lesson: XBRL calculation linkbase validation is not a reporting concern — it is a data modelling concern. The constraint must be enforced in the mart model, not patched in the XBRL generator.

2 · Ranger Tag Synchronisation with OpenMetadata

The plan was to tag columns as GDPR.PII in OpenMetadata and have Ranger automatically pick up those tags via the Ranger Tag Sync service to enforce column masking. In practice, the open-source version of Ranger Tag Sync (Apache Atlas integration) does not support OpenMetadata as a tag source out of the box — that integration requires Ranger 2.4+ with a custom service definition, or the commercial Cloudera version.

The workaround adopted here: OpenMetadata PII tags are informational (they drive the catalog UI and column-level lineage annotations), while Ranger policies are defined explicitly by column name in column_mask_policies.json. The two systems agree on what is PII — but they do not sync automatically. A CI step in the pipeline validates that every column tagged GDPR.PII in OpenMetadata has a corresponding Ranger mask policy, and fails the build if the two diverge.

# tools/check_pii_ranger_sync.py
# Run in CI — compares OM PII tags against Ranger mask policy coverage

import requests, json, sys

OM_URL    = "http://localhost:8585"
RANGER_URL = "http://localhost:6080"

# 1. Collect all columns tagged GDPR.PII in OpenMetadata
resp = requests.get(
    f"{OM_URL}/api/v1/search/query",
    params={"q": "tag:GDPR.PII", "index": "column_search_index", "limit": 100},
    headers={"Authorization": "Bearer <token>"},
)
pii_columns = {
    f"{hit['_source']['table']}.{hit['_source']['name']}"
    for hit in resp.json()["hits"]["hits"]
}

# 2. Collect all columns covered by Ranger mask policies
ranger_resp = requests.get(
    f"{RANGER_URL}/service/plugins/policies",
    params={"serviceName": "corep-trino", "policyType": "1"},   # type 1 = masking
    auth=("admin", "<password>"),
)
masked_columns = {
    f"{res['table']}.{col}"
    for policy in ranger_resp.json()["policies"]
    for res in policy["resources"].get("table", {}).get("values", [])
    for col in res.get("column", {}).get("values", [])
}

unprotected = pii_columns - masked_columns
if unprotected:
    print(f"[FAIL] PII columns with no Ranger mask: {unprotected}")
    sys.exit(1)
print("[PASS] All PII columns covered by Ranger mask policies.")

3 · Synthetic Data Consistency Across Six Tables

The six source CSV tables (capital_instruments, rwa_exposures, loans, counterparties, liquidity_assets, liquidity_outflows) are individually easy to generate. Making them internally consistent — so that the COREP templates computed from them produce plausible ratios that exercise all dbt code paths without triggering the GX regulatory floor checks — required careful reverse-engineering.

The constraints that had to be simultaneously satisfied:

The resolution: generate the data bottom-up. Start with the tier amounts (CET1, AT1, T2), compute Own Funds as their exact sum, derive RWA from a target CET1 ratio, then generate loans and exposures consistent with that RWA. A single Python script enforces all constraints at generation time so the pipeline never has to work around bad input data.

Running the Complete Pipeline — What the End-to-End Looks Like

The submission ZIP contains:

corep_submission_2025-12-31.zip
├── corep_c0100_2025-12-31.xbrl        ← XBRL instance (Own Funds)
├── corep_c0200_2025-12-31.xbrl        ← XBRL instance (RWA)
├── corep_c0300_2025-12-31.xbrl        ← XBRL instance (Capital Ratios)
├── corep_c4700_2025-12-31.xbrl        ← XBRL instance (LCR)
├── validation_report_2025-12-31.json  ← Arelle pass/fail with all error codes
├── gx_data_docs/                      ← GX HTML reports for raw + mart gates
└── manifest.json                      ← Pipeline run metadata, versions, hashes

What a Real Bank Would Need on Top of This

This project is a complete proof-of-concept. Moving it to a production environment at a regulated institution would require the following additions:

18-Day Retrospective: What the Journey Looked Like

By the Numbers

Why Open Source Is Sufficient for Regulatory Reporting

The most common objection to this stack in the banking industry is: “we can’t use open-source tools for regulatory reporting — we need vendor support.” That objection conflates two different concerns.

Vendor support is a procurement and risk management concern — if Arelle has a bug that generates invalid XBRL, who do you call? The answer is: the EBA’s own XBRL rendering service runs Arelle. Every major European bank uses Arelle for XBRL generation. The EBA itself publishes the taxonomy in Arelle-compatible format. For dbt, the company (dbt Labs) offers enterprise support contracts. For Airflow, Astronomer does. Apache Ranger and OpenMetadata both have commercial distributions (Cloudera CDP, Collibra).

The tools themselves are fit for purpose. What this project demonstrates is that the open-source stack can produce a submission-ready, regulator-auditable XBRL package with full end-to-end lineage, role-based access control, and adversarially-tested governance gates — at zero infrastructure cost. A real bank adds vendor support contracts and production hardening on top of the same architecture. The architecture itself does not change.

Tomorrow, the final blog: Day 18 — the production readiness checklist. What it takes to move from this Docker Compose proof-of-concept to a deployment that a Chief Data Officer would sign off on before a real ECB submission.