Day 18 of 18 · COREP Governance Pipeline · Production Readiness
We built a working COREP reporting pipeline across 17 days — valid XBRL output, end-to-end lineage, Ranger row-column masking, GX quality gates, and an Airflow DAG that ties it together. A Docker Compose proof-of-concept that runs on a developer laptop is not the same thing a Chief Data Officer signs off on before submitting to the ECB. This final post is the honest gap analysis and the production checklist your CDO will actually ask for.
The Honest Gap: What the POC Has vs What Production Needs
Before writing a single Helm chart, you need to know the full scope of what changes. The table below maps each POC component to its production equivalent and the regulatory driver for the upgrade.
| Component | POC (Docker Compose) | Production Requirement | Regulatory Driver |
|---|---|---|---|
| Container orchestration | docker-compose.yml on one laptop | Kubernetes cluster (3+ nodes) with Helm charts, resource limits, liveness probes | DORA Art. 11 — ICT service continuity |
| PostgreSQL | Single container, no backups | Managed DB (RDS / Azure Database) — HA, PITR 35 days, AES-256 at rest, TLS in transit | EBA GL 2021/05 §5.7 — data integrity |
| Object storage | MinIO single-node, no object lock | S3 / Azure Blob with object lock (WORM), 7-year retention, versioning enabled | EBA Guidelines on data retention |
| Secrets management | .env files checked into Git | HashiCorp Vault / Azure Key Vault — dynamic secrets, rotation, audit log | BCBS 239 Principle 11 — data security |
| Authentication | Plain username/password in config | LDAP / Active Directory integration for Trino, Airflow, Ranger, OpenMetadata | EBA GL 2021/05 §7 — identity management |
| TLS / encryption | HTTP only (port 8080, 8088, etc.) | TLS everywhere — internal service mesh (mTLS) + external HTTPS with valid certs | DORA Art. 9 — ICT security |
| 4-Eyes approval | Not implemented | Airflow HumanInTheLoop sensor — preparer → reviewer → CFO sign-off chain | EBA ITS on supervisory reporting — submission governance |
| Audit log | pg_audit to stdout | Append-only audit table in PostgreSQL + streaming to SIEM, 7-year retention | BCBS 239 Principle 7 — audit trail |
| EBA taxonomy updates | Manual download, no versioning | Automated quarterly taxonomy diff + Git version pinning + dbt re-test gate | EBA ITS — taxonomy version compliance |
| Multi-entity reporting | Single hardcoded entity | legal_entity_id on all mart models, per-entity XBRL contexts, Ranger LEI row filter | ECB supervisory reporting — consolidated vs solo |
| Disaster recovery | None — restart the laptop | RTO 4 hours / RPO 24 hours with tested runbook, DR drill every 6 months | DORA Art. 11 — business continuity |
| Change management | Push to main branch | GitOps — PR → CI pipeline → staging → approval gate → production deploy | EBA GL 2021/05 §8 — change management |
| Monitoring & alerting | Airflow task logs only | Prometheus + Grafana dashboards, PagerDuty alerts, SLA breach notifications | DORA Art. 10 — ICT monitoring |
| Network isolation | Docker bridge network, all ports exposed locally | Kubernetes NetworkPolicy, private subnets, no public endpoints for data services | DORA Art. 9 §2 — network security |
| Data lineage coverage | OpenLineage events emitted, not verified complete | 100% XBRL concept coverage verified by automated lineage coverage test in CI | BCBS 239 Principle 3 — data lineage |
| Quality gate history | Single run, no trend | GX checkpoints pass rate 100% for 3 consecutive reporting periods before first live submission | BCBS 239 Principle 2 — data accuracy |
| Regulator connectivity | ZIP file saved to local disk | ONDB / EUCLID API integration with signing certificate and submission receipt validation | EBA ITS 2021/07 — electronic submission |
⚠️ 17 production gaps — none of them are optional. An ECB SREP team examining your submission infrastructure will check for all of these. A missing audit trail or absent 4-eyes approval is not a technical issue; it is a governance failure that can trigger a supervisory finding.
1. Infrastructure Hardening — Replacing Docker Compose
Container Orchestration
POC
docker-compose up -d on one machine
Production
Kubernetes 1.29+ with Helm charts, HPA, PodDisruptionBudgets, readiness probes
PostgreSQL — Managed DB
POC
postgres:16 container, data in Docker volume
Production
RDS Multi-AZ or Azure DB Flexible Server, PITR 35 days, AES-256, private subnet
Object Storage — WORM
POC
MinIO single-node, no object lock
Production
S3 with Object Lock (Compliance mode), 7-year WORM retention, versioning on
Secrets — Vault Integration
POC
.env files in Git (please don’t)
Production
HashiCorp Vault with dynamic PostgreSQL credentials, 90-day auto-rotation
Networking
POC
All ports exposed on 127.0.0.1
Production
Kubernetes NetworkPolicy — data plane services on private subnets only, Istio mTLS
CI/CD Pipeline
POC
Git push directly to main
Production
GitOps: PR → GitHub Actions CI → ArgoCD staging → approval gate → production sync
Helm Chart Structure
Each platform service gets its own Helm chart. The repository layout mirrors what you already have in platform-helm-charts/:
# platform-helm-charts/src/main/resources/
Chart.yaml
values.yaml
values_resources.yaml # CPU/memory limits per environment
templates/
trino-deployment.yaml
trino-service.yaml
nessie-deployment.yaml
marquez-deployment.yaml
openmetadata-deployment.yaml
ranger-deployment.yaml
airflow-deployment.yaml
superset-deployment.yaml
networkpolicy.yaml # restrict cross-service traffic
poddisruptionbudget.yaml # prevent simultaneous evictions
servicemonitor.yaml # Prometheus scrape config
2. Authentication & Identity — Real Users, Real Roles
The POC uses static usernames. Production requires every service to validate identities against your corporate directory. Here is the integration map:
| Service | POC Auth | Production Auth | Key Config |
|---|---|---|---|
| Trino | PASSWORD_FILE authenticator | LDAP authenticator bound to AD | http-server.authentication.type=LDAP + TLS required |
| Apache Ranger | admin / rangeradmin local account | LDAP sync — groups map to Ranger roles | ranger.usersync.ldap.url in ranger-usersync.properties |
| Apache Airflow | admin / admin | LDAP auth_backend, RBAC enabled | [webserver] rbac = True, FAB LDAP config in webserver_config.py |
| OpenMetadata | admin@open-metadata.org | SSO via OIDC (Okta / Entra ID) | openmetadata.yaml — authProvider: azure / okta |
| Apache Superset | admin / admin | OAuth2 / SAML2 via OIDC provider | superset_config.py — AUTH_TYPE = AUTH_OAUTH |
| PostgreSQL | postgres / postgres | IAM authentication (RDS) or Vault-issued dynamic credentials | pg_hba.conf: host all all 0.0.0.0/0 md5 → remove; use cert + IAM |
💡 Ranger policy evaluation uses the authenticated identity, not a header value. Until Trino is wired to LDAP, your row-level and column-level masking policies are enforced against service account names — not real human identities. This is the most common security gap found in COREP pipeline reviews.
3. The 4-Eyes Principle — A Regulatory Non-Negotiable
EBA supervisory reporting standards require that the person who prepares a regulatory submission is not the same person who approves it. This is the 4-Eyes (dual control) principle. Your Airflow DAG must enforce this workflow before the submission ZIP is finalised.
Airflow Approval Task Implementation
# dags/corep_pipeline_dag.py — approval chain addition
from airflow.sensors.external_task import ExternalTaskSensor
from airflow.operators.python import PythonOperatordef await_reviewer_approval(**ctx):
“””Block until reviewer sets Airflow variable corep_{run_id}_approved=True.”””
from airflow.models import Variable
run_id = ctx[‘run_id’]
var_key = f”corep_{run_id}_approved”
approved = Variable.get(var_key, default_var=“false”)
if approved.lower() != “true”:
raise AirflowException(“Awaiting reviewer approval — set Variable to true to proceed”)
approver = Variable.get(f”corep_{run_id}_approver”, default_var=“unknown”)
ctx[‘ti’].xcom_push(key=‘approver_identity’, value=approver)
# approver_identity ends up in the submission ZIP audit artefactawait_reviewer = PythonOperator(
task_id=‘await_reviewer_approval’,
python_callable=await_reviewer_approval,
retries=720, # poll every 5 min for up to 60 hours
retry_delay=timedelta(minutes=5),
dag=dag
)await_cfo = PythonOperator(
task_id=‘await_cfo_signoff’,
python_callable=await_cfo_approval,
retries=720,
retry_delay=timedelta(minutes=5),
dag=dag
)# Chain: validate_xbrl >> await_reviewer >> await_cfo >> finalise_submission_zip
validate_xbrl >> await_reviewer >> await_cfo >> finalise_zip
Approval Workflow — Sequence
| Step | Actor | Action | Audit Evidence |
|---|---|---|---|
| 1 | Data Engineer (preparer) | Triggers DAG run, monitors quality gates and XBRL validation | Airflow task log with authenticated LDAP user ID |
| 2 | Risk Data Owner (reviewer) | Reviews Superset dashboard, sets Airflow Variable corep_{run_id}_approved=true | Variable set timestamp + Airflow user audit log |
| 3 | CFO or delegate | Final sign-off — sets corep_{run_id}_cfo_approved=true | XCom value stored in submission ZIP manifest JSON |
| 4 | System (Airflow) | Finalises ZIP, uploads to S3 WORM bucket, triggers EUCLID API | S3 object ETag + EUCLID submission receipt ID in audit log |
4. EBA Taxonomy Update Process
The EBA publishes new XBRL taxonomies quarterly. Each release can change concept labels, add new validation rules, or deprecate templates. Without a controlled update process, a taxonomy upgrade can silently break your COREP generation and you will only find out when Arelle throws errors.
# tools/update_eba_taxonomy.py
import hashlib, json, subprocess
from pathlib import PathEBA_TAXONOMY_URL = “https://www.eba.europa.eu/risk-analysis-and-data/reporting-frameworks/corep-3.4”def download_and_diff(new_zip_path: str, current_version_file: str = “taxonomy_version.txt”):
current = Path(current_version_file).read_text().strip()
new_hash = hashlib.sha256(Path(new_zip_path).read_bytes()).hexdigest()
if new_hash == current:
print(“Taxonomy unchanged — no action required”)
return
# Extract and diff concept labels
diff = subprocess.run(
[“python”, “-m”, “arelle.CntlrCmdLine”, “–validate”, new_zip_path,
“–conceptsFile”, “new_concepts.json”],
capture_output=True, text=True
)
# Write new version pin — triggers CI dbt re-test gate
Path(current_version_file).write_text(new_hash)
print(f”Taxonomy updated. Run ‘dbt test’ and ‘python -m xbrl.validate_instance’ to validate.”)
💡 Pin the taxonomy version hash in
taxonomy_version.txt and commit it to Git. Any PR that changes the hash must pass: (1) full dbt test suite, (2) Arelle validation against the new taxonomy, and (3) manual review by the Risk Data Owner. This is your change management gate for taxonomy updates.5. Multi-Entity Reporting Architecture
Banks report both solo (individual legal entity) and consolidated (group) COREP. Your pipeline must support multiple legal entities with strict data isolation — a data engineer for Entity A must never see Entity B’s raw exposures.
Data Model Changes
— All mart models add legal_entity_id dimension
ALTER TABLE mart.capital_ratios ADD COLUMN legal_entity_id VARCHAR(20); — LEI format
ALTER TABLE mart.credit_risk_exposures ADD COLUMN legal_entity_id VARCHAR(20);
ALTER TABLE mart.liquidity_ratios ADD COLUMN legal_entity_id VARCHAR(20);— Ranger row-filter policy: restrict each team to their entity
— Policy expression on mart.* tables:
— legal_entity_id = get_user_attribute(‘lei_code’)
— (lei_code is an LDAP user attribute set by IAM team)
# xbrl/xbrl_gen.py — per-entity context
def build_xbrl_context(entity_lei: str, period_end: str) -> dict:
return {
“id”: f”ctx_{entity_lei}_{period_end}”,
“entity”: {
“identifier”: entity_lei,
“scheme”: “http://www.gleif.org/data/schema/leidata/2016”
},
“period”: {“instant”: period_end}
}
6. Immutable Audit Trail — 7 Years, No Exceptions
EBA data retention guidelines and national supervisory requirements mandate a 7-year immutable audit trail for regulatory submissions. “Immutable” means no row can be deleted or modified after it is written — not even by a DBA.
PostgreSQL Append-Only Audit Table
— audit/001_create_append_only_log.sql
CREATE TABLE audit.pipeline_run_log (
id BIGSERIAL PRIMARY KEY,
run_id TEXT NOT NULL,
event_type TEXT NOT NULL, — DAG_START, QUALITY_PASS, XBRL_VALID, APPROVAL, SUBMISSION
actor TEXT NOT NULL, — authenticated LDAP user or ‘system’
payload JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);— Revoke modification rights from ALL roles including superuser via trigger
CREATE OR REPLACE FUNCTION audit.deny_mutation() RETURNS TRIGGER AS $$
BEGIN
RAISE EXCEPTION ‘audit.pipeline_run_log is append-only — UPDATE/DELETE forbidden’;
END; $$ LANGUAGE plpgsql;CREATE TRIGGER trg_audit_immutable
BEFORE UPDATE OR DELETE ON audit.pipeline_run_log
FOR EACH ROW EXECUTE FUNCTION audit.deny_mutation();— S3 lifecycle rule: transition to Glacier after 90 days, expire after 7 years
SIEM Integration
Streaming audit events to a SIEM (Splunk / Microsoft Sentinel) gives your security team real-time visibility into who accessed what data and who approved which submission. Configure Airflow’s AuditLogPlugin and PostgreSQL’s logical replication to feed events to Kafka → SIEM.
7. Disaster Recovery — Meeting the COREP Deadline After an Outage
COREP submissions have hard deadlines (typically T+15 business days after quarter end). Your DR plan must guarantee you can recover and re-run the full pipeline within the submission window.
| DR Metric | Target | How Achieved |
|---|---|---|
| Recovery Time Objective (RTO) | 4 hours | Kubernetes pod restart + RDS failover is <5 min; 4 hours allows full DAG re-run from checkpoint |
| Recovery Point Objective (RPO) | 24 hours | RDS automated daily snapshot + continuous WAL archiving to S3 |
| Backup verification | Weekly automated restore test | GitHub Actions job restores last snapshot to ephemeral RDS, runs smoke test suite |
| DR drill | Every 6 months | Full failover simulation with documented runbook, results filed in audit log |
| Runbook location | Git-versioned, accessible without VPN | docs/DR_RUNBOOK.md in repo, mirrored to Confluence and printed copy in server room |
⚠️ DORA Article 11 requires ICT business continuity testing results to be available to supervisors upon request. Your DR drill reports must be stored in the immutable audit bucket alongside your COREP submission artefacts.
8. Monitoring, Alerting & SLA Enforcement
The Airflow SLA callbacks you implemented in Day 13 are the first layer. Production adds infrastructure-level monitoring via Prometheus and Grafana, with PagerDuty escalation for any breach within the T+15 submission window.
| Metric | Threshold | Alert Channel |
|---|---|---|
| COREP DAG SLA breach | Any task > defined SLA | PagerDuty — Risk Data Owner |
| GX checkpoint failure rate | > 0% on any reporting period | Slack #corep-data-quality + Jira ticket auto-created |
| Arelle XBRL validation errors | Any error | PagerDuty — Data Engineer on-call |
| Ranger policy sync lag | OpenMetadata PII tag > 4 hours without matching Ranger policy | Slack #data-security |
| RDS storage utilisation | > 80% | CloudWatch alarm → PagerDuty |
| 4-eyes approval timeout | Pending approval > 48 hours before submission deadline | Email + SMS to CDO |
| Taxonomy version mismatch | Deployed taxonomy hash ≠ Git-pinned hash | CI pipeline failure → block deployment |
9. Regulator Submission Integration
The final step the POC leaves unimplemented: actually sending the XBRL instance document to the regulator. In the EU, COREP is submitted via the EBA’s EUCLID platform or the national competent authority’s (NCA) equivalent portal.
# xbrl/submit_euclid.py — skeleton for EUCLID API submission
import requests
from pathlib import Pathdef submit_to_euclid(
zip_path: str,
client_cert: str, # mTLS client certificate — issued by your NCA
client_key: str,
euclid_endpoint: str,
reporting_entity_lei: str
) -> dict:
with open(zip_path, ‘rb’) as f:
response = requests.post(
euclid_endpoint,
files={‘file’: (‘corep_submission.zip’, f, ‘application/zip’)},
data={‘lei’: reporting_entity_lei},
cert=(client_cert, client_key),
verify=True, # never disable TLS verification
timeout=120
)
response.raise_for_status()
receipt = response.json()
# Store receipt ID in immutable audit log
log_audit_event(‘SUBMISSION_ACCEPTED’, payload={
‘receipt_id’: receipt[‘submissionId’],
‘timestamp’: receipt[‘receivedAt’],
‘lei’: reporting_entity_lei
})
return receipt
🏛️ mTLS client certificates for EUCLID are issued by your National Competent Authority. The application process typically takes 4–8 weeks. Start this in parallel with your infrastructure work — it is the longest lead-time item on the production checklist.
The CDO’s Sign-Off Checklist
This is the artefact your CDO will ask for. Print it, fill in the evidence column, and file it in your governance register alongside the submission package. Every cell in the Status column must be COMPLETE before the first live ECB submission is authorised.
| Category | Checklist Item | Evidence Required | Status |
|---|---|---|---|
| Infra | All services running on Kubernetes with resource limits and health probes | kubectl get pods –all-namespaces output showing all Running | REQUIRED |
| Infra | PostgreSQL on managed service with HA, PITR enabled, AES-256 at rest | RDS/Azure DB configuration screenshot + encryption status export | REQUIRED |
| Infra | Object storage WORM policy enabled with 7-year retention (Compliance mode) | S3 Object Lock configuration JSON + Bucket Policy JSON in Git | REQUIRED |
| Infra | All secrets in Vault — no plaintext credentials in Git or environment variables | Vault audit log showing secret access + git-secrets scan output (zero hits) | REQUIRED |
| Infra | TLS enforced on all service endpoints — no HTTP plaintext | nmap or testssl.sh scan report showing all ports TLS-only | REQUIRED |
| Security | All services authenticating against LDAP/AD — no static admin passwords | LDAP integration test results for Trino, Ranger, Airflow, OpenMetadata, Superset | REQUIRED |
| Security | Ranger policies peer-reviewed by CISO and signed off | Policy export JSON in Git + CISO sign-off email archived in document management | REQUIRED |
| Security | OpenMetadata PII tags fully synced to Ranger masking policies (zero gaps) | tools/check_pii_ranger_sync.py CI run output showing 0 unprotected PII columns | REQUIRED |
| Security | Penetration test completed on data platform endpoints | Pen test report (internal or third party) with all Critical/High findings remediated | REQUIRED |
| Audit | Append-only audit log with mutation trigger deployed | pg_dump of audit schema + trigger DDL + proof that UPDATE raises exception | REQUIRED |
| Audit | Audit events streaming to SIEM | SIEM dashboard screenshot showing COREP pipeline events with <5 min latency | REQUIRED |
| Governance | End-to-end data lineage for every XBRL concept traceable to source column | Marquez lineage snapshot JSON included in submission ZIP for each concept | REQUIRED |
| Governance | Lineage coverage test passing in CI — 100% XBRL concept coverage | CI pipeline run showing tests/test_lineage_coverage.py: PASSED | REQUIRED |
| Governance | GX checkpoint pass rate 100% for 3 consecutive reporting periods | GX HTML data docs for last 3 quarters stored in MinIO WORM bucket | REQUIRED |
| Governance | Business data glossary complete in OpenMetadata — all mart columns described | OpenMetadata export showing 0 undescribed columns in mart schema | REQUIRED |
| XBRL | Zero Arelle validation errors for 3 consecutive test submissions | Arelle validation report JSON for last 3 test runs stored in MinIO | REQUIRED |
| XBRL | EBA taxonomy version pinned in Git and change process documented | taxonomy_version.txt in repo root + taxonomy change runbook in docs/ | REQUIRED |
| XBRL | XBRL calculation linkbase consistency verified — all parent totals match children | Arelle calculation validation report showing zero calc inconsistencies | REQUIRED |
| Governance | 4-Eyes approval workflow deployed and tested — preparer ≠ approver verified | Airflow task log from test run showing two distinct authenticated LDAP identities | REQUIRED |
| Governance | CFO sign-off chain configured and identity logged in submission artefact | Submission ZIP manifest.json showing approver_identity and cfo_approver fields populated | REQUIRED |
| DR | RTO/RPO targets documented and tested | DR runbook (docs/DR_RUNBOOK.md) + last DR drill report in audit bucket | REQUIRED |
| DR | Automated backup restore test passing weekly in CI | GitHub Actions workflow output for backup-restore-test job — last 4 weeks green | REQUIRED |
| DR | DR drill completed within last 6 months | DR drill report filed in governance register and linked from submission package | REQUIRED |
| Infra | EUCLID mTLS client certificate obtained and tested against NCA staging environment | Successful test submission receipt from NCA staging EUCLID endpoint | REQUIRED |
| Governance | Multi-entity Ranger row filters tested — Entity A user cannot see Entity B data | Security test results: tests/test_security_policies.py::test_entity_isolation PASSED | REQUIRED |
| Governance | Change management process documented — taxonomy/model changes require PR + approval | Branch protection rules screenshot + CODEOWNERS file showing Risk Data Owner on mart/ path | REQUIRED |
| Governance | User access review completed — all Ranger roles verified with HR employee list | Access review sign-off sheet with CISO and Risk Data Owner signatures | REQUIRED |
Regulatory Framework References
The production requirements above are not arbitrary engineering preferences. Each maps to a specific regulatory obligation:
| Regulation / Guideline | Relevant Articles / Principles | Pipeline Requirement |
|---|---|---|
| BCBS 239 (Principles for Risk Data Aggregation) | Principle 2 — Data accuracy Principle 3 — Completeness Principle 6 — Adaptability Principle 7 — Accuracy (reporting) Principle 11 — Data governance | GX quality gates, lineage coverage test, multi-entity support, SIEM audit trail, Vault secrets management |
| EBA GL/2021/05 (Internal Governance) | §5.7 Data integrity §7 Identity management §8 Change management | Append-only audit log, LDAP auth integration, GitOps PR approval for model changes |
| DORA (Digital Operational Resilience Act) | Art. 9 — ICT security Art. 10 — ICT monitoring Art. 11 — Business continuity Art. 30 — ICT third-party risk | TLS everywhere, Prometheus alerting, RTO/RPO DR plan, open-source component inventory |
| EBA ITS 2021/07 (Supervisory Reporting) | Electronic submission requirements Taxonomy version compliance XBRL technical standards | EUCLID mTLS integration, taxonomy version pinning, Arelle zero-error gate |
| ECB SREP Methodology | Data quality assessment Governance arrangements | 4-Eyes approval workflow, CDO sign-off evidence package, data lineage for every XBRL fact |
Realistic Timeline to Production
| Phase | Duration | Key Deliverables |
|---|---|---|
| Phase 1 — Infrastructure | 6–8 weeks | Kubernetes cluster provisioned, Helm charts deployed, managed DB live, Vault integrated, TLS everywhere |
| Phase 2 — Identity & Security | 4–6 weeks | LDAP integration for all services, Ranger LDAP group sync, PII masking re-tested with real AD identities, pen test completed |
| Phase 3 — Governance Controls | 4–5 weeks | 4-Eyes workflow, append-only audit log, SIEM integration, multi-entity row filters, access review |
| Phase 4 — EUCLID Connectivity | 4–8 weeks (NCA lead time) | mTLS certificate obtained, staging submission tested, receipt validation in pipeline |
| Phase 5 — Parallel Run | 3 reporting periods | Run pipeline in parallel with existing submission process, validate GX pass rate and XBRL zero errors across 3 consecutive quarters |
| Phase 6 — CDO Sign-Off & Go Live | 2 weeks | All 27 checklist items evidenced, CDO sign-off obtained, first live ECB submission via new pipeline |
💡 Total realistic timeline: 6–9 months from POC to first live submission. The longest lead-time items are the EUCLID certificate (start immediately) and the parallel run (3 full reporting periods required by most NCAs before accepting a new submission system).
What the CDO is Actually Signing
When the CDO puts their name on the submission, they are attesting that: (1) the data lineage from source to XBRL fact is complete and auditable, (2) dual control was applied and the approver identity is logged, (3) the data quality checks passed for this and the prior two periods, (4) the XBRL instance is valid against the current EBA taxonomy, and (5) the audit trail is immutable and available for supervisory inspection for 7 years. The 27-item checklist above is the evidence package that supports every one of those attestations.
Key Takeaways — Day 18
- A Docker Compose POC proves the architecture works. It does not prove it is safe, auditable, or resilient enough for a real ECB submission — 17 specific gaps exist between the two states.
- The longest lead-time item is the EUCLID mTLS client certificate from your NCA — apply for it on Day 1 of your production build, not at the end.
- 4-Eyes approval is a regulatory requirement, not a nice-to-have. The approver identity must be logged in the immutable audit trail and included in the submission artefact.
- Ranger policy enforcement only works with real authenticated identities. LDAP integration for Trino is the security upgrade that makes every column masking and row filter policy actually meaningful.
- The 7-year WORM retention requirement means your S3 Object Lock policy must be configured in Compliance mode — not Governance mode. Compliance mode cannot be overridden even by the bucket owner.
- EBA taxonomy updates are a quarterly operational process, not a one-time setup. Build the taxonomy update pipeline as a first-class CI workflow before go-live.
- Plan for a 3-period parallel run. Most NCAs will not accept a new submission system without evidence that it produced identical output to the incumbent system for three consecutive quarters.
- The CDO sign-off checklist (27 items) is the governance artefact that protects the CDO personally. Every item needs written evidence, not just a verbal confirmation.
Series Complete — EU Banking Data Governance in 18 Days
Day 1: Data Governance Fundamentals · Day 2: PostgreSQL Audit Logging · Day 3: Data Catalogue with OpenMetadata · Day 4: Apache Ranger Policies · Day 5: Data Quality with Great Expectations · Day 6: dbt Data Transformation · Day 7: Data Lineage with Marquez · Day 8: Lakehouse with Trino + Nessie + MinIO · Day 9: COREP Regulatory Taxonomy · Day 10: Synthetic Test Data · Day 11: XBRL Generation · Day 12: XBRL Formula Validation · Day 13: Airflow Orchestration · Day 14: End-to-End Data Lineage · Day 15: Superset Regulatory Dashboard · Day 16: Testing the Pipeline · Day 17: Capstone Retrospective · Day 18: Production Readiness Checklist
Tags: COREP, ECB, EBA, Regulatory Reporting, XBRL, Data Governance, Production Readiness, CDO, Kubernetes, Helm, HashiCorp Vault, Apache Ranger, Apache Airflow, 4-Eyes Principle, BCBS 239, DORA, EBA GL 2021/05, EUCLID, Data Lineage, Great Expectations, OpenMetadata, Immutable Audit, WORM, MinIO, PostgreSQL, Trino, dbt, Open Source, EU Banking, Data Engineering, Docker Compose to Production