Data Engineering · Azure & Databricks
ChileCompra Data Platform
End-to-end batch Data Engineering platform built on Azure to process more than 1.1 million purchase orders and 2.8 million line items from Mercado Público / ChileCompra, combining historical CSV ingestion with daily incremental ingestion from REST APIs, orchestrated with Azure Data Factory and Databricks Jobs.
Azure Data Factory
ADLS Gen2
Azure Databricks
PySpark
Spark SQL
Delta Lake
Unity Catalog
Azure Key Vault
GitHub Actions
- Implemented a Medallion architecture (Landing → Bronze → Silver → Gold) and automated the daily pipeline with Azure Data Factory and Databricks Jobs; purchase order and tender branches converge before five independent Gold transformations executed in parallel.
- Implemented idempotent and recoverable processing using process_date, run_id, replaceWhere, Delta MERGE and freshness rules, supporting reruns and backfills without generating duplicates or overwriting newer versions.
- Added critical pre-write and post-write Data Quality controls for keys, duplicates, statuses and referential integrity; critical inconsistencies stop the task and propagate the failure from Databricks to Azure Data Factory.
- Configured service-to-service security with Azure Key Vault, Managed Identity, Azure RBAC, Databricks Access Connector and Unity Catalog, and automated the refresh of a Databricks AI/BI dashboard only after all Gold tasks complete successfully.
1.1M+
Purchase orders processed
2.8M+
Line items processed
5
Parallel Gold transformations
All notebooks, Azure Data Factory pipelines and the analytics dashboard are version-controlled in GitHub.
