Archive
A collection of experiments, systems, and artifacts built for the digital realm.
Unity Catalog Migration
Executing a 1PB+ data migration from Hive Metastore to Unity Catalog. Deep dive into SYNC, Federation, and Security.
The Pipeline Guardian
Self-healing data pipeline agent hosted on Databricks Apps. Automates RCA and recovery for production failures.
ADF to Fabric Migration
Automated migration of Azure Data Factory pipelines to Fabric Data Factory using PowerShell.
Serverless to Fabric Migration
Migrating from Synapse Serverless SQL to Microsoft Fabric Lakehouse. Automating Delta shortcuts via PySpark.
Lakeflow Pipelines
Moving from Airflow to Databricks Lakeflow. Implementing Spark Declarative Pipelines (SDP) for 10TB+ streaming data.
Metadata Driven Framework
Orchestrating 500+ tables via ADF & Databricks using a single dynamic pipeline. Features Unity Catalog governance and Control Table logic.
Enterprise Data Platform
A centralized data lakehouse on AWS processing 2TB+ daily. Built with Terraform, Spark, and Iceberg.