Delta Table Maintenance & S3 Storage Reclamation

Delta tables keep old Parquet file versions around until VACUUM removes them, and a lakehouse that nobody has been maintaining accumulates terabytes of files that no query can ever read again. I audited the tables behind the platform, tuned VACUUM retention to what the teams actually needed for time travel and recovery, and cleared the orphaned Parquet files left behind by rewrites and failed jobs. The cleanup removed roughly 20 TB of unnecessary data from S3 — AWS savings that sit on top of the ~20% Databricks cost reduction — and the maintenance now runs as a scheduled part of the platform rather than a one-off.