Databricks Cost Attribution & Optimisation

When I joined, nobody could say where the Databricks bill was going. Rather than start by reorganising the team, I started with the data: Databricks system tables expose billing usage per workspace, job, cluster, and SKU, so I queried them to attribute every dollar of spend to the workload that caused it. That attribution surfaced idle and oversized clusters, jobs running more often than their consumers needed, and compute that could be right-sized without changing what any downstream team saw. Acting on it cut Databricks cost by roughly 20%, about $187K on an annualised basis, with AWS storage savings on top (see the storage reclamation project). The same system-table queries now run as ongoing bill tracking, so the cost picture stays visible instead of being rediscovered once a quarter.