mPokket Experience
Leading the Data Platform at mPokket
Tech Lead for the Data Platform team at mPokket, a consumer-lending fintech. The team owns the Databricks lakehouse and the AWS DMS replication feeding it — the tables that analytics, data science, and business teams all read from — plus the pipelines, email reports, compliance datasets, and audit submissions built on top.
20%
Databricks Cost Reduced
$187K
Annualised Savings
20 TB
Reclaimed from S3
Org-wide
Teams Served by the Lakehouse
Core Systems
Lakehouse Ingestion & Replication
- ·AWS DMS replication from production databases into Databricks
- ·Builds and maintains the Databricks tables every other team reads from
- ·Pipeline development, scheduling, and day-to-day maintenance
- ·Debugging data issues end to end, from source database to dashboard
- ·Pipeline code versioned and reviewed through GitLab
Tech
Process flow
- Production Databases
- AWS DMS Replication
- Databricks Lakehouse
- Curated Tables
- Analytics · Data Science · Business
Reporting, Compliance & Audit Data
- ·Scheduled reports delivered over email to business stakeholders
- ·Curated datasets for reporting, dashboards, and downstream analysis
- ·Compliance datasets and audit data submissions for a regulated lender
- ·One agreed set of numbers shared by analytics, data science, and non-tech teams
Tech
Process flow
- Curated Tables
- Scheduled Jobs
- Email Reports
- Dashboards
- Compliance & Audit Submissions
Databricks Cost & Storage Optimisation
The first problem I picked up: nobody could say where the Databricks bill was going. Databricks system tables made spend attributable per job and cluster, and the savings followed from what that exposed.
- ·Queried Databricks system tables to attribute spend by job, cluster, and workload
- ·Cut Databricks cost by ~20% — about $187K annualised — with AWS savings on top
- ·Tuned Delta VACUUM retention and cleared stale Parquet files
- ·Reclaimed ~20 TB of unnecessary data from S3
Tech
Process flow
- System Tables
- Spend Attribution
- Idle & Oversized Workloads
- Right-sizing
- VACUUM & Parquet Cleanup
- Ongoing Bill Tracking
Team Leadership & Delivery
- ·Owns platform architecture decisions for the data platform
- ·Sprint planning and workload management across the team
- ·Hiring for the data platform team
- ·Point of contact for analytics, data science, and business data requests
Scope
One month in, the rule from Apple still holds: find the thing nobody is watching and own it. At mPokket that was the Databricks bill — so the first thing I built was visibility into spend, and the first real win came straight out of it.