Databricks Lead Engineer (Contract)
? Central London (Hybrid – 3 days per week in office)
? £400–£500 per day (Outside IR35)
? 6-month rolling contract
Overview
An exciting opportunity has arisen for an experienced Databricks Lead Engineer to join a cutting-edge data engineering programme within a fast-growing, data-driven environment. This is a hands-on role focused on leading a large-scale re-engineering initiative, transforming complex data architecture into a modern, scalable platform.
The Role
You will play a key role in the migration and redesign of a high-volume data platform, moving towards a Delta Lake-based architecture. Working at significant scale, you’ll help shape the future of data storage, processing, and delivery, ensuring performance, reliability, and scalability across the platform.
Key responsibilities include:
- Designing and implementing a scalable Delta Lake architecture
- Modelling and partitioning high-cardinality datasets (e.g. order book data)
- Implementing metadata management, compaction, and versioning strategies
- Supporting migration from legacy data platforms to Delta Tables and Iceberg-compatible structures
- Designing systems to support multiple downstream data delivery use cases
- Establishing robust backup and recovery strategies (targeting 1-day RTO)
- Acting as a technical lead while remaining hands-on in engineering delivery
Key Skills & Experience (Essential)
- Strong commercial experience with Databricks
- Expertise in Delta Lake, Delta Tables, and Unity Catalog
- Experience with Delta UniForm (Universal Format)
- Hands-on experience with Apache Iceberg
- Strong AWS experience (including S3 and Lake Formation)
- Linux-based development experience
- Familiarity with modern engineering practices (CI/CD, testing, version control)
- Ability to work autonomously and drive delivery
- Strong problem-solving and communication skills
Desirable Skills
- Experience with Snowflake
- Experience working with petabyte-scale datasets
- Proficiency in Python
- Exposure to financial or market data environments
Why Apply?
- Work on a high-impact, large-scale data transformation project
- Opportunity to shape architecture decisions in a complex data environment
- Hybrid working with a Central London base
- Competitive day rate outside IR35
If you’re a Databricks expert looking for your next contract with real technical ownership and scale, this role offers a fantastic opportunity to make a tangible impact.
