Lead Data Engineer
5 days ago
Norbiton
Lead Data Engineer (Databricks)Salary £65,000Type Full time, permanentHours 37.5 per weekLocation Remote or Hybrid UK, with occasional Richmond office attendance (a couple of times per month)OverviewWe are looking for a Lead Data Engineer to join the Retail Insight Data and Engineering team and own the architecture and development of our Databricks environment. This is a senior, hands on role where you will design and build scalable data infrastructure that underpins our products, while guiding a small team of engineers and defining platform standards, data architecture and engineering best practices supporting analytics, machine learning and AI across the business.Key responsibilities Lead the design, development and optimisation of a Databricks based data platform, ensuring reliability, performance and scalability Design enterprise scale data architectures on Databricks and cloud platforms such as Azure, AWS or GCP Define lakehouse standards, data modelling approaches and governance frameworks Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies Establish security, access control, data quality and compliance standards Optimise performance, scalability and cloud cost management Lead and mentor a small team of data engineers, setting technical standards and reviewing code Partner with data science, analytics and product teams to ensure the platform supports their work Own data quality, governance and observability across pipelines Support AI and ML enablement including feature stores and model deployment patterns Translate business requirements into technical solutions Contribute to platform roadmap, tooling selection and continuous improvementRequired experience Strong experience as a senior or lead data engineer building and scaling production data platforms Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows Python and SQL proficiency Infrastructure as code experience using Terraform, Bicep or CloudFormation Cloud data platform experience, ideally Azure including Azure Data Factory, ADLS Gen2 and Azure DevOps Strong understanding of data modelling, ETL and ELT patterns, distributed processing Experience with security, governance and data platform operations Experience leading or mentoring engineers, with strong stakeholder management Rigorous approach to data quality and testing Databricks certificationDesirable Unity Catalog, MLflow and Databricks Asset Bundles dbt or similar transformation frameworks Retail or CPG data such as POS, inventory, supply chain Streaming technologies such as Kafka or Event HubsEducation* Degree in computer science, data engineering or related field, or equivalent practical experienceTPBN1_UKTJ