Lead Data Engineer
10 days ago
London
\n We are seeking an experienced Lead Data Engineer with deep expertise in Python, Databricks and Spark to lead the delivery of large-scale data engineering initiatives. \n This is a unique opportunity for a technical leader who enjoys combining hands-on engineering with project delivery, team leadership and stakeholder engagement in a complex enterprise environment. \n \n \n Client Details \n \n Our client is a leading international financial institution with a long-established presence across major global markets. Serving a diverse client base that includes corporates, financial institutions and investors, the organisation delivers a broad range of banking, financing and capital markets services. \n With significant investment in digital transformation and data-led innovation, the organisation is modernising its technology estate and expanding its enterprise data capabilities. Data plays a critical role in supporting business operations, regulatory obligations, analytics and strategic decision-making, making this an exciting opportunity to contribute to large-scale, business-critical data initiatives within a complex global environment. \n \n \n Description \n\n \n • Lead the end-to-end delivery of data engineering projects, ensuring successful outcomes against business objectives, timelines and quality standards\n, • Design, develop and optimise scalable data pipelines using Python, Databricks and PySpark\n, • Build and enhance ETL/ELT workflows, reusable frameworks and modern data engineering solutions\n, • Provide technical leadership and architectural guidance across data platform initiatives\n, • Design and govern Lakehouse architectures leveraging Databricks and Delta Lake\n, • Manage sprint planning, backlog prioritisation and delivery tracking within Agile teams\n, • Identify and mitigate delivery risks, dependencies and technical challenges\n, • Troubleshoot complex data, platform and performance-related issues\n, • Drive engineering best practices across coding standards, testing, performance optimisation and code reviews\n, • Act as a key interface between business stakeholders and engineering teams, translating requirements into scalable technical solutions\n, • Collaborate with architects, product owners and senior stakeholders to shape delivery roadmaps and technical direction\n, • Lead, mentor and develop a team of data engineers, fostering a culture of ownership and engineering excellence\n, • Ensure data quality, governance, reliability and security across data platforms\n, • Implement monitoring, logging and alerting capabilities to support operational excellence\n, • Champion modern engineering practices, including CI/CD, DevOps and automation\n, • Support production environments and drive continuous improvement across data solutions\n \n\n \n Profile \n \n You will bring: \n\n • 10+ years' experience in data or software engineering\n, • Proven experience as a Lead Data Engineer, Technical Lead, Engineering Lead, or similar hands-on leadership role\n, • Strong hands-on expertise in Python\n, • Extensive experience with Databricks, including workflows, notebooks and Delta Lake\n, • Strong experience with Apache Spark / PySpark\n, • Experience building and optimising enterprise-scale ETL pipelines\n, • Strong understanding of modern data architecture and Lakehouse concepts\n, • Advanced SQL and distributed data systems expertise\n, • Experience with cloud platforms such as AWS, Azure or GCP\n, • Experience working within Agile delivery environments\n, • Excellent stakeholder management and communication skills\n, • Ability to balance technical leadership with hands-on delivery responsibilities\n, • Experience delivering data engineering solutions within Banking, Financial Services, Capital Markets, Insurance, or other highly regulated enterprise environments would be highly advantageous\n\n Desirable experience includes: \n\n • Databricks certifications\n, • Kafka or Structured Streaming\n, • CI/CD and DevOps practices\n, • Airflow or Databricks Workflows\n, • Delta Lake optimisation techniques\n, • Docker and Kubernetes\n, • Exposure to machine learning pipelines\n\n \n \n Job Offer \n \n\n • Competitive day rate of £650-880 per day Inside IR35 (Umbrella)\n, • Initial 6-month contract\n, • Hybrid working arrangement - 2/3 days onsite\n, • Opportunity to lead strategic data transformation initiatives\n, • Access to modern cloud and data engineering technologies\n, • Significant influence over technical direction and delivery\n, • Collaborative, high-performing engineering environment\n\n