Databricks Architect
1 day ago
City of London
Job Title: Databricks Architect Location: London, UK Work Mode: Onsite Experience: 10–15 Years (minimum 5+ years in Databricks) Employment Type: Full-time About the Role We are seeking a highly experienced Databricks Architect to lead the design, development, and optimization of scalable data platforms and advanced analytics solutions. This role requires a deep understanding of Databricks Lakehouse architecture, big data engineering, cloud ecosystems, and enterprise data strategy. The ideal candidate will partner with business and technology leaders to deliver high-performance data systems that support analytics, AI/ML, and real-time insights. Key Responsibilities Architecture & Solution Design • Architect end-to-end Databricks Lakehouse solutions including data ingestion, ETL/ELT pipelines, Delta Lake, data warehousing, and data governance frameworks., • Define and enforce best practices, security standards, and performance optimization across Databricks workloads., • Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog, and MLflow., • Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta., • Work closely with business leaders and product teams to translate requirements into robust technical solutions., • Guide development teams, perform architecture reviews, and ensure platform engineering excellence., • Optimize Databricks clusters, query performance, and cost management., • Implement data governance standards using Unity Catalog, RBAC, and compliance guidelines (GDPR, ISO)., • 10–15 years overall experience in data engineering or data architecture., • 5+ years of hands-on architecture experience specifically in Databricks., • Expert in Apache Spark, PySpark, Databricks SQL, Delta Lake, and distributed systems., • Strong understanding of Lakehouse architecture, BI ecosystems, and modern data platforms., • Cloud expertise in Azure / AWS / GCP (Azure preferred for many UK clients)., • Experience with CI/CD, Infrastructure-as-Code (Terraform preferred), and DevOps., • Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Architect)., • Experience building ML solutions using MLflow or integrating with cloud ML services., • Experience in BFSI, Retail, Telecom, or Healthcare data environments., • Excellent communication and stakeholder management skills., • Strong problem-solving and solution-oriented mindset., • Ability to lead in a fast-paced and dynamic environment., • Work onsite with a high-impact, cutting-edge data team in London., • Opportunity to lead large-scale data transformation initiatives., • Exposure to enterprise data strategy, advanced analytics, and AI projects., • Competitive compensation and career-growth opportunities.