Databricks Architect
6 days ago
Leeds
Job Title: Databricks Architect Location: Leeds, England, United Kingdom Work Mode: Onsite Experience: 10–15 Years (minimum 5+ years in Databricks) Employment Type: Full-time About the Role We are seeking a highly experienced Databricks Architect to lead the design, development, and optimization of scalable data platforms and advanced analytics solutions. This role requires a deep understanding of Databricks Lakehouse architecture, big data engineering, cloud ecosystems, and enterprise data strategy. The ideal candidate will partner with business and technology leaders to deliver high-performance data systems that support analytics, AI/ML, and real-time insights. Key Responsibilities Architecture & Solution Design • Architect end-to-end Databricks Lakehouse solutions including data ingestion, ETL/ELT pipelines, Delta Lake, data warehousing, and data governance frameworks., • Define and enforce best practices, security standards, and performance optimization across Databricks workloads., • Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog, and MLflow., • Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build • Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta., • Oversee data quality frameworks, metadata management, lineage, and monitoring. Leadership & Stakeholder Engagement • Work closely with business leaders and product teams to translate requirements into robust technical solutions., • Guide development teams, perform architecture reviews, and ensure platform engineering excellence., • Conduct technical workshops, POCs, and roadmap planning for the Databricks environment. Performance, Security & Governance • Optimize Databricks clusters, query performance, and cost management., • Implement data governance standards using Unity Catalog, RBAC, and compliance guidelines (GDPR, ISO)., • Ensure resilience, scalability, and disaster recovery readiness. Required Skills & Qualifications • 10–15 years overall experience in data engineering or data architecture., • 5+ years of hands-on architecture experience specifically in Databricks., • Expert in Apache Spark, PySpark, Databricks SQL, Delta Lake, and distributed systems., • Strong understanding of Lakehouse architecture, BI ecosystems, and modern data platforms., • Cloud expertise in Azure / AWS / GCP (Azure preferred for many UK clients)., • Experience with CI/CD, Infrastructure-as-Code (Terraform preferred), and DevOps., • Proven experience leading teams and delivering enterprise-grade data solutions. Preferred Qualifications • Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Architect)., • Experience building ML solutions using MLflow or integrating with cloud ML services., • Experience in BFSI, Retail, Telecom, or Healthcare data environments., • Experience with real-time data processing (Kafka, EventHub, Kinesis). Soft Skills • Excellent communication and stakeholder management skills., • Strong problem-solving and solution-oriented mindset., • Ability to lead in a fast-paced and dynamic environment., • Collaborative, proactive, and able to provide strong technical direction. Why Join Us? • Work onsite with a high-impact, cutting-edge data team in London., • Opportunity to lead large-scale data transformation initiatives., • Exposure to enterprise data strategy, advanced analytics, and AI projects., • Competitive compensation and career-growth opportunities.