Lead Azure Databricks Platform Engineer / Architect
hace 9 días
London
Lead Azure Databricks Platform Engineer / ArchitectHands-on Platform Engineering | Serverless | FinOps | POSIT/RStudio Migration \n 6 Month contract \n Inside IR35 - 500 a day \n London/Hybrid \n \n Role Purpose\n We are seeking a highly experienced, hands-on Azure Databricks Platform Engineer / Architect to enhance and optimise an enterprise Data Platform. The role combines architecture with direct implementation: the successful candidate must be able to configure, develop, troubleshoot and optimise Azure Databricks rather than operate only at design or governance level. The role is centred on three outcomes: enabling and optimising Databricks Serverless, strengthening FinOps and platform controls, and enhancing the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio. Key Responsibilities1. Databricks Serverless Enablement and Optimisation\n\n • Assess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost.\n, • Enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines.\n, • Establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads.\n, • Implement compute policies, autoscaling, quotas, budget controls and operational guardrails.\n, • Measure cost and performance outcomes, identify idle or oversized compute, and recommend optimisation actions.\n2. FinOps and Enterprise Platform Controls\n\n, • Define and embed a practical FinOps operating model covering ownership, accountability, projects, environments, teams, applications and cost centres.\n, • Implement mandatory tagging and integrate validation into CI/CD so non-compliant resources are prevented from being provisioned.\n, • Provide granular cost attribution by workspace, project, application, workload, job and team/user where technically appropriate.\n, • Implement budget policies, thresholds, proactive alerts and usage reporting to prevent uncontrolled spend.\n, • Use platform usage and billing data to identify idle compute, inefficient workloads, unnecessary storage/data movement and cost anomalies.\n3. Databricks Discovery Zone and POSIT/RStudio Migration\n\n, • Enhance the Databricks Discovery Zone to support migration from POSIT/RStudio\n, • Enable application deployment, secure API integrations, external data ingestion, LLM integration, scheduling, BI connectivity, local IDE-based development and operational reporting.\n, • Define reusable onboarding and migration patterns that reduce technology sprawl while improving security, supportability and delivery speed.\n4. Data Engineering and Integration\n\n, • Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake.\n, • Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls.\n, • Design reusable integration patterns for REST APIs, SaaS platforms, databases, files, object storage, document repositories, enterprise applications and public/external data providers.\n, • Implement secure authentication and credential handling for external and internal integrations.\n, • Build end-to-end data flows from source through governed ingestion and curated layers to BI, ML or application consumption.\nRequired Hands-on Technical Skills\n Deep hands-on Azure Databricks implementation and troubleshootingDatabricks Serverless and compute/workload optimisationAzure identity, networking, security, secrets, monitoring and private connectivityDatabricks SQL, Delta Lake and performance optimisationPython, PySpark and SQLJobs/workflows, incremental processing, CDC and data qualityREST/API and external data integration patternsFinOps, cost attribution, tagging, budgets, monitoring and operational support Experience and Candidate Profile\n\n • Significant experience delivering enterprise Azure Databricks platforms in production environments.\n, • Demonstrable ability to move between architecture, implementation, debugging and optimisation without depending entirely on specialist engineering teams.\n, • Strong understanding of platform security, data governance, operational support and controlled delivery in regulated or complex enterprises.\n, • Experience working collaboratively with data engineers, data scientists, architects, security teams, platform teams and business stakeholders.\n, • Clear communication skills and the ability to document standards, patterns, decisions and operational guidance.\nHighly Desirable but not Mandatory\n\n, • POSIT/RStudio migration or consolidation experience.\n, • Migration of analytical/data science workloads (convert and migrate R development/Libraries to Databricks).\n, • AI/ML, LLM integration, model lifecycle, RAG/vector retrieval or model-serving experience.\n, • Large-scale enterprise platform transformation and regulated-industry experience.\n, • Strong cost optimisation and FinOps delivery experience across Azure and Databricks.\n