Databricks Platform Engineer
6 days ago
Barcelona
Overview In this role you will own and evolve Zurich’s Databricks platform to support the Global Finance Data Hub, ensuring scalable, secure, and cost-conscious data solutions. You will build and maintain ingestion and processing pipelines, govern data with Unity Catalog, and implement CI/CD and IaC practices in a collaborative, tech-driven environment. You’ll work with cross-functional teams to deliver a robust enterprise data platform that accelerates Zurich’s data strategy and operational efficiency. This is a chance to shape a world-class data foundation within a global insurer. Compensaciones / Beneficios • flexible working model, • abroad up to 25 days yearly, • home office allowance, • wide trainings including language classes, • health insurance, • life and retirement plans Responsabilidades, • Own and evolve the Databricks platform underpinning the Global Finance Data Hub with focus on scalability, reliability, security, and cost efficiency, • Design, build, and maintain data ingestion frameworks for batch, streaming, API, and CDC across multiple sources, • Develop and manage Delta Live Tables pipelines with data quality, schema enforcement, and CDC, • Administer and optimize Delta Lake environments (performance, schema evolution, time travel, VACUUM, OPTIMIZE), • Implement Unity Catalog governance (access controls, lineage, classification, namespaces), • Build, deploy, and support Databricks Workflows for complex pipelines and event-driven processing, • Establish CI/CD pipelines using Git and Databricks Asset Bundles across dev/test/prod, • Collaborate with cloud/infrastructure teams on IaC using CloudFormation or CDK, • Manage authentication/authorization (IAM roles, profiles, service principals, secrets), • Monitor platform health and costs with Databricks system tables, CloudWatch, Grafana, • Define cluster policies, compute strategies, and DBU governance for optimization, • Ensure security (PrivateLink, VPC endpoints, row/column security), • Partner with data engineering, finance, architecture, and product teams to deliver scalable data platform, • Drive best practices, automation, and continuous improvement across the data ecosystem Requisitos principales, • 5-8 years in Data Engineering/Platform Engineering/Cloud Data Architecture, • Minimum 3 years owning and managing Databricks platforms in enterprise environments, • Strong expertise in Databricks services (Unity Catalog, Delta Lake, Delta Live Tables, Workflows, Asset Bundles), • Experience designing/supporting batch, streaming, API, and CDC data ingestion frameworks, • Advanced PySpark, Python, and Databricks SQL for scalable pipelines, • DevOps practices including Git, CI/CD, automated testing, and IaC (CloudFormation or CDK), • AWS experience (S3, IAM, networking, security, secrets), • Experience with large-scale data platforms in regulated industries; financial services desirable, • Preferred: Databricks/AWS certifications; Airflow, Kafka/MSK; data governance frameworks, • collaboration, • effective communication, • problem-solving, • Databricks Unity Catalog, • Delta Lake, • Delta Live Tables