Expert - Data Engineer (Cross Domain)
hace 8 horas
Barcelona
Overview In this Lead Data Engineer role, you will own end-to-end data pipelines across Bronze, Silver and Gold layers, enabling scalable data products for the Procurement domain. You will drive DevOps-centric deployment practices in Azure DevOps and GitHub, collaborating with modelers, stewards, and business partners to deliver production-grade solutions. You’ll work with dbt, Databricks, PySpark, and CI/CD automation to shape data architecture and governance at scale. This position offers leadership opportunities, cross-functional impact, and exposure to cutting-edge data engineering practices in a global setting. Compensaciones / Beneficios • growth opportunities, • global teamwork, • culture of collaboration, • learning and development, • voice and influence in shaping future, • opportunity to impact millions Responsabilidades, • Design and implement end-to-end data pipelines for ingestion, transformation, and storage across Bronze, Silver, and Gold layers, • Develop and manage data ingestion processes from source systems via ETL and API-based methods, • Build modular transformation frameworks using dbt, PySpark, and SQL with staging models and load metadata in alignment with Data Vault 2.0, • Design, build, and maintain CI/CD and DevOps processes in Azure DevOps and GitHub Actions with automated testing and governance, • Implement IaC and deployment automation for data platform resources (dbt, Databricks, clusters, dependencies), • Establish monitoring, observability, and operational excellence for pipeline reliability and rapid issue resolution, • Drive data quality, governance, and compliance with automated testing, lineage, and FAIR data principles, • Provide technical leadership and cross-functional collaboration across data engineers, modelers, stewards, BI developers, data scientists, and business SMEs Requisitos principales, • Strong technical expertise in dbt, SQL, Python, Spark (PySpark), Databricks, Git, Azure DevOps, and GitHub, • Proven experience designing and maintaining production-grade data pipelines (5+ years), • Deep CI/CD and automation experience with Azure DevOps Pipelines and GitHub Actions; IaC knowledge (Terraform, Bicep) is a plus, • Advanced Git and repository management; governance across Azure DevOps and GitHub, • Cloud data engineering experience (Azure) with Databricks jobs, clusters, and workflows, • Extensive knowledge of Data Vault 2.0 architecture (Raw Vault, Business Vault, Gold-layer models), • Experience with data ingestion patterns (batch ETL, incremental, CDC, API-based ingestion), • Strong focus on data governance and quality; exposure to scientific datasets is a plus, • Leadership mindset with ability to lead teams, manage projects, and collaborate across functions and geographies, • leadership, • collaboration, • curiosity, • dbt, • SQL, • Python