Big Data Engineer
hace 5 días
Madrid
Scope (functional and technical perimeter) The Senior Data Engineer supports internal and external Amadeus airline domain customers by designing, building, deploying, and maintaining Big Data and advanced analytics solutions on the Amadeus Big Data Platform. Scope includes: • Transformation of raw datasets into actionable, production ready data products., • Design, development, and automation of data pipelines (ETL/ELT, ML/AI pipelines)., • Collaboration with data scientists and platform teams to industrialize advanced analytics., • Contribution to platform tooling, configuration, and infrastructure definition., • Support to product, sales, consulting missions, and customer engagements., • Acting as reference engineer for best practices, mentoring, and technical guidance. 3.1.1 Planning (Phases, Delivery date) Typical phases covered by the role include: • Requirements gathering with stakeholders (internal/external)., • Architecture and solution design for Big Data components., • Development and deployment of pipelines, libraries, and analytics workflows., • Integration and testing (unit tests, pipeline validation, performance checks)., • Production rollout, monitoring, and continuous optimization (compute/storage cost control)., • Documentation and knowledge transfer to teams and customers. Delivery timelines depend on product scope, customer commitments, and project prioritization. 3.1.2 Activities/Deliverables/Tasks/ Core Activities • Transform raw data or combine existing datasets using Databricks, Spark, Snowflake, Power BI or similar tools., • Develop and maintain libraries for data preparation and feature engineering., • Design, implement, test, and automate data pipelines and workflows., • Support data scientists to operationalize and industrialize analytical/ML/AI models., • Contribute to platform reliability through tooling, maintenance, and configuration., • Provide expertise on Big Data architectures and influence platform design decisions., • Define infrastructure and services required for cloud based data solutions., • Participate in customer or internal workshops, and technical reviews., • Translate technical decisions into clear business implications for stakeholders., • Mentor junior engineers, • Keep documentation updated (technical, functional, architectural). Deliverables • Production ready pipelines (ETL/ELT, ML/AI)., • Data models, schemas, and metadata specifications., • Technical documentation and architectural recommendations., • Cost optimized compute/storage setups., • Tools, scripts, libraries, dashboards, and BI outputs. 3.1.3 Methodology and tools Methodologies • Agile development (SAFE, scrum), iterative delivery, code reviews, CI/CD., • Data engineering best practices, versioning, testing discipline., • Cloud native design and cost optimization principles. Tools & Technologies • Big Data & Cloud: Databricks, Spark, Azure (Azure ML, Databricks Workflows),., • Programming: Scala, Python, Java., • Pipelines & Automation: Jenkins, GitHub Actions., • Data & BI: Power BI, Snowflake, Tableau, Qlik, SQL/NoSQL databases,, • DevOps & Versioning: Git, Linux shell, CI/CD toolchains., • Integration & Streaming (when relevant): Kafka, Flink. 3.1.4 Skills (technical, languages…) Technical Skills • Big Data engineering (Spark, Databricks, Azure, Snowflake)., • Advanced programming skills in Scala, Python, Java., • Data modelling: star schemas, snowflake schema, normalized modelling, Inmon and Kimball data warehousing., • Experience with high volume / high dimensional data., • Design and automation of pipelines/workflows., • Experience with ML/AI pipelines and MLOps concepts., • DevOps experience: CI/CD, Git, containerization practices., • BI tools (Power BI, Tableau, Qlik)., • Knowledge of airline/travel industry data is a strong plus (Reservation, Inventory, Ticketing, DCS…). Domain skills • Understanding of airline, hospitality, loyalty, or travel industry datasets is a strong plus., • Additional strong plus is the knowledge of ti-tables framework and ti-tables data models. Languages • Fluent English, • additional languages (Spanish, German, French) beneficial. Soft Skills • Strong communication and ability to translate technical choices for non technical stakeholders., • Analytical mindset with flexible problem solving approach., • Collaboration with globally distributed teams., • Adaptability, autonomy, reliability, and customer focus., • Mentoring, guidance, and contribution to team development.