Data Engineer
hace 10 días
London
Data Engineer (Hybrid – London, UK) Overview An opportunity for a skilled and motivated Data Engineer to join a growing data engineering team within a fast-paced and innovative environment. This position is ideal for someone who enjoys designing robust, scalable data infrastructure using modern open-source technologies. The role offers exposure to diverse datasets, complex data challenges, and opportunities to influence technical direction and best practices. Key Responsibilities • Design, develop, and maintain data ingestion pipelines using open-source frameworks and tools., • Build and optimise ETL/ELT processes for small to large-scale data processing requirements., • Develop data models and schemas to support analytics, business intelligence, and product needs., • Monitor, troubleshoot, and optimise data pipeline performance and reliability., • Collaborate with analysts, engineers, and product teams to gather and translate data requirements., • Implement data validation and quality checks to ensure data integrity., • Participate in architecture discussions and contribute to technical roadmap planning. Skills and Attributes • Strong analytical mindset with exceptional attention to detail., • Excellent problem-solving and debugging skills., • Ability to work independently and manage multiple priorities effectively., • Clear and confident communication skills, with the ability to explain technical concepts to non-technical stakeholders., • Experience working in agile development environments., • Passion for continuous learning and staying current with emerging data technologies. Technical Requirements • Proficiency in SQL, including complex query design and optimisation., • Strong Python programming skills, particularly with libraries such as pandas, NumPy, and Apache Spark., • Experience building and maintaining data ingestion pipelines and optimising performance., • Hands-on experience with open-source data frameworks such as Apache Spark, Apache Kafka, or Apache Airflow., • Knowledge of distributed computing and big data concepts., • Experience using version control systems (Git) and CI/CD practices., • Familiarity with relational databases (PostgreSQL, MySQL, or similar)., • Experience with containerisation technologies (Docker, Kubernetes)., • Understanding of data orchestration tools (e.g., Airflow or Dagster)., • Knowledge of data warehousing principles and dimensional modelling., • Understanding of cloud platforms and infrastructure-as-code (IaC) practices., • Awareness of real-time streaming and data quality monitoring solutions. Preferred Qualifications • Bachelor’s degree in Computer Science, Engineering, Mathematics, or a related field., • 2–5 years of experience in data engineering or similar technical roles., • Familiarity with data processes and challenges in the energy or commodities sectors (beneficial but not required).