Principal Data Engineer
hace 29 días
Houston
Job DescriptionJoin the ECI Group's Professional Technical Services Network as a Principal Data Engineer. This is a Houston based a six-month contract role with an anticipated March 30, 2006 start date and ending on September 30, 2026. NOTE: You must be a US Citizen or Permanent US resident to qualify for the position. Roles and Responsibilities: The Principal Data Engineer is the one who conceptualizes and designs data foundations and end to end solutions including data platforms for the Shell Business to maximize value from data. The role helps create a data-driven thinking within the organization, not just within IT teams, but also in the wider business stakeholder community. A Principal Data Engineer is expected to be a subject matter expert, and an active thought partner in the organization's data strategy and the key driver to define the Vison and Data Strategy for robust and future ready IT solutions Role Objective This is not a standard data engineer role. We are looking for a deeply technical, hands-on individual contributor who can: • Diagnose performance, latency, and cost issues in a large-scale cloud data platform, • Take a top-down, platform-level view across multiple projects, • Improve architecture, efficiency, and cost optimization, not just write Spark code, • Act as a technical problem-solver and mentor, guiding other data engineersThis person is expected to make the platform better, not just execute tasks. Current Platform & Architecture Data Flow: • On-premise systems → Cloud (Azure), • Streaming ingestion → Azure Data Lake Storage (ADLS), • Data processed into two separate containers:, • Crude trading, • Product tradingTechnologies in Use:, • Qlik Replicate (formerly Attunity), • Streaming data from on-prem to Azure, • Azure Data Lake Storage (ADLS), • Databricks, • Delta Live Tables (DLT), • Spark / PySpark, • Python 1. Data Latency • High-volume streaming data, • End-to-end latency issues that need root-cause analysis2. Databricks / DLT Cost Spikes, • DLT costs are far higher than expected, • Known contributors:, • Very high data volume (expected), • Inefficient lookup logic used to split data into separate containers, • The current solution works but is not optimalNote: This role exists because generic recommendations are not enough. Key Characteristics • Hands-on technical delivery leader, • Visible community leader for data engineers, • Conducts quality, community & conference presentations/trainings, • Has vast array of technology skills with deep expertise in at least 2 core technologies (Azure, AWS, SAP), • A skilled communicator capable of speaking to both technical developers and business managers., • Able to drive change and improvement through a deep understanding of the inter-relationship between People, Process, Data and Technology, • Coordinate the change management process, incident management and problem management process., • Drive implementation efficiency and effectiveness across the pilots and future projects to minimize cost, increase speed of implementation and maximize value delivery., • Drives & contributes to community building initiatives like CoE, CoP.Mandatory skills:, • AWS/Azure/SAP – Master, • ELT – Master, • Data Modeling – Master, • Data Integration & Ingestion – Master, • Data Manipulation and Processing – Skill, • GITHUB, Action, Azure DevOps – Skill, • Data factory, Databricks, SQL DB, Synapse, Stream Analytics, Glue, Airflow, Kinesis, Redshift, SonarQube, PyTest – SkillProgramming & Data Skills, • Strong Python (mandatory), • PySpark (advanced, not basic), • Advanced SQL, • Complex queries, • Stored procedures, • Analytical logicOptional skills:, • Experience in project management, running a scrum team., • Experience working with BPC, Planning., • Exposure to working with external technical ecosystem., • Managing small teams. Please Note: The ECI Group uses AI‑assisted tools in the early stages of application review, with all decisions guided by human oversight and fully compliant with General Data Protection Regulation (GDPR) and ECI Group’s applicable data protection policies and Privacy Notice. Bottom of Form Powered by JazzHR tC8fKTz5bk