Senior Data Engineer
6 days ago
Sheffield
The Opportunity - (Fully Remote, Perm role) This is a greenfield data platform build. ___ is in the middle of a major AI transformation â a multiâagent copilot platform (KAI) is already live in production, serving manufacturing clients globally. The AI roadmap is ambitious (agentic AI capabilities, multiâagent orchestration, voice interfaces), and the next wave of capabilities depends on a strong data engineering foundation. A Databricks lakehouse POC is being evaluated (Unity Catalog, Delta Sharing, PrivateLink to RDS), and the team is assessing the right longâterm analytics platform strategy. This role builds the entire data engineering function from scratch â defining the architecture, tooling, and patterns that power the next generation of AI capabilities. This is an entirely forwardâlooking role. While there is an active platform modernisation underway (the engineering team is migrating a .NET platform from SQL Server to PostgreSQL and containerising for Kubernetes), this roleâs primary focus is building new data pipelines and analytics infrastructure, not maintaining legacy systems. That said, collaboration with the platform engineering team is important, as the data pipelines need to ingest from both the existing SQL Server estate and the new PostgreSQL databases. Core Technical Requirements ⢠Python â primary language for data pipelines, scripting, and integration with the AI stack. This is a Pythonâfirst team, ⢠SQL (SQL Server + PostgreSQL) â deep experience required across both engines. The platform runs a largeâscale multiâtenant database estate with perâclient schemas (local, global, corporate). The new platform targets Aurora PostgreSQL, with several modules already migrated. Pipelines must bridge both., ⢠Databricks / Apache Spark â a lakehouse POC is being evaluated with Unity Catalog and Delta Sharing via AWS PrivateLink. Experience with Databricks or equivalent analytics platforms is important for assessing and scaling the right approach, ⢠Apache Kafka â event streaming experience is essential. The platform is evaluating Kafka as the strategic backbone for ordered event logs, consumer group replay, and guaranteed partition ordering as the microservices fleet scales, ⢠ETL/ELT pipeline design â building and maintaining pipelines into a medallion architecture (bronze/silver/gold layers) on Databricks or equivalent analytics platform, ⢠Delta Lake â understanding of lakehouse table formats, time travel, and ACID transaction patterns, ⢠AWS data services â RDS, S3, Lambda, Glue or equivalent. The platform runs entirely on AWS ⢠Linux / CLI proficiency â must be comfortable working AI-Assisted Development Methodology -> Solvace is transitioning towards AIâassisted development as a core engineering practice. Candidates should demonstrate: ⢠Handsâon experience with AI coding tools â Claude Code, OpenAI Codex, GitHub Copilot, Cursor, or similar. Weâre looking for engineers who have integrated these tools into their professional workflow, not just experimented casually, ⢠Specâdriven development â ability to write clear technical specifications that can be used to drive both human and AIâassisted implementation, with strong evaluation criteria and test coverage, ⢠Portfolio evidence â professional projects or side projects that demonstrate AIâassisted development practices. Contributions to or experimentation with emerging projects like OpenClaw are a strong signal, ⢠Testing and evaluation rigour â experience building robust test suites, automated quality gates, and evaluation frameworks that ensure AIâassisted code meets production standards Nice-to-Have -> ⢠Apache Flink â for realâtime stream processing as the platform moves from batch to streaming analytics, ⢠Apache Iceberg â experience with open table formats alongside or as an alternative to Delta Lake, ⢠Go or Rust â valued as evidence of strong backend engineering and systems thinking, even though Python is the primary language, ⢠Multiâtenant data architectures â the platform runs perâclient schemas with tenant isolation requirements. Understanding tenant isolation patterns is a significant advantage, ⢠Terraform / IaC â infrastructure is managed via Terraform; the Databricks POC has been codified in Terraform, ⢠dbt or similar transformation framework, ⢠Manufacturing / industrial data domain experience (sensor data, quality metrics, OEE, SPC), ⢠CDC (Change Data Capture) â the platform is building CDC pipelines for the SQL Server to PostgreSQL transition, ⢠Weaviate or other vector database experience â currently used for semantic search in the AI agent layer Why Join? ⢠Greenfield data platform â the entire data engineering practice needs to be built from scratch. A lakehouse POC is being evaluated but the production architecture is theirs to define â patterns, tooling, and platform decisions from day one, ⢠Direct AI impact â every pipeline they build directly unblocks AI agent capabilities. The KAI copilot is live in production and evolving towards agentic AI â autonomous agents that take actions, orchestrate workflows, and operate across systems. This role provides the data foundation that makes agentic capabilities possible, ⢠Event streaming architecture â opportunity to design and implement a Kafkaâbased event backbone, shaping the platformâs realâtime data architecture, ⢠Manufacturing AI â opportunity to work at the intersection of industrial operations and generative AI, a domain with massive untapped potential. Rich, realâworld data: quality inspections, KPIs, maintenance records, safety observations, process parameters across global manufacturing sites, ⢠Small team, high autonomy â the Innovation Hub operates with startupâlevel autonomy inside a funded, growing company, ⢠Leadership trajectory â the data function is critical to the platform and will grow. As an early hire, there is an open career path that will grow with the function â earned, not guaranteed