Senior Technical Lead
2 days ago
Norwich
Job Title: Senior Technical Lead Location: Norwich, Norfolk (3 Days a week) Job Type: Contract (Inside IR35) Duration: 6 Months The Role We are looking for a Senior Technical Lead who combines hands-on engineering excellence with strong leadership and stakeholder management. You will own the end-to-end technical delivery of data platforms and pipelines built in AWS-with a focus on AWS Glue, Managed Workflows for Apache Airflow (MWAA), and Python-and collaborate closely with Directors, Senior Architects, and Program Leadership to deliver business outcomes at scale. This is a player-coach role: you will design, build, review, and optimize complex data workflows while mentoring engineers and driving engineering best practices. Ideal for: Someone who has delivered multiple production programs in a modern AWS data engineering landscape, can communicate trade-offs clearly to senior stakeholders, and can lead teams through ambiguity to predictable, high-quality outcomes. Your Responsibilities: * Lead the design and implementation of scalable, secure, and cost-efficient ETL/ELT pipelines using AWS Glue, Python (PySpark), and MWAA (Airflow). * Define solution architectures, data models, orchestration patterns, and CI/CD for data workflows. * Own the technical roadmap, decomposition, and delivery plan-including sizing, sprint planning, and risk mitigation. * Drive performance optimization (e.g., partitioning strategies, Glue job tuning, job bookmarks, dynamic frames vs DataFrames, retry/backoff strategies in Airflow). * Ensure robust observability (logging, metrics, tracing) and data quality (unit tests, Great Expectations/Deequ-style checks, validations). * Act as the technical point of contact for Senior Architects, and Program Managers; translate business needs into technical designs and delivery milestones. * Present architecture decisions, trade-offs, and TCO to senior stakeholders with clarity, data, and rationale. * Manage vendor/partner coordination where relevant. * Establish coding standards, code review practices, branching strategies, and secure-by-design principles. * Implement DevSecOps for data: infrastructure-as-code (IaC), secrets management, environment promotion, and automated testing. * Ensure compliance with data governance, security, and regulatory requirements (e.g., PII/PCI, encryption, auditability, lineage). * Mentor and upskill engineers; foster a culture of learning, ownership, and continuous improvement. Your Profile Essential skills/knowledge/experience: * 10+ years of total experience in software/data engineering, with 5+ years leading delivery of production solutions in an AWS data engineering environment. Advanced hands-on expertise with: * Python (including PySpark & data engineering patterns) * AWS Glue (Jobs, Crawlers, Glue Studio, Glue Catalog, PySpark, Job bookmarks) * MWAA (Apache Airflow) (DAG design, scheduling, sensors, retries, XComs, task isolation, best practices) Strong across broader AWS services: * S3, Lambda, Step Functions, IAM, CloudWatch, KMS, Secrets Manager, Athena, EMR (nice to have), Redshift (nice to have) * Proven experience delivering multiple end-to-end programs (architecture build test deploy operate) with measurable outcomes (SLAs, cost targets, performance). * Excellent stakeholder communication and executive presence; able to engage Directors, Senior Architects, and Program Leadership. * Solid grounding in data modeling, data governance, security/compliance, and cost optimization on AWS. * Experience with CI/CD (e.g., CodePipeline/GitHub Actions/Bitbucket Pipelines), IaC (CloudFormation/Terraform), and containerization (Docker). * Architectural thinking: designs for scale, reliability, cost, and evolvability. * Delivery excellence: breaks down complex work, sets milestones, manages risks, and delivers on time. * Communication & influence: distills complexity for senior stakeholders; backs decisions with data. * Hands-on leadership: sets the technical bar through reviews, pairing, and exemplars. * Ownership & clarity: aligns teams on problem statements, success criteria, and measurable outcomes. Languages: * Python (PySpark), SQL AWS: * Glue, MWAA (Airflow), S3, IAM, KMS, CloudWatch, Lambda, Step Functions, Athena, Redshift (nice), EMR (nice) DevOps: * Git, CI/CD (CodePipeline/GitHub Actions), Terraform/CloudFormation, Docker Data Quality/Observability: * Great Expectations/Deequ (nice), OpenLineage (nice) Desirable skills/knowledge/experience: * Domain experience in BFSI (risk, pricing, regulatory reporting, underwriting, fraud, payments, or actuarial data). * Experience with event-driven and near-real-time pipelines (Kafka/Kinesis, streaming ETL). * Knowledge of data quality frameworks (Great Expectations, Deequ) and data lineage/catalog (Atlas, Alation, Collibra). * Exposure to Databricks or EMR for advanced Spark workloads. * Certifications: AWS Solutions Architect / Data Analytics / DevOps Engineer. * Prior experience leading multi-team programs with offshore/nearshore models. JBRP1_UKTJ