Senior QA Engineer (AI Systems & Automation)
6 days ago
Rockville
Job DescriptionSenior QA Engineer (AI Systems & Automation)Essential Software Inc. (ESI)OverviewEssential Software Inc. is a trusted partner to federal agencies, including the National Cancer Institute (NCI), delivering secure, cloud-based platforms that support large-scale cancer data and biomedical research.As a Senior QA Engineer (AI Systems & Automation), you will lead quality strategy and test automation for critical data platforms and AI-powered experiences. You will ensure both traditional software and AI/agentic systems are reliable, explainable, and safe in a federal, mission-driven environment.You will: • Own end-to-end quality for complex web, API, data, and AI/ML-powered features, • Design AI-aware test strategies and automation that leverage GenAI and agentic frameworks, • Mentor QA engineers and collaborate closely with cross-functional teams and government partnersRoles and ResponsibilitiesCore QA & Automation, • Develop and maintain test plans, test cases, traceability, and test data for product and AI features, • Execute manual and automated tests for web applications, RESTful APIs, data workflows, and AI/ML features, • Own automated regression suites, release readiness criteria, and provide clear go / no-go quality signals, • Participate in agile ceremonies, validate end-to-end functionality, and ensure user stories (including AI features) meet acceptance criteria, • Manage the full defect lifecycle, including triage, prioritization, root cause analysis, and verification of fixes, • Maintain QA documentation, runbooks, and quality dashboardsAI & Agentic QA Systems, • Design and execute test strategies for AI/LLM-powered capabilities, including virtual agents, chatbots, copilots, and RAG-based systems, • Use LLM-powered tools (e.g., ChatGPT, Claude, Copilot) to accelerate test design, data generation, exploratory testing, and script authoring, • Build and refine QA-focused AI agents that can:, • Scrape UI and verify DOM structures, • Validate data against backend or ground-truth sources, • Auto-generate and maintain test scripts, • Run self-correcting / autonomous test flows, • Evaluate and integrate agentic frameworks (e.g., OpenAI Assistants API, AWS Bedrock Agents, LangGraph, MCP) into QA workflows, • Define and monitor AI-specific quality metrics (accuracy vs. ground truth, hallucination and error rates, safety / policy adherence), • Ensure AI and virtual agent experiences are accurate, consistent, and high quality in a federal contextNon-Functional, Data Quality & Collaboration, • Plan and execute performance, load, and scalability testing (e.g., JMeter or equivalent), • Validate data integrity and transformation quality across complex biomedical data pipelines and AI-enhanced workflows, • Partner with engineers and data scientists to ensure AI/ML models and integrations are testable, observable, and measurable post-deployment, • Mentor QA team members in both traditional and AI-augmented QA practices, • Collaborate with development, DevOps, product, UX, and data teams to improve testability, shift-left quality, and increase automated coverage, • Integrate automation into CI/CD (e.g., GitHub Actions, Jenkins, Azure DevOps, GitLab CI), monitor test health and flakiness, and address coverage gaps, • Communicate quality risks, trends, and mitigation plans to technical and non-technical stakeholders, including government partnersBasic Qualifications, • Bachelor’s degree in computer science, Information Technology, Engineering, or related field, • 5+ years of software QA experience (manual and automation) in production environments, • 2+ years providing technical or process leadership (e.g., lead QA, primary product QA owner, mentor, or manager), • Strong experience with UI automation tools (Selenium WebDriver, Playwright, or Cypress), • Experience testing RESTful APIs and microservices architectures, • Hands-on experience integrating automated tests into CI/CD pipelines (GitHub Actions, Jenkins, Azure DevOps, or GitLab CI), • Professional proficiency in Python or JavaScript for test automation, • Hands-on use of GenAI tools (e.g., ChatGPT, Claude, Copilot) for QA tasks such as test-case generation, data creation, and exploratory testing, • Understanding of AI/agentic concepts:, • Tool-calling / function invocation, • Multi-step / chain-of-thought workflows, • Autonomous / self-healing test flows, • AI-driven data comparison and validation, • Experience with performance / load testing (e.g., JMeter or equivalent), • Proficiency with Jira or similar issue tracking tools, • Strong written and verbal communication skills, including the ability to explain AI-related quality risks to stakeholders, • Ability to prioritize, multitask, and operate effectively in complex, mission-driven environmentsPreferred Qualifications, • AWS Cloud Practitioner certification, • Experience with modern automation stacks (Playwright or Cypress) and API testing tools (Postman, REST-assured, pytest, or similar), • Experience testing AI/ML-powered features (LLM applications, RAG systems, agents, recommendation engines, or chatbots), • Experience with one or more:, • LangChain or LangGraph, • AWS Bedrock Agents or OpenAI Assistants API, • MCP (Multi-Context Protocol) or similar orchestration frameworks, • Experience designing or testing internal QA copilots or automation bots for test authoring or execution, • Familiarity with test management tools (e.g., TestRail, Zephyr), • Knowledge of accessibility standards (WCAG) and basic security testing practices, • Prior QA experience in healthcare, life sciences, biomedical informatics, or other regulated data environments, • ISTQB or similar certificationHow We Work, • Mission driven and customer focused, working directly with scientists, product owners, and data experts, • Curious and learning oriented, with encouragement to experiment with new AI tools, patterns, and technologies, • Work on cutting-edge biomedical informatics and AI projects with real-world impact, • Full-time role; primarily remote, with approximately four days per month onsite in Rockville, MD Powered by JazzHR sb9BhRMwjH