Associate Vice President, Psychometrics & Assessments
20 days ago
Chicago
Job Description This is a hybrid position requiring 2 days per (Tuesday and Wednesday) in-person each week. Office is located in downtown Chicago, IL in the Streeterville/Mag Mile area. JOB SUMMARY Reporting to the VP, Certifying Board Services, the AVP of Psychometrics and Assessments is the organization’s lead Psychometric member that ensures all assessment practices meet the highest standard for rigor, fairness, legal defensibility, and compliance with all licensure, accreditation standards and regulatory requirements. The AVP is responsible for the overall direction of Certifying Board Services' ("CBS") exam development and psychometrics activities and will partner with CBS staff and all AOA certifying boards to deliver valid, reliable, and legally defensible AOA osteopathic board certification exams and assessments for both initial board certification and recertification purposes. The role may occasionally require evening and weekend work to accommodate the schedules of our physician board members and volunteers, as well as to facilitate meetings (both in-person and virtual) addressing essential exam development activities. KEY RESPONSIBLITIES • Lead the design, analysis, deployment, and validation of all certification assessments (e.g., written, oral, clinical, and other performance-based assessments) for Osteopathic Physicians across multiple specialties and sub-specialties., including all research, test development, publication, psychometric, and operational data analysis functions within CBS., • Oversee all psychometric functions including test development, item analyses, equating, scoring, and standard setting, ensuring alignment with AERA/APA/NCME, NCCA, and ANSI/ISO 17024 guidelines and standards., • Coordinate closely with both the AVP, Operations, and the AVP, Policy, to develop processes and procedures, oversee execution of key strategic initiatives, such as the launch of innovative testing processes and/or systems, and coordinate staff functions., • Oversee and lead continuous quality improvement activities that optimize structure, staffing and workflows to ensure sustainable efficiencies and facilitate innovation adoption for current and future assessments., • Understand, navigate, and communicate diplomatically within a multi-faceted environment while executing a complex system of inter-related activities., • Interact seamlessly with AOA Physician and Executive Leadership, BOS Chair and board members, Senior Leadership Team, CBS, and AOA colleagues on all levels within the association, as well as all affiliates and other external partners or stakeholders when appropriate with advocating for and promulgating Psychometric best practices and AOA/CBS/BOS innovations, policy changes, and revisions/updates to any/all exam development/administration/scoring changes as they occur., • Handle highly sensitive situations and confidential material with the utmost discretion., • Evaluate a team of Psychometricians, test developers, test publishers, data analysts, and other assessment professionals by identifying internal high potential individuals and providing them with effective development experiences and assignments across operational, developmental, and research initiatives. ESSENTIAL FUNCTIONS Psychometrics • Develop, deploy, and manage a comprehensive and sustainable CBS research agenda which positively and substantively improves the AOA board certification value proposition over time., • Develop and/or oversee reliability and validity studies and other psychometric activities to support all AOA certifying boards., • Ensure compliance with all best practices and third-party accreditation standards with respect to exam security., • Optimize and maintain legal defensibility of exam development, administration, scoring, and maintenance activities, as well as candidate and diplomate rights through complying with the American with Disabilities Act (ADA) and Equal Employment Opportunity Commission (EEOC), and other legal and regulatory requirements, as appropriate. MINIMUM QUALIFICATIONS OR EQUIVALENTS Education: • A minimum of ten (10) years of experience in psychometrics, measurement, analytics, high-stakes healthcare certifications (preferably within a medical board or licensure context), or high-stakes testing required., • Ability to prioritize conflicting goals while negotiating in a persuasive manner during group decision making. This would generally occur during item review calls and the evaluation of the suitability of new or equated pass points to guide subject matter expert decisions that are congruent with psychometric and assessment best practices. Expert knowledge and track record of success with qualitative and/or quantitative data handling, knowledge of assessment standards, testing methodologies, data validation and analysis and reporting is required. This position requires the ability to interact tactfully and in a sensitive manner with professionals on all levels within and outside the AOA. Key competencies also include: • Solid understanding of complex data structures and the ability to manage, manipulate and summarize small and large quantities of data, • In-Depth knowledge of statistical theory, IRT with an emphasis on Rasch models, certification testing, research methodologies, and applied measurement, • Self-starter, organized, versatile, capable of performing work independently with minimal direction, • Excellent problem-solving skills and ability to prioritize simultaneous, competing client activities, • Takes personal responsibility for service and quality of work delivered, • Excellent communication skills in both written and spoken English, • Strong interpersonal skills with the ability to act in team leader and team member roles, integrate competing ideas, and develop solutions, • Detail oriented, with strong time and project management skills PHYSICAL, MENTAL DEMANDS/WORKING ENVIRONMENT • This is a hybrid role requiring Tuesdays and Wednesdays in-person., • Work may include weekday and weekend travel., • Extended workday hours, including occasional evenings and weekends, may be necessary around major meetings, volunteer engagements, and other high-volume work periods. Annual Salary Range: $165,000.00 - $175,000.00 The AOA offers the following benefits: Medical, Dental, Vision, Life and AD&D, Long and Short-term Disability, FSA/HSA, Commuter Benefits, 401(k) with match We are an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, disability status, genetic information, protected veteran status, or any other characteristic protected by law. American Osteopathic Association is unable to sponsor work visas at this time.