Apply here: Career Site

Requisition ID: req1408
Employment Type: Unclassified Temporary Full-Time (UFP)
Division: Testing & Quality Assurance
Compensation: 67,081.00 – 83,852.00, Annual Salary
Job Closing: July 10, 2026 5:00PM (CST)
Location: TRS Headquarters Building 2
1900 Aldrich Street
Austin, Texas, 78723
United States

WHO WE ARE:

The Information Technology (IT) Division lays the foundation for TRS to deliver excellent service experiences across the organization and with our members. We serve with purpose through mentorship and collaboration across a broad variety of teams unified by innovation to create technology and information solutions that have a positive impact on our members’ lives.

We invite you to join one of Austin’s Top Workplaces. TRS offers a best-in-class combination of technology and continuous learning opportunities to equip you to solve problems, expand your knowledge, and create impact for 1 in 20 Texans.

The AI QA Analyst Associate will support the Testing and Quality Assurance team within the AI Technology Platform (AITP) program. The incumbent will focus on validating AI-enabled solutions, including LLM agents, and intelligent automation solutions. Responsibilities include evaluating model behavior, prompt effectiveness, response accuracy, system reliability, as well as testing AI workflows, data pipelines, and system integrations, assist in developing automated testing strategies, supporting CI/CD pipelines, and improving regression test coverage.

This Fellowship Program position is a full-time, temporary, limited term assignment not to exceed three years from the date of hire. It is intended for applicants that have graduated with a bachelor’s degree or a master’s degree from an accredited college or university in Computer Science, Information Technology, Software Engineering, Data Science, or a closely related field awarded  within the last 12 months.

This position requires onsite work at our Austin TX headquarters. 

WHAT YOU WILL DO:

AI Quality Assurance & Test Automation
• Assists in designing, developing, and executing automated test strategies for AI-enabled applications.
• Supports testing of AI workflows, data pipelines, and system integrations.
• Assists with prompt testing, hallucination detection, and evaluation of AI model performance using defined metrics.
• Supports basic bias and safety validation activities and participates in human-in-the-loop review processes to ensure accurate and appropriate AI outputs.
• Identifies and documents defects, inconsistencies, and risks during testing cycles.
• Assists with validating AI model inputs, outputs, and data flows to ensure accuracy and reliability.
• Supports testing of AI-driven features, including conversational agents and workflow automation.

Support Functions
• Supports data analysis related to AI system performance, including reviewing evaluation outputs, identifying trends,
• Assisting with reporting model quality metrics.

Fellowship Curriculum
• Participates in trainings and projects related to the TRS Fellowship Program including webinars, quarterly check-ins, classes or other learning activities, book group, projects, and the mentoring program.
• Meets TRS Fellowship Program deadlines and goals for completion of trainings in deliverables in a timely manner.

Performs related work as assigned. 

WHAT YOU WILL BRING:

Required Education
• Bachelor’s degree or Master’s degree in Computer Science, Information Technology, Software Engineering, Data Science, or a closely related field awarded within the last 12 months.

Required Experience
• None.

Required Registration, Certification, Licensure
• None.

Preferred Qualifications
• Experience with prompt engineering
• Proficiency in Python or a similar programming language
• Experience with database queries (e.g., SQL) for validating data integrity
• Familiarity with testing AI workflows or data-driven applications
• Exposure to Microsoft Copilot Studio, including testing conversational agents and integrations
• Familiarity with Azure AI services (e.g., Azure AI Foundry) and understanding of AI data flows
• Exposure to CI/CD tools and automated testing frameworks

Knowledge, Skills, and Abilities

Knowledge of:
• Fundamentals of Machine Learning and Large Language Models
• AI system evaluation techniques
• Prompt engineering concepts and conversational AI design
• Responsible AI principles
• Basic QA methodologies, software testing concepts, and automation approaches
• Software development lifecycle and modern DevOps practices

Skill in:
• Evaluating AI generated outputs using quantitative and qualitative methods
• Writing effective prompts for conversational AI systems
• Using evaluating datasets and defining expected AI behavior
• Writing and executing test cases (manual and automated)
• Analyzing data and identifying defects or inconsistencies
• Communicating technical findings to technical and non-technical stakeholders
• Managing multiple tasks in a structured, team-based environment

Ability to:
• Analyze AI system behavior and identify non deterministic issues
• Evaluate ambiguous outputs and apply judgement based validation
• Collaborate with data scientists and AI engineers on model improvements
• Work with business users on requirement clarification, user acceptance tests, and troubleshooting issues.
• Plan, organize, and coordinate work assignments to effectively meet frequent and/or multiple deadlines; handling multiple tasks simultaneously; and managing conflicting priorities and demands.
• Work in a team test environment.
• Analyze problems and devising effective solutions.
• Effective verbal and written communication of complex technical information
• Establish and maintain harmonious working relationships with co-workers, agency staff, and external contacts.
• Work effectively in a professional team environment.

Military Occupational Specialty (MOS) Codes:
Veterans, Reservists or Guardsmen with experience in the Military Occupational Specialty ( https://www.trs.texas.gov/files/trs-military-crosswalk.xlsx ) along with the minimum qualifications listed above may meet the minimum requirements and are highly encouraged to apply. Please contact Talent Acquisition at  careers@trs.texas.gov  with questions or for additional information.

To view all job vacancies, visit www.trs.texas.gov/careers  or www.trs.csod.com/careersite.

For more information, visit www.trs.texas.gov.

Apply here: Career Site