Ai QA Engineer II/III

About IntelePeer.ai

IntelePeer delivers rapidly deployable communications solutions for an always connected world. Our Conversational AI Platform instantly improves your customers’ communications experience. IntelePeer provides industry leading time-to-value with Agentic AI solutions that work seamlessly with your infrastructure. Our no-code templates, low-code, co-creation, and developer API options provide you with simple, easy-to-use tools that can be utilized by anyone.

Job Summary

The AI QA Engineer II/III leads the end-to-end quality validation of AI models and conversational agents to ensure they deliver accurate, safe, and contextually relevant user experiences. This role analyzes conversational requirements to design specialized LLM prompt testing, adversarial edge cases, and dialogue flow evaluations that safeguard against hallucinations and bias. By integrating real-world conversation analytics with functional and regression testing, the engineer monitors model performance and intent recognition to ensure reliability. Through meticulous documentation of testing objectives and results, the role ensures that every conversational interaction meets established quality standards and user expectations before deployment.

Responsibilities: AI Model Evaluation & Testing

• Build prompt‑based test cases to evaluate LLM outputs for correctness, stability, safety, and non‑functional targets (latency, determinism, cost).
• Execute scripted model tests to validate agent behaviors across intents, flows, and edge‑cases.
• Maintain voice‑to‑voice and API regression tests to detect model drift or unintended degradation.

Data Analysis & Insight Generation

• Use Smart Analytics/Analysis Agents to review real user interactions and identify issues (misclassification, routing errors, hallucinations).
• Summarize patterns and provide example‑driven insights to internal teams.

Model Oversight & HITL Operations

• Apply HITL policies for low‑confidence model predictions, unknown intents, or out‑of‑scope cases.
• Document CX and accuracy impact for model improvement cycles.

Quality Reporting & Evidence Logging

• Publish evaluation results in TestRail with structured evidence.
• Use the nuanced pass taxonomy (not binary pass/fail) to communicate model readiness.

Required Qualifications

2-4 years of experience in QA, LLM prompt testing, and conversational AI for level II and 4-7 years’ experience for level III

Preferred: Coursework or projects involving NLP, LLM prompt engineering, model evaluation or automated testing frameworks.

Education & Experience

• Bachelor’s degree in computer science, Data Science, Linguistics, Cognitive Science, HCI or related AI– adjacent field.

Competencies

• Solid understanding of LLM behavior (hallucination patterns, determinism, prompt sensitivity)
• Progressive experience testing API’s, conversational UX and/or machine learning
• Strong analytical skills for pattern recognition in model outputs
• Clear, concise communications skills
• The ability to work in a fast-paced environment and be adaptable to change.
• Strong initiative, self-motivated, proactive, and resourceful.
• Team player who is willing to go above and beyond to help others

Physical Requirements

• Sedentary work lifting no more than 10 pounds.
• Occasional lifting, carrying, and standing.
• Frequent hand/eye coordination to operate office equipment.
• Vision sufficient to read computer screens, reports, and related department documents.
• Dexterity to operate computer keyboards and other related office equipment.
• Endurance sufficient to sit and work at a computer for extended periods of time.
• Frequent speech communication and hearing.

Why You'll Love It Here

• Unlimited Vacation for exempt employees
• Paid Holidays
• Competitive medical, dental & vision insurance for employees and their dependents
• 401K Retirement Plan
• Stock Options
• Company-paid life insurance
• Health & Flexible Savings Accounts
• Cell phone, gym, and internet reimbursement
• Paid Parental Leave
• Tuition Reimbursement
• Employee Assistance Program (EAP)
• Free snacks (Denver, and or Fort Lauderdale)
• Fun events (virtual and in-person)

**Applicants must be authorized to work for any employer in the U.S.

We are unable to sponsor or take over sponsorship of an employment visa at this time.Any requests to exercise your rights as a data subject under GDPR should be submitted to [email protected] for prompt processing. Please refer to our Privacy Policy (at *www.intelepeer.com/privacy/intelepeer-privacy-policy) for any questions on how IntelePeer complies with GDPR.*

For California residents only: Please refer to the link below for IntelePeer’s Applicant CCPA Privacy Notice. https://intelepeer.com/privacy/intelepeer-california-applicant-privacy-notice/

IntelePeer participates in E-Verify.

At IntelePeer, we value diversity and are proud to be an Equal Opportunity Employer. We do not discriminate on the basis of race, color, religion, sex, national origin, age, disability, genetic information, or any other protected status.

We strive to provide reasonable accommodations to applicants and employees with disabilities to support them in performing the essential functions of their roles.

If you have any questions or need assistance, please contact our Director of Recruiting.