QA & Support Engineer — AI Voice Platform

QA & Support Engineer — AI Voice Platform

Location: Remote
Preferred: India-based
Type: Full-time

About Us

We’re building a production AI platform that puts autonomous voice agents to work — conducting real, live phone conversations for sales, support, and engagement at scale.

Our agents handle multi-turn dialogue, execute actions, remember context, extract useful information from conversations, and generate post-call intelligence from every call.

This is not an AI demo. We’ve built the full stack: a real-time voice pipeline, an agentic AI runtime, a memory system, an AI evaluation layer, and a campaign orchestration platform — all running in production with real customers.

Our system is Python-first and works deeply with modern LLMs, voice AI APIs, real-time speech systems, automated testing, monitoring, and cloud-native infrastructure.

We’re a small technical team moving fast and building practical AI systems for real-world business use cases.

The Role

We’re looking for a QA & Support Engineer to own product quality, AI behaviour testing, production monitoring, and customer support operations across our AI voice platform.

This is not a manual click-through QA role. You’ll test real AI agents, validate live call flows, write automated test cases, monitor production issues, reproduce bugs, communicate with customers, and work closely with engineering to improve system reliability.

You’ll work across Python backend systems, LLM workflows, voice AI pipelines, campaign flows, post-call analysis, dashboards, logs, monitoring tools, and customer-reported issues.

What You’ll Work On

Feature Testing & QA

• Write and maintain automated test suites for APIs, agent workflows, and campaign logic
• Test backend APIs, webhooks, queues, background jobs, and async workflows
• Validate end-to-end campaign flows including call lifecycle, contact progression, retries, and post-call analysis
• Test dashboard features including reports, filters, analytics, call data, and real-time session state
• Perform regression testing for every release, including AI model, prompt, and workflow changes
• Run smoke tests, integration tests, and load/stress tests for high-concurrency call scenarios

AI Behaviour Testing

• Test whether AI agents follow instructions, stay on script, handle objections, and complete goals
• Validate tool calling, workflow execution, memory injection, fallback handling, and edge cases
• Test LLM outputs for accuracy, consistency, hallucinations, and instruction-following
• Build golden test cases for agent conversations with expected inputs, expected outputs, and pass/fail criteria
• Catch regressions after prompt changes, model changes, workflow updates, or backend changes
• Validate AI evaluation scores, post-call summaries, sentiment analysis, and outcome detection

Voice AI Pipeline Testing

• Test real-time voice flows across speech-to-text, LLM, and text-to-speech stages
• Validate call setup, audio routing, turn-taking, interruption handling, and response timing
• Test voice AI tools and APIs such as OpenAI, Gemini, Deepgram, ElevenLabs, or similar platforms
• Monitor latency across speech recognition, LLM response, and voice synthesis
• Identify failed calls, dropped calls, timeout issues, silence handling problems, and audio quality issues

Monitoring & Observability

• Monitor production dashboards for uptime, error rates, latency, queue health, and campaign KPIs
• Set up and maintain alerts for failed calls, API errors, timeout spikes, queue failures, and agent crashes
• Read and debug logs, stack traces, structured errors, and API responses
• Identify production failure patterns before they become customer complaints
• Track AI quality drops, latency spikes, and unexpected agent behaviour
• Work with engineering to improve logging, metrics, tracing, and monitoring coverage

Production Support

• Own customer support communication for technical issues
• Triage customer-reported bugs, reproduce issues, and identify root causes
• Distinguish between user errors, configuration issues, data issues, and product bugs
• Escalate issues to engineering with clear logs, steps to reproduce, screenshots, recordings, and expected vs actual behaviour
• Write clear customer-facing updates during bugs, incidents, and issue resolution
• Build and maintain internal runbooks, QA checklists, support docs, and known-issue documentation

Backend & Data Validation

• Validate data correctness across databases, dashboards, call records, campaign analytics, and post-call outputs
• Write SQL or NoSQL queries to verify system behaviour and debug data issues
• Test webhook delivery, retry logic, async processing, and eventual consistency
• Validate contact status updates, call outcomes, lead progression, and analytics calculations
• Work with Python-based backend systems, APIs, background jobs, and cloud-native deployments

You’re a Fit If You

• Have experience writing automated tests for APIs and backend systems
• Can test more than UI flows — APIs, async jobs, webhooks, queues, and data pipelines
• Can read logs, trace errors, debug failures, and reproduce issues independently
• Understand async systems, retries, webhooks, eventual consistency, and background workers
• Are comfortable testing AI-powered features, LLM outputs, and agent behaviour
• Have worked with Python, JavaScript/TypeScript, or similar languages for test automation
• Have experience with monitoring, alerting, production debugging, or technical support
• Can communicate clearly with both customers and engineering teams
• Are detail-oriented, structured, and comfortable working in a fast-moving product environment

Required Skills

• API testing
• Automated test writing
• Python or JavaScript/TypeScript for test automation
• Backend testing
• Integration testing
• Regression testing
• Smoke testing
• Load testing basics
• Debugging logs and stack traces
• SQL or NoSQL data validation
• CI/CD testing workflows
• Webhooks, queues, retries, and async systems
• Production monitoring and alerting
• Customer support and issue triage
• Clear written communication
• AI/LLM feature testing
• Voice AI pipeline testing
• Test case design and QA documentation

Bonus Points

• Experience testing LLM-powered applications
• Experience with OpenAI, Gemini, Deepgram, ElevenLabs, or similar AI/voice APIs
• Experience testing real-time voice, audio, video, or streaming systems
• Experience with AI agents, tool calling, prompt regression, and model output validation
• Familiarity with observability tools, dashboards, alerts, and log aggregation
• Experience with Kubernetes, Docker, or cloud-native deployments
• Experience with load testing tools and performance benchmarking
• Experience in technical support, QA operations, or production operations
• Experience writing runbooks, support documentation, and internal knowledge base articles
• Early-stage startup experience

What We Offer

• Direct collaboration with technical leadership and AI engineers
• High autonomy over QA, monitoring, and support processes
• Opportunity to work on production AI systems used by real customers
• Work at the frontier of real-time voice AI and agent systems
• Competitive compensation based on experience