We are looking for an experienced Senior QA Engineer to own quality for our AI voice virtual agent. This role is for someone who is equally comfortable reasoning about a large language model's behavior and reading a SIP trace.
You should be interested in testing non-deterministic systems. A voice agent takes a live phone call, transcribes speech that may be accented or noisy, interprets intent, reasons over an LLM, generates a spoken reply, and decides whether to resolve the request or hand it to a human—all within a latency budget callers will notice. Nothing in that chain can be validated with a simple assertion of equality, so you will design the evaluation approach that makes it testable anyway.
You should also be interested in the underlying telephony. Testing this agent means placing real calls: provisioning SIP endpoints and softphones, driving calls programmatically, controlling media conditions, and distinguishing a model failure from a codec, jitter, or call-routing issue.
- ADVANCED SPOKEN ENGLISH IS REQUIRED.

Responsibilities
- Embed fully within a Scrum team,



participating in all ceremonies and maintaining associated artifacts.
- Own the end-to-end test strategy for the AI voice virtual agent across the LLM, speech, and telephony layers.
- Design test scenarios for complex conversational behavior, including multi-turn dialogues, interruptions and barge-in, ambiguous or incomplete input, out-of-scope requests, human escalation/transfer, and failure recovery.
- Build and maintain an evaluation framework for non-deterministic output, leveraging golden datasets, semantic/rubric-based assertions, and automated regression detection for prompt, model, or retrieval changes.
- Adversarially test the agent against hallucinations, incorrect grounding, guardrail bypasses, prompt injections, and unauthorized disclosure of customer data.
- Prepare and provision SIP devices, endpoints, and call scenarios to automate call origination end-to-end.
- Diagnose failures ac

📌 Senior QA Automation Engineer (Santiago)
🏢 Lever
📍 Santiago