Software Engineer — AI Agent QA & Observability

About Softprobe

Softprobe builds quality infrastructure for AI agents. We're a US/China developer-tools startup working on the business layer of AI agent quality — not infra, but whether agents actually do the right thing for real users. One of our business-layer evals, for example, is customer emotion detection.

We build in the open. Our two open-source products:

• Arex — automated API testing that captures real user traffic and replays it as test cases. No hand-written test suites to maintain, no costly throwaway test environments. Trusted by teams at Trip.com, Yeahka, United Imaging, Lexin, ZTO, and others. thelake — an open-source AI evidence lake on DuckDB, a LangSmith / Datadog alternative built for long-term storage of full payloads (prompts, responses, tool I/O, HTTP bodies) as open Parquet — about 200x cheaper than index-heavy observability. OTLP in, DuckDB SQL out. It's the durable evidence foundation underneath Softprobe Agent QA.

What you'll do

• Build and ship across both products: Arex's traffic-capture and replay engine, and thelake's OTLP ingest, DuckLake storage, and query layer.
• Build the business-layer evaluation surface of Softprobe Agent QA — evals, Explorer, diagnosis, and collaboration on top of the evidence lake.
• Design evals that judge agents on business outcomes, not just token-level metrics — and make the results explainable to the teams running the agents.
• Work directly with real agent traffic from production systems, including replaying and regressing against recorded sessions.

We're looking for

• An expert in AI agents — this is the core requirement. You've built, shipped, and debugged LLM-powered agents in production, not just demos. You know how agents fail in practice (bad tool calls, lost context, wrong escalation, hallucinated confidence) and how to measure and fix that.
• Strong backend engineering fundamentals, and comfort owning a feature end to end — from design through production.
• Experience with LLM evaluation: building evals, benchmarks, or red-teaming, and turning evaluation results into product improvements.
• Familiarity with observability and tracing (OpenTelemetry, structured logging, trace analysis) and with data systems for large-scale storage and querying.

Nice to have

• Open-source contributions, especially in testing, observability, or agent frameworks.
• Experience with replay / record-and-replay testing systems.
• Rust and/or Java, DuckDB, and columnar storage (Parquet).
• Experience with customer-facing AI (support, sales, or service agents) and the quality bar those users expect.

Our stack

Java (Arex) · Rust (thelake) · DuckDB / DuckLake · OpenTelemetry · MongoDB · Redis