Senior AI Engineer, Agent Evaluation and Governance

Niuro is hiring a Senior AI Engineer for an enterprise AI platform company where AI agents take real actions: sending messages, updating records and preparing decisions. You will own the evaluation and governance layer that decides when each agent capability is ready to go live and keeps it safe in production.

You will

• Design evaluation sets, metrics, quality thresholds and go-live gates per agent capability.
• Detect regressions after model, prompt or instruction changes.
• Build runtime safeguards in code: approvals, data restrictions, deterministic checks, kill switches and quality-based suspension.
• Log and trace every model call for accountability and auditability.
• Turn governance rules and acceptance scenarios into executable validation with platform, cloud and product teams.
• Write clear design notes before implementation.

You bring

• 5+ years of software engineering experience.
• Production experience with tool-using, multi-step LLM agents.
• Evaluation sets, quality thresholds and regression detection in production.
• Applied statistics: review sample sizing, confidence bounds, error-rate estimation.
• Strong Python or TypeScript, with testing and code review.
• C1 English.

Nice to have: knowledge graphs and ontologies (RDF, OWL, SHACL, LinkML), RAG, human-in-the-loop systems, access control, multi-provider LLM integration, financial services.

Contract: full-time, fully remote, monthly contract in USD.

Apply through Niuro: https://app.niuro.io/source/CbmWjJkM?ref=ps_FtixrvAJ3nEyj0nPwo6ANNKEFFN0KXqr-c5os_3iSCU&utm_source=wellfound&utm_medium=job_board