Agent Evaluation Engineer

*About Us: *

BDIPlus, a US-based leading transformation Consulting & Customer Data Platform (CDP) company, is dedicated to delivering cutting-edge capabilities and solutions that foster the development of enduring competitive advantages. Our innovative solutions showcase our unparalleled proficiency in technology and our profound domain expertise within the Financial Services and Insurance sectors. By synergizing our unmatched technical skills with a comprehensive grasp of each client’s institutional landscape and distinctive areas for improvement, we empower them to convert data into actionable and well-organized information. This facilitates precise decision making, increased efficiency, and rampant business growth.

*About the Role *

The Agent Eval Engineer designs and builds the evaluation system that tells us whether an agent is good enough to ship. Tests cover normal cases, bad or missing data, different customer groups, the approval steps, and deliberately tricky inputs.

Key Responsibilities

• Design the evaluation harness architecture: test-data management, the scoring pipeline, and automatic detection of regressions between versions.
• Design how evaluations gate releases, and how offline tests relate to online/live-like checks.
• Design the scoring architecture — including using an LLM as a grader — so results are consistent and reproducible.
• Build and maintain the test suites for each agent, agreed with the business early on.
• Cover the full range of cases: expected inputs, degraded data, different customer segments, and adversarial inputs meant to trip the agent up.
• Set up scoring in Langfuse and run tests against live-like traffic.
• Run the full test round at project milestones and report results to the Lead.

Required Skills

• Hands-on experience testing agents or LLMs across both normal and adversarial cases.
• System design for evaluation: test-data management, scoring pipelines, regression detection, and release gating.
• Langfuse (or a similar testing/monitoring tool) experience.
• Strong Python 3.11+ and pytest.
• Careful about defining test groups, segments, and failure cases.

Preferred Skills

• Experience using an LLM as a grader and testing against live-like traffic.
• Familiarity with tests based on data quality and model drift.

Our Purpose and Culture:

At BDIPlus, our mission is to help enterprises utilize their resources more efficiently, implement effective information management, and empower them by enabling richer insights and intelligence.

We are driven by a single purpose: empower the technology transformation. We are passionate about creating foundational technology platforms for enterprise data and information management. Our employees are at the heart of the work we do at BDIPlus. We are committed to encouraging and celebrating innovation, creativity, and hard work among our team members.

Working at BDIPlus offers:

A diverse, fun, highly intelligent, and innovative team.
Provident Fund
A competitive salary and an annual bonus.
Standard time off, sick leave, and holidays.
An environment where creative thinking is encouraged, and innovation drives everything we do.