Observability & AgentOps Engineer

*About Us: *

BDIPlus, a US-based leading transformation Consulting & Customer Data Platform (CDP) company, is dedicated to delivering cutting-edge capabilities and solutions that foster the development of enduring competitive advantages. Our innovative solutions showcase our unparalleled proficiency in technology and our profound domain expertise within the Financial Services and Insurance sectors. By synergizing our unmatched technical skills with a comprehensive grasp of each client’s institutional landscape and distinctive areas for improvement, we empower them to convert data into actionable and well-organized information. This facilitates precise decision making, increased efficiency, and rampant business growth.

*About the Role *

The Observability & AgentOps Engineer designs and runs the systems that keep the agents reliable in production — how they are monitored, deployed, traced, and recovered when something goes wrong. This is as much a system-design role as an operations one: setting the reliability targets and building the architecture (tracing, logging, deployment, and recovery) that the whole team runs on.

Key Responsibilities

• Design the end-to-end observability architecture — tracing, metrics, and structured logging — deciding what each agent step should record so runs can be understood and debugged.
• Define reliability targets (SLIs/SLOs) for the agents and design the alerting and on-call response around them.
• Design the deployment topology and orchestration: AWS Step Functions state machines, Docker packaging, and CI/CD pipelines, with sensible retries, timeouts, and idempotency.
• Design for failure — retries, fallbacks, graceful degradation, and replay/recovery of interrupted agent runs, including at the approval steps.
• Design the log and trace data model and its storage: retention tiers, indexing, and scaling for the ELK audit trail at production volume.
• Design capacity and cost controls for LLM/agent workloads — concurrency limits, rate limiting, and token/cost tracking.
• Set up and run day-to-day monitoring (Langfuse, OpenTelemetry) and support handover as the client's own team takes over.

Required Skills

• Hands-on experience designing observability systems from the ground up: distributed tracing, metrics, and structured logging (Langfuse, OpenTelemetry, ELK, or equivalents).
• Distributed-systems design skills: retries, idempotency, timeouts, backpressure, and failure-mode analysis.
• Experience defining SLIs/SLOs and designing alerting and incident response.
• CI/CD, Docker, and AWS Step Functions experience.
• Strong Python 3.11+ and a production-operations mindset focused on reliable, traceable systems.

*Preferred Skills *

• Experience designing and running LLM or agent workloads in production, including cost and concurrency controls.
• Infrastructure-as-code and AWS deployment/architecture experience.
• llmExperience designing high-volume log and trace storage and retention.

Our Purpose and Culture:

At BDIPlus, our mission is to help enterprises utilize their resources more efficiently, implement effective information management, and empower them by enabling richer insights and intelligence.

We are driven by a single purpose: empower the technology transformation. We are passionate about creating foundational technology platforms for enterprise data and information management. Our employees are at the heart of the work we do at BDIPlus. We are committed to encouraging and celebrating innovation, creativity, and hard work among our team members.

Working at BDIPlus offers:

A diverse, fun, highly intelligent, and innovative team.
Provident Fund
A competitive salary and an annual bonus.
Standard time off, sick leave, and holidays.
An environment where creative thinking is encouraged, and innovation drives everything we do.