Senior Software Engineer (Generalist, AI Systems)

Note

We encourage you to apply to our job board directly to have an edge. Your resume is our first look into your past experiences and impact. If you have an old resume, keep it next to you for inspiration, but start with a blank document and create a resume specifically designed for this job.

Align your skills and experience with the job description. Ensure you meet the minimum qualifications (MQs), and your resume clearly demonstrates alignment with the MQs.

Role Details

You’ll be a senior generalist engineer at ISHI Health, shipping product features across our stack while bringing specialized depth in building evaluation and validation harnesses for AI agents and LLM-powered systems. In a lean team, you’ll move fluidly between product development, platform work, and the verification infrastructure that lets us trust AI in clinical settings. You’ll partner closely with the Staff Engineer, Platform on architecture decisions and with the broader team on the AI-powered features that sit at the heart of our product.

Why this role matters

ISHI’s product depends on AI agents and LLM-powered workflows that surface clinical signals to clinicians and assist in care decisions. Building these features quickly is necessary; verifying that they behave correctly is non-negotiable. We need a generalist who can do both, i.e. ship product across the stack and bring the rigor that AI verification demands in a clinical context.

What you'll do

• Build and ship features across the ISHI stack (backend services, APIs, data pipelines, and integrations), partnering with the Staff Engineer, Platform and other engineers on the team.
• Design and maintain evaluation harnesses for ISHI’s AI agents and LLM-powered features, including prompt regression suites, retrieval quality metrics for RAG pipelines, and trajectory/behavioral testing for agentic workflows.
• Contribute to AI agent and LLM feature development directly, building the agent loops, tool integrations, and retrieval systems alongside the verification infrastructure that keeps them honest.
• Write strong tests as part of normal feature development (unit, integration, and end-to-end), and improve the test infrastructure where it slows the team down.
• Work with AWS services across the stack, including Cognito for authentication flows and EKS for containerized service deployment.
• Contribute to CI/CD pipelines, observability, and on-call practices appropriate for a clinical-grade system, in partnership with the Staff Engineer, Platform.
• Help shape engineering practices that will support our FDA SaMD regulatory pathway, including verification documentation for AI components.

What You’ll Bring (MQs)

• 5+ years of software engineering experience shipping production systems, with breadth across backend, infrastructure, and at least some exposure to data or ML systems.
• Strong proficiency in at least one backend language (Python, Go, Java, or TypeScript/Node.js) and demonstrated comfort working across the stack.
• Direct experience building evaluation harnesses, prompt regression suites, or behavioral test frameworks for LLM-powered systems or AI agents, this is the specialized depth we need alongside generalist breadth.
• Working familiarity with RAG architectures, agent frameworks (LangChain, LangGraph, or equivalent), and the unique challenges of verifying non-deterministic systems.
• Production experience with cloud infrastructure (AWS preferred), including comfortable working knowledge of Cognito and EKS.
• Experience operating in lean or early-stage environments where you’ve had to make pragmatic tradeoffs and move quickly without sacrificing quality where it matters.

Nice to have

• Familiarity with AI observability and evaluation tooling (LangSmith, Braintrust, Langfuse, or similar).
• Experience in regulated environments (healthcare, fintech, medical devices) and familiarity with verification/validation expectations.
• Working knowledge of HIPAA-compliant practices, including handling of synthetic PHI and test data segregation.
• Exposure to FDA SaMD verification practices, IEC 62304, or SOC 2 Type II compliance programs.
• Familiarity with FHIR APIs, HL7 integration, or EHR connectivity.
• Experience with streaming or time-series data systems and the testing approaches that suit them.
• Frontend or full-stack experience that lets you contribute to clinician-facing tooling when needed.

How We Work

We operate fully remote across time zones, with overlap hours for collaboration. We value written communication, async-first decision-making, and focused individual contribution time. Our hiring is oriented toward senior engineers who can operate with minimal oversight and strong context.

We expect engineers to use modern tooling, including AI coding assistants, to work efficiently. We care about outcomes and judgment.