Avatar for Flex.ai
Flex.ai
Actively Hiring
More jobs
The AI Infrastructure company

AI Inference Engineer

Posted: 2 weeks ago• Recruiter recently active
Job Location
Remote Work Policy

In office

Visa Sponsorship

Not Available

RelocationAllowed
Skills
TensorRT
vLLM
RoCM/CUDA
vLLM, SGLang, Mixture of Experts, FlashInfer, KV Cache Optimization Frameworks
PD Disaggregation
Hiring contact
Raj Deepika
Employee
image

About the job

Note:- This is not a RAG engineer or AI Agents development role. Apply only if you have worked on optimising inference layer e.g vllm, sglang, kv cache, disaggregation, rocm, tensorRT, cuda etc.

Roles and Responsibility

Own the Inference Runtime:
Own the serving stack for LLMs and multimodal models: engine configuration, parallelism (tensor, expert, data), and deployment across NVIDIA and AMD GPUs.
Bring new open-weight models to production quickly and correctly: chat templates, tool-calling and reasoning parsers, structured outputs, quantized checkpoints.
Maintain and contribute patches to open-source inference engines (vLLM, SGLang) and upstream fixes where it makes sense.

Drive Performance and Cost Efficiency:
Profile and optimize throughput, latency (time to first token, inter-token latency) and cost per token, from kernels to the request path.
Apply and evaluate techniques such as quantization (FP8, FP4), speculative decoding, KV-cache reuse and offloading, prefix caching, and prefill/decode disaggregation.
Build first-principles performance models of models and hardware to know what is achievable, then close the gap with measured, reproducible benchmarks.

Make Serving Reliable at Scale:
Run inference on Kubernetes across multiple clusters and clouds, with autoscaling, zero-downtime rollouts and graceful failure handling.
Build observability into the serving path: engine metrics, request tracing, and correctness checks that catch silent quality regressions before customers do.
Build the tooling for benchmarking, load testing, evaluation and model release.

Similar Jobs

Nosh Robotics company logo
Nosh Robotics
Building cooking robot to make healthy and personalized food at ease!
Sosuv Consulting company logo
Sosuv Consulting
We provide specialised Product, Software Development and Support services to the global fi
Gyrus.AI company logo
Gyrus.AI
AI for Image/Video Analytics
Fuze company logo
Fuze
Digital assets infrastructure platform