- B2B
- Scale StageRapidly increasing operations
- Top InvestorsThis company has received a significant amount of investment from top investors
- +2
Senior Director, Network Reliability Engineering
- $399k – $531k
- |
- |8 years of exp
- |Full Time
In office - WFH flexibility
Not Available
About the job
Lambda, The Superintelligence Cloud, is a leader in AI cloud infrastructure serving tens of thousands of customers. Our customers range from AI researchers to enterprises and hyperscalers. Lambda's mission is to make compute as ubiquitous as electricity and give everyone the power of superintelligence. One person, one GPU.If you'd like to build the world's best AI cloud, join us.Note: This position requires presence in our Bellevue office location 4 days per week; Lambda’s designated work from home day is currently Tuesday.Every customer workload on Lambda depends on the network, from a single-GPU experiment to a training run spanning thousands of GPUs. When an optic degrades, a link flaps or a switch fails, customers feel it as stalled jobs, restarts and lost compute time. We will operate the existing network flawlessly for all customer workloads while we reinvent how failures are found and fixed, and the automation underneath all of it.Lambda has 10x'd over the last three years, and the network engineering organization is scaling to match. We're looking for a Senior Director of Network Reliability Engineering to own the health of that network end to end: the break-fix operations that restore service today, and the software that detects, isolates and repairs failures automatically so that most never reach a human or a customer.You'll lead an organization spanning network SRE and automation engineering, incident management, and break-fix coordination with our data center teams. We're looking for an experienced leader of leaders who treats every repeated failure as a software problem, measures reliability and builds toward a self-healing network where detection, diagnosis and remediation run as automated closed loops. You'll hire and grow a talented team of software and network proficient engineers, shape team culture, and partner with technical leadership to keep reliability ahead of our scale ambitions.This role will report to the VP of Cloud and AI Networking, working alongside a highly capable and passionate team of engineers. If you'd like to join a great team of customer obsessed engineers building the world's best AI cloud, come join us.What You'll DoOwn the reliability of Lambda's network end to end, from GPU compute fabrics to data center, backbone and edge, and be accountable for availability, time to detect and time to restoreRun break-fix operations at scale: fault detection, triage, hardware replacement and RMA, optics and cabling remediation, and coordination with Data Center Operations and remote-hands partners across a growing fleet of data centersLead the networking ops excellence and incident management program, including on-call, incident command, escalation paths, customer communication and blameless postmortems that turn into engineering fixesSet the multi-year strategy for a self-healing network: automated loops that detect and isolate failing components, shift traffic, drain and restore capacity, and open repair work without human interventionConvert manual break-fix work into durable software. The goal is a network that repairs itself, not a faster set of runbooks. Measure toil, set targets to reduce it, and track the share of failures resolved automatically as a primary outcome for your orgDefine network SLOs and error budgets with product and customer-facing teams, and use them to prioritize reliability work against feature and capacity workBuild automation that is safe to trust, with blast-radius limits, staged rollouts, audit trails and rollback for every automated action, including AI-assisted triage and remediationOwn the operational bar for your org: how incidents are detected, handled, reviewed, measured and learned fromBuild and lead the organization that delivers it: hire, develop and retain engineers and the leaders who manage them, and design the team structure as the network and the org growManage through leads and senior ICs, developing both, with regular 1:1s, clear performance feedback, growth planning, and sponsorship of meaningful workPartner with Network Engineering, Network Capacity Automation, Software and Principal Engineers so reliability requirements are designed in from the start and failure data informs architecture, hardware and vendor decisionsPartner with Data Center Operations, Supply Chain and vendors on spares strategy, RMA turnaround and hardware failure trendsRepresent your org's work, reliability metrics and technical direction to senior leadership, and to enterprise customers and partners during major incidents and reviewsContribute to org-wide engineering process improvements: how we plan, how we ship, how we learn from incidentsYouHave 8+ years of engineering management experience, including managing managers and directly managing senior individual contributorsProven track record of leading SRE, network reliability or production operations organizations for large-scale cloud or data center networks with 24x7 availability requirementsHave a network or software engineering background. You've shipped production systems and can engage credibly on architecture, technical tradeoffs, and code qualityHave measurably reduced toil and incident impact through automation, and can show the data, such as lower time to restore or a higher share of incidents remediated automaticallyFamiliarity designing, building, scaling and operating large-scale networks, including routing (BGP), leaf-spine fabrics, optics and hardware failure modes, and network telemetry and monitoringExperience building closed-loop or event-driven remediation systems, and the safety controls that keep automation from causing incidentsHave run incident management at scale: incident command, a blameless postmortem culture, and turning lessons into engineering work that shipsFoster a data driven, automation first, high velocity organizationHave a track record of building high-performing teams in fast-moving, technically demanding environmentsAre skilled at translating ambiguous business and product goals into clear team priorities and executable engineering plansShow strong judgment about when to go deep technically, when to delegate, and when to escalateStay calm and decisive during major incidents, and communicate clearly with customers and executives while one is in progressExcellent communicator both written and verbal. You use high quality written artifacts to inform high quality decision making and get approval on problems and solution opportunitiesNice to HaveExperience in GPU cloud, HPC, or AI/ML infrastructure environmentsExperience operating InfiniBand or RoCE fabrics at scale, and understanding how network faults affect distributed training jobsExperience applying ML or LLM-based tooling to operations, such as anomaly detection, alert correlation, automated triage or root-cause analysisExperience with streaming network telemetry at scale (gNMI, OpenConfig or similar)Salary Range InformationThe annual salary range for this position has been set based on market data and other factors. However, a salary higher or lower than this range may be appropriate for a candidate whose qualifications differ meaningfully from those listed in the .About LambdaFounded in 2012, with 500+ employees, and growing fastOur investors notably include TWG Global, US Innovative Technology Fund (USIT), Andra Capital, SGW, Andrej Karpathy, ARK Invest, Fincadia Advisors, G Squared, In-Q-Tel (IQT), KHK & Partners, NVIDIA, Pegatron, Supermicro, Wistron, Wiwynn, Gradient Ventures, Mercato Partners, SVB, 1517, and Crescent CoveWe have research papers accepted at top machine learning and graphics conferences, including NeurIPS, ICCV, SIGGRAPH, and TOGOur values are publicly available: https://lambda.ai/careersWe offer generous cash & equity compensationHealth, dental, and vision coverage for you and your dependentsWellness and commuter stipends for select roles401k Plan with 2% company match (USA employees)Flexible paid time off plan that we all actually useEqual Opportunity EmployerLambda is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by local, state, or federal law.Compensation Range: $399K - $531K
About the company
- B2B
- Scale StageRapidly increasing operations
- Top InvestorsThis company has received a significant amount of investment from top investors
- Valuation $1B+This company has a valuation of $1B or more
- Recently fundedRaised funding in the past six months











