Lead Data Engineer - View Integrations

At Klaviyo, we love tackling tough engineering problems and look for engineers who specialize in certain areas but are passionate about building, owning & scaling features end to end from scratch and breaking through any obstacle or technical challenge in their way. We push each other to move out of our comfort zone, learn new technologies and work hard to ensure each day is better than the last.

Team Overview:

Klaviyo operates a real-time and offline data analytics platform coded primarily in Python that is built for massive scale and hosted on Amazon Web Services (AWS). The DataLake and View Integration teams ingest billions of interactions per day to power core, Klaviyo functionality and generate data-driven insights. These interactions are ingested from disparate sources, persisted into the Data Lake, materialized into analytical storage, and used to power AI models.

As a Lead Engineer on the View Integrations Team, you will have ownership over defining the evolutionary, technical vision for materialized view frameworks. You will be responsible for the design, implementation and tuning of highly visible mission critical materialized view systems. You will be constantly challenged to improve these systems from every angle, such as monitoring, failure recovery, data quality, tooling, performance, ease of use and much more.

Team Tech Stack:

• Python
• Spark
• Airflow
• RabbitMQ, Kafka
• DynamoDB
• MySQL
• Kubernetes
• AWS

How You'll Make a Difference

• Responsible for coaching engineers, managing/reviewing technical documentation and articulating a phased approach to achieving the team's overall technical vision.
• Wholly owning features that 50,000+ customers rely on daily to reach 4B+ consumers to drive their business forward.
• Working on crafting our highly scalable real-time data pipeline that processes billions of events weekly across data sets growing to petabyte scale.
• Working on a deep analytics system that provides insight into hundreds of terabytes of data.
• Be responsible for the mission, technical direction and operations of projects that have immediate impact.
• Helping architect and design robust and high performance data processing and storage systems leveraging new databases to serve different internal and external use cases. Contribute to open source data processing technologies.
• Contributing to the company as a subject matter expert in multiple areas, constantly pushing yourself to be a better engineer and to level up all of your peers within your team and within Klaviyo.

Must Have:

• 10+ years of experience in software development, built and delivered at least one complex data processing system
• 2 years of experience in a people management role or team leadership role, coordinating the delivery of nontrivial projects.
• 5 years of experience testing and launching software products
• Experience working with reliable, fault-tolerant, and highly performant distributed systems.
• Experience with high throughput orchestration tools like temporal or cadence
• Bachelors degree in Computer Science or equivalent experience.