Data Engineer II

To be considered, please submit your application through our applicant tracking system: Apply Here.

About Us

We are a small, fast-paced team of all-rounders who build reliable systems across our product. We value ownership, comfort with ambiguity, and people who care about making real progress.

The Role

We run a large and growing set of scrapers and data pipelines that power our product. Much of that scraper code is written with AI assistance, so the systems around it matter more than any single script. As a Data Engineer II, you'll build and run that layer: how data is sourced, kept fresh, verified, indexed, and served.

Key Responsibilities

• Build and operate the infrastructure behind a large fleet of scrapers: scheduling, queueing, retries, proxy and session management
• Design validation and completeness checks that tell us when data is missing, stale, or wrong, without anyone having to read the scraper
• Define the testing, review, and deployment practices that make AI-assisted scrapers safe to ship and easy to replace
• Support on-demand data fetching alongside scheduled ingestion
• Own indexing and storage decisions for search and analytics at scale
• Build the alerting and observability that surface failures before users do
• Evaluate new data sources and decide what's worth building
• Review code, mentor teammates, and raise the bar

Required Skills

• 2–4 years building and running data pipelines or scrapers in production
• Strong Python
• Solid grasp of HTTP at depth: sessions, auth patterns, rate limiting, bot protection, proxies
• Experience with queues and async processing (SQS, RabbitMQ, Kafka), including retries, idempotency, and dead-letter handling
• SQL and NoSQL fluency, and experience with a search system (Solr, Elasticsearch, OpenSearch)
• Working knowledge of AWS
• A habit of instrumenting what you build: metrics, dashboards, alerts

Nice to Have

• Headless browser automation at scale
• OCR and document extraction tooling
• LLM-based extraction or evaluation pipelines
• Grafana or similar observability tooling

What We Value

You don't need to have worked in our domain. We care about how you think about failure, data quality, and scale, and whether you can take something ambiguous, break it down, and ship something others can rely on.