Building the infrastructure to rigorously test and benchmark large language models.
Evaluations Engineer
Language models are reshaping how we work, but the tools to measure their actual capabilities haven't kept pace. This team is closing that gap—designing evaluation pipelines that probe reasoning, coding, retrieval, and safety at a depth most benchmarks gloss over. As an Evaluations Engineer, you'll own the full lifecycle of an eval: from authoring novel test suites and architecting scalable execution backends in Django and Python, to building React interfaces that let researchers interact with results. The work sits at the intersection of rigorous engineering and empirical AI research, and every experiment you ship directly shapes which models get deployed in high-stakes settings.
What they're looking for
- 1–4 years of professional experience building and shipping production software in Python, with fluency in Django or a comparable web framework
- Comfortable working across the stack—you can debug a React component, design a REST endpoint, and reason about cloud infrastructure on AWS
- Hands-on experience with LLM APIs (such as OpenAI) and a strong intuition for prompt design, structured output, and model failure modes
- A bias toward clean abstractions and reproducible experiments; you treat evaluation code like production code, not a one-off script
- Thrives in a collaborative, on-site team in San Francisco where engineering, product, and research decisions happen in the same room