Early-stage startup building training infrastructure for autonomous AI agents

Head of Quality, Member of Technical Staff

San FranciscoOn-site$180K - $280K1 - 5 yearsReferral bonus eligible
Referral bonus eligible. Know someone who would be great for this role? Refer them to a Fluency Digital recruiter. If they're placed, you may be eligible for a referral bonus of up to $5,000 (terms apply). Refer someone for this role
You'll architect and run quality systems that determine whether agent behaviors are reliable enough to ship. This means designing evaluation frameworks, building automated testing pipelines, and establishing the bar for what 'good' looks like when software teaches itself. You'll report directly to technical leadership in a San Francisco office where ten people are figuring out how to make agent training reproducible at scale.

What they're looking for

  • 3+ years in software quality, ML evaluation, or agent systems—preferably in production environments
  • Deep familiarity with evaluation metrics for generative or agentic models; you can spot a brittle benchmark from a mile away
  • Experience building test infrastructure from scratch, not just optimizing existing suites
  • Comfort owning tradeoffs between research thoroughness and shipping velocity at a seed-stage company
  • Strong Python and systems thinking; you debug processes as eagerly as you debug code

Apply for this role

We use AI to help match your experience to the roles where you fit best. A member of our team reviews every application, and all hiring decisions are made by people.

This role is one we're recruiting for on behalf of a client company; the client's identity is kept confidential at this stage. A Fluency recruiter will follow up with details.

Share this role