Staff Software Engineer (AI Evals)
Remote work open to candidates in: Cameroon, South Africa, Algeria, Angola, Benin, Botswana, Burkina Faso, Burundi, Cabo Verde, Comoros, Côte d'Ivoire, Djibouti, Egypt, Eritrea, Eswatini, Ethiopia, Gabon, Gambia, Ghana, Guinea, Equatorial Guinea, Guinea-Bissau, Kenya, Lesotho, Liberia, Libya, Madagascar, Malawi, Mali, Morocco, Mauritius, Mauritania, Mayotte, Mozambique, Namibia, Niger, Nigeria, Uganda, Central African Republic, Congo, The Democratic Republic of the, Congo, Réunion, Rwanda, Western Sahara, Saint Helena, Ascension and Tristan da Cunha, Sao Tome and Principe, Senegal, Seychelles, Sierra Leone, Somalia, Sudan, South Sudan, Tanzania, Chad, Togo, Tunisia, Zambia, Zimbabwe, Åland Islands, Albania, Germany, Andorra, Austria, Belarus, Belgium, Bosnia and Herzegovina, Bulgaria, Cyprus, Croatia, Denmark, Spain, Estonia, Finland, France, Gibraltar, Greece, Guernsey, Hungary, Isle of Man, Faroe Islands, Ireland, Iceland, Italy, Jersey, Latvia, Liechtenstein, Lithuania, Luxembourg, North Macedonia, Malta, Moldova, Monaco, Montenegro, Norway, Netherlands, Poland, Portugal, Romania, United Kingdom, Russian Federation, San Marino, Holy See (Vatican City State), Serbia, Slovakia, Slovenia, Sweden, Switzerland, Svalbard and Jan Mayen, Czechia, Ukraine
Job description
Evaluate the quality of interactions with modern coding agents such as OpenAI Codex and Claude Code. This is not a traditional engineering role: you will assess whether models think and communicate like great engineers, rather than write production code. You will evaluate AI-generated coding interactions in real-world scenarios, focusing on whether responses are useful, correct at a high level, and aligned with strong engineering judgment. You will assess explanations and reasoning, distinguish levels of response quality, provide direct feedback, and help define what good interactions with tools like Cursor look like. The work requires making subjective but rigorous judgments about whether explanations are helpful, whether models guide users well, and whether interactions build or erode trust. Engagement details: Contract; approximately 20 hours or more per week; through October, with possible extensions; start as soon as possible. The selection process includes a take-home evaluation exercise, a background check, and an offer.
Requirements
Staff / Principal-level engineer or equivalent experience. Strong background in Go, Python, Rust, TypeScript, or JavaScript. Hands-on experience using OpenAI Codex, Claude Code, and Cursor. Deep familiarity with modern AI-assisted development workflows. Able to evaluate code without needing to execute it or review every line in depth. Comfortable giving direct, opinionated feedback and maintaining a high bar for engineering quality. Nice to have: Experience with Cursor or similar AI-first IDEs; prior exposure to prompt design or evaluation workflows; experience mentoring senior engineers or defining engineering standards.