Security Engineer, AI Evaluation (PhD)

Job Type

Flexible Contract (Remote) Job Summary: Cobalt is hiring PhD-level security engineers to review reinforcement learning tasks and environments used to test whether frontier AI models can carry out real security work. Your job is to make sure each task is correct, fair, realistic and actually measures what it claims to measure before it is accepted. If you have built your research or career around finding the flaw that scanners miss, the patch that only looks like a fix, or the exploit path nobody planned for, this role puts that experience to work at the frontier of AI evaluation.

What you will do

• Review security tasks end to end, including the task description, environment code, containers, vulnerable code, dependencies, reference solution and automated tests or graders, and confirm that each part is correct and consistent with the others.
• Run reference solutions and graders yourself to confirm that correct fixes pass, incorrect or incomplete fixes fail, and results are reproducible across runs.
• Look for ways an agent could pass without doing the intended work, such as reading the expected output directly, special-casing the grader’s test inputs, disabling a feature instead of fixing the flaw, or applying a patch that blocks the test exploit but leaves the underlying vulnerability open. Document each issue so the task author can fix it.
• Confirm that graders verify real security outcomes thoroughly and deterministically, including variant inputs and regression checks that keep legitimate behavior working.
• Review how a frontier AI model attempted each task and judge whether its failure reflects a real capability gap, or instead reflects ambiguous wording, a hidden assumption, a broken environment or an unfair grader.
• Judge whether each task reflects how security work is actually done in industry, including realistic codebases, messy dependencies and real-world constraints, and push back on tasks that are unrealistic, trivial or testing the wrong skill.
• Write clear, specific feedback to task authors and work with them through revisions until the task meets the acceptance standard.

Who we are looking for

• A PhD in computer science, security, engineering or a closely related STEM field. PhD candidates and students are welcome to apply.
• Hands-on research or industry experience in security, such as vulnerability research, secure code review, reverse engineering, malware analysis, applied cryptography or systems security. Ten or more years of experience is ideal.
• At least one publication, academic or professional, such as a paper, CVE write-up, conference talk or technical advisory.
• Expert command of Python and at least one systems or web language, and the ability to read code and explain exactly why it is or is not vulnerable.
• Hands-on experience with security tooling, such as debuggers, fuzzers, static and dynamic analysis tools or reverse engineering tools.
• Deep familiarity with Linux, Docker and Git, enough to set up an environment, run a task and trace a failure.
• Careful, direct written communication. Much of the job is explaining precisely what is wrong and why.

Strong advantages

• Experience designing technical assessments, CTF challenges or interview problems, writing autograders, or serving as a peer reviewer for a security conference or journal.
• Significant open source security contributions, bug bounty track record or published vulnerability disclosures.
• Experience with machine learning security, or with reinforcement learning, including how reward signals can be gamed.

Why Cobalt AI

• Work on frontier AI evaluation. Your judgment directly determines which tasks are used to measure what leading models can and cannot do in security.
• See how frontier models are trained and assessed. Reviewing many tasks and model attempts gives you a close view of where current models succeed in security work and where they break.
• Fit the work around your other commitments. Set your own schedule, with flexible weeks of 10 to 40 hours.
• Shape the work. Strong reviewers can lead task design, run projects and collaborate on research.

Job Type: Part-time

Work Location: Remote