Remote opportunity at
DEF CONModel Test and Measurement Engineer
DEF CON operates as an insights organization utilizing artificial intelligence, data analytics, mathematical optimization, and software engineering to help complex systems achieve resilient optimization. The analytics…
Career Tools
About This Role
DEF CON operates as an insights organization utilizing artificial intelligence, data analytics, mathematical optimization, and software engineering to help complex systems achieve resilient optimization. The analytics and AI engineering group develops a platform that merges numerous data sources, accurately identifies individuals, highlights priority items for analysts, and generates transparent recommendations within a secure government cloud framework. The organization seeks a Model Test and Measurement Engineer to oversee the evaluation framework…
Job Description
DEF CON operates as an insights organization utilizing artificial intelligence, data analytics, mathematical optimization, and software engineering to help complex systems achieve resilient optimization. The analytics and AI engineering group develops a platform that merges numerous data sources, accurately identifies individuals, highlights priority items for analysts, and generates transparent recommendations within a secure government cloud framework.
The organization seeks a Model Test and Measurement Engineer to oversee the evaluation framework that guarantees system performance. This position suits an independent professional who can act as an objective voice to validate scoring and generative artificial intelligence capabilities without building models or authorizing releases directly. The function involves establishing labeled ground truth datasets, creating audit and sampling techniques, tracking performance across software versions, and producing evidence documentation for deployment choices.
Collaborating alongside data scientists, technical leadership, and AI engineers, this individual measures workflow efficiency including throughput, review duration, backlog progression, and override metrics. The role operates on a fully remote basis, though occasional travel may occur to partner facilities, customer sites, and company headquarters.
Responsibilities
- Build and maintain labeled ground truth datasets for evaluating models
- Develop and execute sampled audit methodologies
- Gate software releases using model versions, maintaining inventory, evaluation logs, and rollback triggers
- Conduct drift and override reviews
- Generate documentation concerning human oversight, fairness, and disparate impact
- Track workflow efficiency metrics such as review duration, throughput, backlog movement, and rework frequencies
- Design quality control procedures during evaluation phases
- Define measurement events for other teams and verify that the review workspace emits workflow events
Requirements
- Minimum of 5 years of professional experience with model validation as a primary responsibility
- Direct background in ground-truth creation and sampling design
- Proficiency in Python and SQL
- Ability to operate independently from teams whose models are under evaluation
- United States citizenship
- Active US Secret clearance
Qualifications
- Background in government or regulated-industry model risk
- Previous engagement with fairness and disparate-effect testing plus human-oversight documentation
- Familiarity with the NIST AI RMF framework or comparable practices
- Active Top Secret security clearance
Core Skills
Benefits
- Competitive salary, equity, and bonus offerings
- Comprehensive health insurance covering medical, dental, and vision 100 percent for employees and families
- Unlimited paid time off subject to manager approval
- Flexible work environment centered on results
- 14 weeks of fully paid parental leave
Frequently Asked Questions
What is the remote work policy for this position?
This is a fully remote role, though it requires occasional travel to company headquarters, customer sites, and partner facilities as needed.
What is the salary range for the Model Test and Measurement Engineer role?
The typical salary range for this position is between $150,000 and $190,000, determined by experience, skills, and other factors.
What security clearance is required to apply?
Applicants must hold United States citizenship and an active US Secret clearance, with elevated personnel security requirements applying to certain portions of the work.
How many years of experience are required?
Candidates must have at least 5 years of experience with model validation as a named responsibility rather than a secondary task.
Sample Interview Questions
AI-generated questions tailored to this specific role — a preview of the full practice set.