A Failed Audit Is Not the End
Receiving a bias audit report that flags adverse impact can be alarming, especially for organizations conducting their first audit. But a failed audit is not a legal finding, not a lawsuit, and not a public scandal. It is an internal finding that gives you the opportunity to fix a problem before it becomes any of those things.
This guide walks you through exactly what to do when your audit identifies bias in your AI hiring tool.
Understanding Your Audit Results
What the Numbers Mean
A bias audit typically reports:
- Impact ratios for each protected group compared to the highest-performing group
- Statistical significance (p-values) indicating whether disparities are likely due to chance
- Intersectional results showing outcomes for combinations of protected characteristics
An impact ratio below 0.80 indicates potential adverse impact under the four-fifths rule. A statistically significant finding (p < 0.05) means the disparity is unlikely to be random.
Severity Assessment
Not all failures are equal. Assess the severity:
- Impact ratio 0.70-0.80, not statistically significant: Moderate concern. Monitor closely and investigate, but do not panic
- Impact ratio 0.70-0.80, statistically significant: Serious concern. Requires prompt investigation and remediation
- Impact ratio below 0.70, statistically significant: Critical concern. Requires immediate action, potentially including suspension of the tool
- Multiple groups affected: Higher severity than a single-group finding
- Intersectional findings: May indicate deeper systemic issues
Step 1: Do Not Panic, Do Not Ignore
Two common reactions are equally harmful:
Panic reaction: Immediately shutting down all AI tools, issuing public statements, or blaming vendors. This creates unnecessary disruption and may attract attention to a problem you can quietly fix.
Ignore reaction: Dismissing the findings as "just statistics" and continuing business as usual. This creates growing legal exposure and guarantees the bias continues to affect real candidates.
The correct response is measured, systematic, and documented.
Step 2: Assemble Your Response Team
Bring together:
- HR Leadership: To understand the business impact and make process decisions
- Legal Counsel: To advise on regulatory and litigation risk (consider engaging outside counsel for privilege protection)
- Data Science / Analytics: To perform root cause analysis
- The AI Vendor: To explain their tool's methodology and participate in remediation
- Compliance: To ensure remediation meets regulatory requirements
Step 3: Conduct Root Cause Analysis
Determine why the bias exists. Common causes include:
Training Data Bias
The model learned from historically biased hiring data. Check whether the training data overrepresents certain groups among "successful" outcomes.
Proxy Variables
Features in the model correlate with protected characteristics. Common proxies:
- Zip code (proxy for race)
- Name patterns (proxy for ethnicity/gender)
- University prestige (proxy for socioeconomic status/race)
- Years since graduation (proxy for age)
- Employment gaps (proxy for gender/disability)
Threshold Effects
The selection threshold may interact with score distributions in ways that disproportionately exclude certain groups. A threshold set at the 70th percentile might create no adverse impact, while a threshold at the 80th percentile creates significant adverse impact.
Feature Weighting
Certain features may receive disproportionate weight in ways that disadvantage specific groups. For example, heavy weighting on "years of continuous employment" may disadvantage women who took parental leave.
Step 4: Develop a Remediation Plan
Based on the root cause, select appropriate remediation strategies:
For Training Data Issues
- Retrain the model on more balanced data
- Apply data augmentation techniques to increase representation of underrepresented groups
- Use debiasing techniques during training
For Proxy Variable Issues
- Remove or mask proxy variables
- Apply adversarial debiasing to reduce the influence of proxy variables
- Replace proxy-laden features with more job-relevant alternatives
For Threshold Issues
- Adjust thresholds to reduce adverse impact while maintaining selection quality
- Implement banded scoring (treating candidates within a score band as equivalent) rather than strict rank-ordering
- Add human review for candidates near the threshold
For Feature Weighting Issues
- Reduce weight of biased features
- Replace biased features with validated, job-relevant alternatives
- Apply fairness constraints during model optimization
Step 5: Validate the Fix
After implementing remediation:
- Re-audit: Run the same bias audit on the remediated tool using the same data
- Compare: Verify that impact ratios have improved for the affected groups
- Check for new bias: Ensure remediation did not create new adverse impact against other groups
- Test with new data: If possible, run the remediated tool on a new dataset to confirm the fix generalizes
- Monitor: Implement ongoing monitoring to ensure the fix holds over time
Step 6: Document Everything
Maintain detailed records of:
- The original audit findings
- Your root cause analysis
- The remediation actions taken
- The re-audit results
- The timeline of all activities
- Decisions made and their rationale
This documentation serves multiple purposes:
- Demonstrates good faith compliance efforts to regulators
- Provides a defense record if litigation arises
- Creates institutional knowledge for future audits
- Supports continuous improvement of your hiring practices
Step 7: Assess Whether Affected Candidates Need Remediation
This is a judgment call that should involve legal counsel:
- If the bias was severe and affected a large number of candidates: Consider re-evaluating candidates who were screened out during the period of biased tool use
- If the bias was moderate and recently identified: Document the finding and focus on forward-looking remediation
- If specific candidates can be identified as harmed: Consult with legal counsel about individual remediation options
Step 8: Strengthen Ongoing Monitoring
A failed audit should trigger improvements to your monitoring process:
- Increase audit frequency (quarterly instead of annually)
- Add continuous monitoring with automated alerts
- Expand the scope of analysis (add intersectional analysis if not already included)
- Establish clear escalation procedures for future findings
- Train additional staff on bias audit interpretation
How OnHirely Supports Post-Audit Remediation
When an OnHirely audit identifies adverse impact, the platform provides specific remediation guidance based on the nature of the findings. After your vendor implements changes, you can re-audit immediately to verify the fix — no waiting for your next annual audit cycle. OnHirely's continuous monitoring catches drift early, preventing a repeat of the same problems.