Introduction
AI hiring tools promise efficiency and objectivity, but they can also perpetuate and amplify existing biases at unprecedented scale. This guide covers everything you need to know about conducting a comprehensive AI hiring bias audit.
Chapter 1: Understanding AI Hiring Bias
What Is AI Hiring Bias?
AI hiring bias occurs when automated employment tools produce systematically unfair outcomes for certain demographic groups. This can happen through:
- Training data bias: Models learn from historical data that reflects past discrimination
- Proxy variables: Seemingly neutral features that correlate with protected characteristics
- Measurement bias: Assessments that measure background rather than capability
- Aggregation bias: Models that perform well on average but poorly for subgroups
Why AI Bias Matters
A single biased AI tool can affect thousands of candidates daily. Unlike human bias, which varies by individual, AI bias is consistent and scalable — the same discriminatory pattern is applied to every candidate.
Chapter 2: The Regulatory Landscape
NYC Local Law 144
The first US law mandating independent bias audits for AEDTs. Requires annual audits, publication of results, and candidate notification.
California AB 331
Broader scope than LL144, covering automated decision systems in employment with impact assessments and opt-out requirements.
EU AI Act
Classifies employment AI as high-risk with comprehensive requirements for risk management, data governance, transparency, and human oversight.
Colorado AI Act
Requires reasonable care to prevent algorithmic discrimination, with impact assessments and reporting requirements.
EEOC Guidance
Clarifies that Title VII applies to AI hiring decisions and employers are liable for vendor-provided tools.
Chapter 3: Bias Audit Methodology
Step 1: Data Collection
Gather historical hiring data including:
- Candidate demographics (race, gender, age)
- Outcomes at each pipeline stage (screened in/out, advanced, hired)
- AI scores or ratings if available
- Time period of at least one year
Step 2: Impact Ratio Calculation
For each protected category and pipeline stage:
- Calculate the selection rate for each group
- Identify the group with the highest rate
- Divide each group's rate by the highest rate
- Flag any ratio below 0.80
Step 3: Statistical Significance Testing
- Chi-squared test for sample sizes > 40
- Fisher's exact test for sample sizes ≤ 40
- Significance threshold: p < 0.05
Step 4: Intersectional Analysis
Cross-tabulate outcomes for combinations of protected characteristics to detect compound discrimination.
Step 5: Score Distribution Analysis
If AI scores are available, use the Kolmogorov-Smirnov test to compare score distributions across groups.
Step 6: Reporting and Remediation
Generate a comprehensive report with findings, compliance status, and actionable remediation recommendations.
Chapter 4: Best Practices
- Audit before deployment: Test AI tools before they affect real candidates
- Audit regularly: Annual minimum, quarterly recommended
- Use independent auditors: Required by LL144 and best practice for credibility
- Include intersectional analysis: Single-axis analysis misses compound bias
- Monitor continuously: Bias can emerge over time as candidate pools change
- Document everything: Maintain audit trails for regulatory defense
- Remediate promptly: Address identified bias before it creates legal liability
Chapter 5: Getting Started with OnHirely
OnHirely makes bias auditing accessible and affordable:
- Sign up for a free account
- Upload your hiring data (CSV or Excel)
- Our AI automatically maps your data columns
- Review your bias audit results in minutes
- Download your compliance-ready PDF report
No statistical expertise required. No consulting engagement needed. Just clear, actionable bias analysis.