Introduction
Not all bias audit tools are created equal. This guide helps you evaluate options and choose the right tool for your organization's compliance needs.
Key Evaluation Criteria
1. Regulatory Compliance
The tool must produce audit reports that meet specific regulatory requirements:
- NYC Local Law 144: Impact ratios for race/ethnicity and sex
- California AB 331: Impact assessments with broader scope
- EU AI Act: Conformity assessment documentation
- EEOC Guidance: Four-fifths rule and statistical testing
2. Statistical Methodology
Look for tools that include:
- Four-fifths rule (impact ratio) calculation
- Chi-squared test for statistical significance
- Fisher's exact test for small samples
- Intersectional analysis across combined categories
- Score distribution analysis (Kolmogorov-Smirnov test)
3. Independence
NYC Local Law 144 requires audits by an "independent auditor." Ensure the tool or provider qualifies as independent — in-house tools do not satisfy this requirement.
4. Data Security
Your hiring data is sensitive. Verify:
- Encryption in transit and at rest
- SOC 2 compliance or equivalent
- Data residency options (important for EU AI Act)
- Data retention and deletion policies
5. Ease of Use
The tool should be accessible to HR professionals, not just data scientists:
- Automated data column mapping
- Clear, non-technical results presentation
- Actionable remediation recommendations
- Self-service operation without consulting support
6. Report Quality
Audit reports should be:
- Publication-ready for website posting (LL144 requirement)
- Tamper-evident (cryptographic hashing)
- Comprehensive with methodology documentation
- Understandable by regulators, lawyers, and HR teams
Questions to Ask Vendors
- Does your audit satisfy NYC LL144 independence requirements?
- What statistical tests do you perform beyond the four-fifths rule?
- Do you include intersectional analysis?
- How do you handle small sample sizes?
- What is the turnaround time for audit results?
- How do you keep up with regulatory changes?
- What data security certifications do you hold?
- Can we run unlimited re-audits?
- What format are reports delivered in?
- Do reports include remediation recommendations?
Red Flags to Avoid
- Tools that only calculate the four-fifths rule without statistical testing
- Providers that don't address the independence requirement
- Tools requiring months of consulting engagement for basic audits
- Vendors that can't explain their statistical methodology
- Solutions without intersectional analysis capability
- Tools that don't generate regulatory-specific reports
Why Organizations Choose OnHirely
OnHirely checks every box:
- Independent auditor that satisfies LL144 requirements
- Comprehensive statistical methodology (four-fifths rule + chi-squared + Fisher's + KS test)
- Full intersectional analysis
- Results in minutes, not months
- Self-service platform with automated data mapping
- SHA-256 verified PDF reports
- Multi-regulation support (LL144, AB 331, EU AI Act)
- Affordable subscription pricing with unlimited re-audits