The law broadly defines a “bias audit” as “an impartial evaluation by an independent auditor” to assess the AEDT’s disparate impact on the categories of race, ethnicity and sex.

New York City employers will soon be required to take a number of affirmative steps before using artificial intelligence (AI) in their employment decision-making processes. The automated employment decision tools (AEDT) law, passed by the New York City Council as Local Law Int. No. 1894-A, is aimed at protecting job candidates and employees from unlawful discriminatory bias based on race, ethnicity or sex when employers and employment agencies use AEDTs as part of employment decisions.

Although the law originally was scheduled to be effective January 1, 2023, the Department of Consumer and Worker Protection (DCWP) recently announced that it will begin enforcement on July 5, 2023, and published its Notice of Adoption of the Final Rule.

To Whom Does the Law Apply?

The law applies to all employers and employment agencies in New York City that use AEDTs to screen candidates or employees for positions located in the city and requires notice to residents for such positions. It does not apply to positions outside of New York City even if city residents apply for the position. As for fully remote positions, the law applies only if an employer has a New York City office, and it may apply to employers with physical locations both inside and outside of New York City depending on a “fact-specific” analysis of the position at issue, including how often work is completed in the city, whether work is performed for the city location and whether in-office work in the city will ever be required.

What Is an AEDT and Bias Audit?

Bias audits are required whenever an AEDT is used. Under the law, an AEDT is:

[A]ny computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision making for making employment decisions that impact natural persons.

The DCWP’s final rule defines the phrase “to substantially assist or replace discretionary decision making” as applied to an AEDT to mean relying solely on a simplified output (score, tag, classification, ranking, etc.) with no other factors considered, or to use a simplified output as one of a set of criteria where the simplified output is weighted more than any other criterion in the set, or to use a simplified output to overrule conclusions derived from other factors including human decision-making. Note, however, that the bias audit is necessary even though the employer is not using the AEDT to make the final hiring decision, but only to screen at an early point in the application process.

The law broadly defines a “bias audit” as “an impartial evaluation by an independent auditor” to assess the AEDT’s disparate impact on the categories of race, ethnicity and sex. The law does not apply to other protected classes such as age and disability. Bias audits must be conducted no more than one year before first using an AEDT. The final rule provides a list of several minimum calculations a bias audit must include when using an AEDT to score candidates for employment or employees being considered for promotion, or to select individuals to move forward in the hiring process, promote employees or classify individuals into groups (e.g., leadership styles).

The content of bias audits includes calculations of the selection rate for each category (or scoring rate, where the tool issues scores instead of classifications/groupings), including sex, race and ethnicity categories and intersectional categories, as well as the impact ratio of each category. “Impact ratio” means either (1) the selection rate for a category divided by the selection rate of the most selected category or (2) the scoring rate for a category divided by the scoring rate for the highest scoring category. The number of individuals whose race and gender are “unknown” and thus excluded from the calculations must be published. An independent auditor may exclude a category that comprises less than 2 percent of the data being used for the bias audit from the calculations of impact ratio, provided the summary of results includes the independent auditor’s justification for the exclusion, as well as the number of applicants and scoring rate or selection rate for the excluded category. Employers will need to ensure they are properly collecting and maintaining required data for the independent auditor’s use in conducting a bias audit.

Sources of Data Used in a Bias Audit

The final rule addresses the use of historical data and test data. Multiple employers or employment agencies using the same AEDT may rely on the same bias audit conducted using historical data of other employers or employment agencies only if the employers or employment agencies provided historical data from their own use of the AEDT to the independent auditor conducting the bias audit or if they have never used the AEDT. Test data may be used in the limited circumstances in which insufficient historical data is available to conduct a statistically significant bias audit. If test data is used, the summary of results must explain why historical data was not used and describe how the test data used was generated and obtained.

Use of an Independent Auditor

The final rule defines an independent auditor as “a person or group that is capable of exercising objective and impartial judgment on all issues within the scope of a bias audit of an AEDT,” and to that end specifically prohibits persons or groups with certain involvement or interest in the AEDT from being an independent auditor.

Published Results

A summary of the bias audit results and the AEDT’s distribution date to which the audit applies must be publicly available on the employer’s or employment agency’s website during use and for at least six months after the last use of the AEDT for any employment decision. The summary must include an explanation of the data used and selection/scoring rates and impact ratios for all categories.

Notice to Candidates and Employees

The law requires that job candidates and employees residing in New York City receive notice from employers and employment agencies concerning (1) the use of AEDTs in hiring and promotion assessments and evaluations, and (2) the job qualifications and characteristics that the AEDT will use. Prior to the use of an AEDT, employers and employment agencies must provide candidates and employees with at least 10 business days’ notice. Candidates and employees must be allowed to request an alternative selection process or a reasonable accommodation, though these terms are not defined under the law or final rule. The final rule provides the following methods by which employers and employment agencies can meet the notice requirement, assuming notice is provided at least 10 business days prior to use of an AEDT:

1. Notice to candidates for employment:
   1. Include notice on the careers or job section of its website;
   2. Include notice in a job posting; or
   3. Provide notice via U.S. mail or email.
2. Notice to employees being considered for promotion:
   1. Include notice in a written policy or procedure;
   2. Include notice in a job posting; or
   3. Provide notice via U.S. mail or email.

Employers and employment agencies will be required to provide information on the employment section of their website describing the type of data collected for the AEDT, the source of the data, their data retention policy and how to make a request for such information. Upon a candidate or employee’s written request, this information must be provided within 30 days, except where provision would violate local, state or federal law or interfere with a law enforcement investigation.

Penalties for Noncompliance

Violations will result in civil penalties of not more than $500 for a first violation (and each additional violation occurring on the same day as the first violation), and between $500 and $1,500 for each subsequent violation. The law makes clear that each day on which an AEDT is used in violation of the law gives rise to a separate violation, and that any failure to provide notice to a candidate or employee in violation of the law is also a separate violation. There is no cap on civil penalties available. Critically, the law authorizes a private right of action, in court or through administrative agencies, for aggrieved candidates and employees.

Related Guidance from the EEOC

Although there is no current comprehensive federal legislation concerning AI bias in employment, employers should look to the federal Equal Employment Opportunity Commission (EEOC)’s May 2022 guidance on AI for the types of issues that may arise based on AEDT use. The EEOC outlined a number of ways in which an employer’s AI tools may be found to violate the Americans with Disabilities Act, including:

1. Employers failing to provide a reasonable accommodation to applicants by relying on the AI tools.
2. The AI tools screen out individuals with disabilities that are able to perform the essential functions of the job with or without accommodation, such as speech pattern analysis for applicants with speech impediments.
3. The AI tools make a disability-related inquiry or otherwise constitute a medical examination.

The EEOC’s guidance also includes a number of practical tips employers should review, such as:  

1. Providing notice to job applicants and employees subject to the AI tool that accommodations are available;
2. Describing the AI tool’s rating factors and methods of assessment;
3. Ensuring the AI tool only measures abilities truly necessary for the job; and
4. Asking the AI tool vendor to confirm the tool’s legal compliance prior to purchase and use.

Then, in May 2023, the EEOC issued technical assistance concerning the use of AI in employment decisions, focusing on preventing discrimination from such use under Title VII of the Civil Rights Act of 1964. The EEOC technical assistance is summarized on our Class Action Defense Blog, with key takeaways including a need for employers to continually self-monitor their own AI use for possible disparate impact on protected classes of applicants and/or employees.

Other Similar Laws

Employers in the New York tri-state area should recognize that this law is likely to be the tip of the iceberg; similar bills have been introduced by the state legislatures of New York (AB 00567) and New Jersey (AB 4909) that seek to cover not only sex, race and ethnicity, but all other protected categories under each state’s respective anti-discrimination laws. Illinois and Maryland have already passed laws restricting use of AI in employment decision-making, with many other jurisdictions proposing similar laws.

Implication for Employers

With the law’s July 5, 2023, enforcement date less than one month away, employers in New York City who are using or plan to use AEDTs to screen candidates or employees should take affirmative steps now to comply with the law’s bias audit requirements and recognize the significant potential exposure to penalties for failure to comply. Employers should be aware of and stay tuned for further updates on these proposed state AI bias laws.

For More Information

If you have any questions about this Alert, please contact Eve I. Klein, Christopher D. Durham, Maria Cáceres-Boneau, Gregory Slotnick, any of the attorneys in our Employment, Labor, Benefits and Immigration Practice Group or the attorney in the firm with whom you are regularly in contact.

Disclaimer: This Alert has been prepared and published for informational purposes only and is not offered, nor should be construed, as legal advice. For more information, please see the firm's full disclaimer.