Algorithm Bias In Medicine

Artificial intelligence (AI), particularly its ability to sort quantitative and qualitative data into a conglomeration of potential symptoms, has long been used to pursue a narrow mission in health care: efficient clinical diagnoses. Beginning in 1971 with the development of the INTERNIST-1, the world's first artificial medical consultant, AI has advanced to a profusion of uses in modern medicine, including robotic-assisted surgery and genomics. But despite the streamlined efficiency of medical diagnoses, the use of AI technology in the health care realm opens a Pandora's box of potential inequities, rendering medical clientele open to discrimination or medical malpractice suits.

Even while AI machines use specific algorithms composed of several learning methods, such as logistic regression, linear regression, clustering, dimension reduction, and bootstrap aggregating, when provided with inclusive data, human biases remain, skewing the results and potentially leading to discriminatory practices. This article explores the intersection of AI-based medical diagnoses and evidentiary considerations for malpractice suits.

Background

Medical diagnostics is the foundation of effective medical treatment. Without an appropriate and accurate diagnosis, treatment is all but impossible. As humanity has consistently evolved, creating new tools to revolutionize the process—such as the discovery of X-rays in 1895, magnetic resonance imaging in the 1970s, and computed tomography (CT) scans in 1972—the training of medical doctors has similarly improved, leading to faster diagnoses. To further advance the process, AI was introduced to compile all images, symptoms, and genealogy, allowing the machine to highlight potential causes and enabling doctors to readily treat patients.

According to a randomized clinical trial published by the JAMA Network on Oct. 28, 2024, titled, "Large Language Models Influence Diagnostic Reasoning," physicians using conventional diagnostic resources had a median reasoning score of 74%, while an AI model working independently had a median reasoning score of 92%, indicating that the AI model outperformed doctors by an absolute score difference of 16%. Per a survey conducted by the American Medical Association, documented by Tanya Albert Henry in AMA coverage of digital health trends on Feb. 26, 2025, 66% of physicians reported that they used AI in 2024, when just a year earlier, 62% reported that they did not use AI—displaying a substantial increase in its use in practice. Considering the significant growth in AI use, the effects of such practices on the legal realm are largely unexamined, leaving ambiguity regarding the admissibility of such diagnostic practices in medical malpractice cases. Rather, it would seem that the industry, intentionally or not, has left itself open to discrimination and medical malpractice claims.

The Humanistic Aspects of AI

Artificial intelligence is no stranger to discriminatory biases. Throughout its development, several issues have arisen. According to Dr. Gideon Christian, an assistant professor at the University of Calgary, who received a grant to examine the impacts of AI facial recognition technology on race, gender, and privacy, "In some facial recognition technology, there is an over 99% accuracy rate in recognizing white male faces. But unfortunately, when it comes to recognizing faces of color, especially the faces of Black women, the technology seems to manifest its highest error rate, which is about 35%." In the creation of AI, bias is introduced into the system in one of four ways: data collection, data labeling, data preprocessing, and deployment. In data collection, if the data used is primarily composed of one race, gender, or national origin, the results will be subjective, particularly highlighting issues that face that specific dataset, resulting in skewed data, similar to the issues confronting facial recognition technology.

Data labeling provides coders the opportunity to introduce inaccurate or prejudicial labels. For example, in the Science article, "Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations," appearing in the Oct. 25, 2019 issue, the authors describe an AI algorithm that used health costs as a proxy for health needs, incorrectly concluding that Black patients are healthier than equally sick white patients because less money was spent on them, thus giving higher priority to white patients when treating life-threatening conditions such as diabetes and kidney disease.

Even when preparing raw data through data preprocessing, bias can be introduced. Processors may deem a result in a cancer treatment data collection to be an outlier, scrubbing the result from the survey. However, the result may be a reflection of historical biases affecting a minority population. For example, according to "The GoodRx Effect" 2023 white paper, about 16.8 million Black Americans (approximately one in three) live in counties with little to no access to heart specialists, requiring them to commute well over 80 miles to reach the nearest cardiology clinic, with the highest number of cardiology desert counties being in Georgia, Mississippi, Virginia, Alabama, and Louisiana.

If data were collected from Georgia as a whole, any data from one of these desert counties could be considered an extreme outlier, requiring it to be removed and causing the AI to assume that fewer Black Americans suffer from heart conditions. Finally, in the deployment of AI, if the underlying training data were biased, the use of the results would certainly present ethical dilemmas. For example, in his Nov. 21, 2023, Medium article "COMPAS: Unfair Algorithm?," Patalay Prathamesh explains that in the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) algorithm used by U.S. courts to assess the risk of an individual reoffending after being previously arrested, Black defendants were twice as likely to be misclassified as high risk, while white defendants were more likely to be misclassified as low risk. As a result of COMPAS, Black defendants received significantly longer sentences in comparison to white defendants. Given the potential for bias to be introduced into AI at any step, medical entities must be diligent in accounting for bias in AI query results. Failing to do so could result in incorrect medical conclusions or discriminatory practices.

Potential Bias for Discrimination & Malpractice Claims

As stated in Costin v. Glens Falls Hosp., 103 F.4th 946 (2d Cir. 2024), citing McGugan v. Aldana-Bernier, 752 F.3d 224, 231 (2d Cir. 2014), "The federal law of discrimination does not review the conduct of a doctor who administers a medical treatment to a patient (or withholds it) because the doctor's medical training leads her to conclude that the treatment is medically appropriate (or inappropriate)[.] If the treatment is merely deficient, imprudent, or harmful, the matter is one of medical malpractice." While this statement stands, a biased AI program could simultaneously spawn medical malpractice claims by leading doctors to perform substandard care, and discrimination claims if such care disproportionately impacts patients on the basis of protected characteristics such as race, gender, disability, age, or national origin. Such discrimination has presented previously in the medical field.

In Welch v. United Network for Organ Sharing, the court recognized that the use of a race-based coefficient, multiplying a patient's glomerular filtration rate (GFR) to determine whether a kidney transplant could occur, was discriminatory, resulting in a systemic underestimation of kidney disease severity for many Black patients. [See Welch v. United Network for Organ Sharing, 767 F. Supp. 3d 746 (M.D. Tenn. 2025), adhered to on reconsideration, No. 3:24-CV-00422, 2025 WL 824137 (M.D. Tenn. March 14, 2025)].

While Title VI of the Civil Rights Act of 1964 requires intentional discrimination, the failure to remedy known acts of discrimination can also satisfy the intent requirements. [42 U.S.C. § 2000d; See Jalen Z. v. Sch. Dist. of Philadelphia, 104 F. Supp. 3d 660, 684 (E.D. Pa. 2015); Welch v. United Network for Organ Sharing, 767 F. Supp.3d at 765-66]. As such, if a hospital uses an AI program knowing the chance for biased results and fails to remedy those biases or disregards the risks, the hospital is susceptible to litigation. Overall, it is critical that the health care industry be aware of the risks of AI use and the potential for harm. Will AI merely be a tool to efficiently diagnose patients, or will it become a new source of litigation? The answer remains to be seen.

Originally published by CLM Magazine

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.