May 31, 2023
In a study conducted at Brigham and Women’s Hospital four years ago, it was found that the AI system, which is supposed to help patients in need of extra care, privileged relatively healthy white patients over sicker black patients. The AI was designed to reduce costs and sort patients based on their previous healthcare costs. However, according to Rachel Thomas, director of the University of San Francisco Center for Applied Data Ethics and co-founder of the research lab fast.ai, focusing on previous healthcare costs is what opened the door to bias: “The healthcare system is less inclined to give treatment to black patients dealing with similar chronic illnesses compared to white patients.” Studies reveal that black patients often have fewer convenient healthcare options despite possessing similar levels of insurance. “So when black patients spend less on medical care for the same illnesses, the algorithm assumes they do not need extra care as much as white patients.” Thomas explained that the root cause of this bias is that the algorithm was given the wrong data for the problem to be solved. Even more concerning is that this AI wasn’t isolated to Brigham and Women’s but…