May 31, 2023
In a study conducted at Brigham and Women’s Hospital four years ago, it was found that the AI system, which is supposed to help patients in need of extra care, privileged relatively healthy white patients over sicker black patients. The AI was designed to reduce costs and sort patients based on their previous healthcare costs. However, according to Rachel Thomas, director of the University of San Francisco Center for Applied Data Ethics and co-founder of the research lab fast.ai, focusing on previous healthcare costs is what opened the door to bias: “The healthcare system is less inclined to give treatment to black patients dealing with similar chronic illnesses compared to white patients.” Studies reveal that black patients often have fewer convenient healthcare options despite possessing similar levels of insurance. “So when black patients spend less on medical care for the same illnesses, the algorithm assumes they do not need extra care as much as white patients.” Thomas explained that the root cause of this bias is that the algorithm was given the wrong data for the problem to be solved. Even more concerning is that this AI wasn’t isolated to Brigham and Women’s but was emblematic of algorithms of this kind that are sold to hospitals and affect up to 200 million people.
The above story reminds me of the Jean-Baptiste Alphonse Karr quote: “plus ça change, plus c’est la même chose”—the more things change, the more they stay the same. It raises the question of whether AI can be trained on the appropriate data to redress systemic racism or if it will only serve to perpetuate structural inequities.
This is the same question for those of us in education who are concerned about equity, given the many applications of AI already in use and wondering if AI can be applied in innovative ways to achieve better learning outcomes. According to Philipa Hardman, Creator of the DOMS™️, “For every AI-powered piece of ed-tech that pushes us towards more effective instruction, there are ten examples which push is in the opposite direction, using AI to automate and scale ineffective “chalk and talk” practices.” She cites two promising examples as the exception, “but the vast majority accelerate, automate and scale traditional, broken methods of instruction.” Given Hardman’s observation, it is apparent that AI’s potential for change depends on our appetite for transformation. Teachers, students,…