AI Tools Used by English Councils Downplay Women’s Health Issues, Study Finds

Artificial intelligence is creeping into the daily workings of England’s councils. Promoted as a way to ease the burden on social workers stretched thin by years of austerity, these tools are already quietly shaping decisions about who gets care, and how much. But new research suggests they may also be introducing a subtler and more dangerous problem: gender bias.

A 2025 study by the London School of Economics and Political Science (LSE) has found that some of the AI models used to summarise social care case notes describe men and women differently - and not in ways that reflect their actual needs.

When identical case notes were entered into Google’s “Gemma” model, simply changing the subject’s gender led to dramatically different assessments.

One summary described an elderly man as having “a complex medical history, no care package and poor mobility.” The same notes, rewritten for a woman, became: “Despite her limitations, she is independent and able to maintain her personal care.”

“It was striking how consistent the pattern was,” says Dr Sam Rickman, lead author of the LSE report, when talking to The Guardian about their research findings. “Men were more likely to be labelled as ‘unable’, ‘disabled’ or ‘complex’. For women, needs were minimised, reframed or missed out altogether.”

A hidden layer of bias

The findings raise uncomfortable questions about how technology is being embedded into frontline services. More than half of England’s councils are already using some form of AI in adult social care, though there is little transparency about which models are deployed, how they are tested or whether staff are aware of potential pitfalls.

Local authorities under financial pressure have embraced automation as a way to speed up paperwork and free social workers to spend more time face-to-face with clients. Summarising lengthy case files is one of the most time-consuming parts of the job, and AI can cut hours down to minutes.

But the LSE research suggests these shortcuts risk reinforcing stereotypes that women are more resilient, more capable of “managing” on their own – and therefore less in need of help. Because eligibility for care is directly tied to how “need” is described, even small differences in wording can translate into real-world disparities.

“Unequal descriptions become unequal decisions,” Rickman warns. “If women’s conditions are consistently downplayed, they could end up with fewer services, less funding and reduced support.”

Austerity, workload and the lure of AI

For years, councils have struggled to meet rising demand for care with shrinking budgets. Since 2010, local government spending power has fallen by nearly a third in real terms. Social workers juggle high caseloads, paperwork backlogs and recruitment crises.

Against this backdrop, the promise of AI has proved irresistible. Officials frame it as a neutral, efficient tool – a way to “do more with less”. But campaigners fear this framing ignores how biases baked into training data can silently skew outcomes.

Google under scrutiny

Among the models tested, Google’s Gemma showed the most pronounced disparities. Meta’s Llama 3, by contrast, produced no measurable gender differences.

Google said it would examine the findings, noting that the study used Gemma’s first generation and that newer versions are expected to perform better. The company has not positioned Gemma as a tool for medical or care decision-making, though in practice public bodies are already experimenting with it for precisely those purposes.

The opacity of this experimentation worries Rickman. “We don’t actually know which models are in use, or how often. That makes it very hard to hold anyone accountable when things go wrong.”

Echoes of older inequalities

The LSE paper adds to a wider body of evidence that machine learning systems can reproduce the prejudices of the data they are trained on. A US review of 133 AI systems found 44% displayed gender bias and 25% showed both gender and racial bias.

In healthcare, this can interact with long-standing systemic inequalities. Women’s pain has historically been taken less seriously by doctors; conditions such as endometriosis have gone undiagnosed for years. AI, far from eliminating these disparities, risks amplifying them.

Calls for regulation

The report concludes with a stark recommendation: regulators should mandate the measurement of bias in any AI used for long-term care, to ensure “algorithmic fairness” is prioritised.

Rickman says safeguards must be built in now, before flawed systems become entrenched. “AI in the public sector is already here. The question is whether it will widen inequalities or help close them. Without transparency and oversight, the risks are clear.”

For social workers on the ground, the technology may be welcome in principle – but many say it cannot replace human judgement. Ultimately, tou can’t capture someone’s dignity or vulnerability in a bullet point summary.

If AI says a woman is coping fine, we need to ask: compared to what?”

The growing use of AI in local government is often portrayed as a story of innovation and efficiency. But as this research shows, it may also be a story about whose needs count, and whose are quietly erased.