Can algorithmic registers solve automated bias?

Can algorithmic registers solve automated bias?

On 6 February 2021, parents targeted in the Dutch child benefits affair protest in front of the official residence of the Dutch prime minister in The Hague holding signs that say “Why have we been unfairly accused?” and “We will go on”.

(ANP/Belga/Phil Nijhuis)

In January 2021 the Dutch government collapsed because of a scandal that highlights the dangers of trying to administer essential governent services with artificial intelligence (AI). Between 2009 and 2019, in what has become known as the toeslagenaffaire (the benefits affair), around 26,000 parents were wrongly accused of committing childcare benefit fraud.

The Dutch tax services heavily targeted families from poorer backgrounds and those with ‘foreign sounding’ names, in some cases unnecessarily forcing them to pay back tens of thousands of euros’ worth of child allowances. Often the individuals involved had no access to legal recourse, driving many families into debt and divorce. Some families even lost their homes.

Part of the scandal involved an algorithm. One of the ways the Dutch government looked for fraud was via an automated system which scanned for potential signs of fraud. The flagged families were then hunted down by heavy-handed tax officials. Yet it turned out that the system was biased: having dual nationality was one reason to be flagged up by the system.

“It wreaked massive havoc,” says Nadia Benaissa, policy advisor at the Dutch NGO Bits of Freedom, which works on issues like internet freedom and algorithmic discrimination. “Discrimination and exclusion from the rule of law were very prominent in this case.”

The toeslagenaffaire is just one example of algorithmic discrimination. Algorithms have been shown to replicate social biases against, for example, ethnic minorities in criminal sentencing, predictive policing and even recruitment. Nevertheless there is a growing movement to make these systems more ethical. For example, a couple of major European cities have recently begun the process of opening up the algorithms they use, and allowing citizens to review them. These transparency initiatives are called algorithmic (or AI) registers, and in September 2020, Amsterdam and Helsinki became the first cities in the world to offer them.

Are AI registers enough to protect citizens?

The registers of Amsterdam and Helsinki offer a list of algorithms, what data goes into them, how they are stored and even what risks are present. For a camera algorithm that detects whether people in the street respect the 1.5 metres social distancing guidelines for Covid-19, the city of Amsterdam notes that the model doesn’t necessarily work well for all “skin colours, ages, sizes or clothing styles”. In another case the city of Amsterdam mentions their automated parking control system, which scans license plates to see if a car is allowed to park in the city centre. In Helsinki the register includes a chatbot for maternity clinics that answers questions related to pregnancy and child development.

Both registers were built by the Finnish company Saidot. Its CEO Meeri Haataja notes that it is more than just a transparency initiative. “It’s an AI governance platform,” she says. “We give public and private organisations that use AI at scale the tools to help them address the risks of those systems.”

Essentially, they sell a software service that gives organisations better oversight over the algorithms they employ. Previously, it might not have been very clear which algorithms a government used, even for members of the government’s own administration. Algorithms might be spread out between different departments, and key information, such as the data on which the system is based, might not be accessible for oversight.

Saidot’s platform collects all of that information in one place, which then allows organisations to better assess the risks posed by AI, and to communicate openly about them to the public or employees.

“We believe transparency is an essential tool for managing ethical risks like bias or unfairness,” says Haataja. “Our platform allows for assessments to take place, which helps accountability within the organisation. In turn you can inform stakeholders, like employees.”

So far, Saidot has done quite well with its proposition, sporting clients from Finnair to the smaller Finish city of Espoo. Similar initiatives have also started to pop up. In the Netherlands, calls have been made for an algorithm watchdog, accompanied by rules that would force organisations to disclose which algorithms they use. Experts have also been calling for mechanisms to audit algorithms and their underlying data.

But that might not be enough. “I think these registers are a good step towards more transparency,” says Benaissa. “Which is something [EU] citizens have a right to, as codified in the General Data Protection Regulation (GDPR). But we need to protect citizens in better ways, which is something a register doesn’t offer.” Haataja from Saidot agrees: “AI registers by themselves won’t solve everything,” she says. “Transparency is a necessary first step, but in the end, better AI governance requires a comprehensive approach.”

Using algorithms to discipline workers

The threat posed by algorithmic bias is very clear, and this extends into the world of work. Aída Ponce Del Castillo, senior researcher at the European Trade Union Institute (ETUI), notes the case of Amazon, where AI-equipped cameras automatically monitor the productivity of warehouse workers; in some cases, those not considered productive enough have been fired. But according to Ponce Del Castillo, this logic could be taken a step further. “Algorithms give information about workers to managers,” she says. “During, say, a unionisation attempt, these algorithms could track who certain workers are talking to.”

Algorithms could in this way discipline workers, which is something that humans have done for centuries, but mechanisms are now being integrated into automated systems. The coronavirus pandemic, for example, forced many workers to work from home, leading to an increase in remote surveillance. “Companies sometimes track and analyse the hours that someone is working in front of their computer,” says Ponce Del Castillo. “In some cases the software can even access the webcam without the worker knowing it. For some call centre workers there is software that analyses which words they use, to determine how friendly they are to customers.”

Making these automated systems public could be a step in the right direction, but that is just one piece of the puzzle. Ponce Del Castillo notes how existing European regulation already offers a framework to combat certain issues. “We have GDPR,” she says. “It addresses data and privacy, but also algorithms and automated decision-making. It doesn’t give all the answers, but it’s a great gateway to achieve better rules. It’s a revolutionary piece of legislation.”

One of the rights raised in GDPR is so-called ‘explainability’. An algorithm or automated decision-making system should, in theory, be able to give an explanation about why it made a certain decision. When an algorithm decides whether to give someone a loan or not, the client has the right to demand why the algorithm made the decision. This may seem straightforward, but such questions come with a whole range of challenges.

“According to GDPR every single automated decision should be explainable when it impacts individuals negatively,” says Ponce Del Castillo. “But what is explainability? Is it the code of the algorithm? Because that code can change over time, or it might be protected as a business secret. Or maybe it refers to the training data which is feeding the code? Explainability isn’t yet very well-defined in practice.”

Ponce Del Castillo mentions one technique that might accomplish explainability, called counterfactuals. Here an automated system would need to give scenarios where its decision might have been different. If an algorithm decides to reject your loan request, you could ask it under what circumstances it would have granted you the loan. The counterfactual could then simply note that if your monthly wage was higher, then the loan would have been granted. The counterfactual could also reveal that its decision depended on a more dubious variable, like gender or ethnicity, after which the decision could be cancelled, and the algorithm removed or redesigned.

Penalised by automated decisions

Benaissa from Bits of Freedom is calling for more specific regulation on algorithms. “The European Union is already working on new regulations for AI,” she says. “So hopefully we’ll see an improvement there in the near future.”

Bits of Freedom wants to go beyond what GDPR already lays out. “We are arguing for the ban of certain applications,” says Benaissa. “Algorithms shouldn’t decide whether someone can access basic services. Here we cannot run the risk of people being penalised by an automated decision. We would also like to ban predictive policing systems because they violate the presumption of innocence.”

Katleen Gabriels, assistant professor specialised in computer ethics at the University of Maastricht in The Netherlands, adds: “We need to push for interdisciplinarity. We need different kinds of people to think about these questions. This can mean diversity in the design stage, among the people making the algorithms. But it’s also needed in the policy area, because many policymakers don’t have a technical background, which sometimes prevents them from realistically assessing these technologies.”

These measures have their limitations, as the firings of Timnit Gebru and Margaret Mitchell from Google’s ethical AI team shows. Yet for Gabriels, education has an important role to play, not only to teach computer programmers about ethics, but also so that ordinary people can understand how algorithms work, and how they impact their lives. “Algorithms aren’t just neutral,” Gabriels says. “Because they are based on mathematics we often assume they are neutral, but they’re not. Better education can help combat these stereotypes.”