AI systems should be accountable, explainable, and unbiased, says EU

Image: European Commission

The European Union today published a set of guidelines on how companies and governments should develop ethical applications of artificial intelligence.

These rules aren’t like Isaac Asimov’s “Three Laws of Robotics.” They don’t offer a snappy, moral framework that will help us control murderous robots. Instead, they address the murky and diffuse problems that will affect society as we integrate AI into sectors like health care, education, and consumer technology.

So, for example, if an AI system diagnoses you with cancer sometime in the future, the EU’s guidelines would want to make sure that a number of things take place: that the software wasn’t biased by your race or gender, that it didn’t override the objections of a human doctor, and that it gave the patient the option to have their diagnosis explained to them.

So, yes, these guidelines are about stopping AI from running amuck, but on the level of admin and bureaucracy, not Asimov-style murder mysteries.

To help with this goal, the EU convened a group of 52 experts who came up with seven requirements they think future AI systems should meet. They are as follows:

You’ll notice that some of these requirements are pretty abstract and would be hard to assess in an objective sense. (Definitions of “positive social change,” for example, vary hugely from person to person and country to country.) But others are more straightforward and could be tested via government oversight. Sharing the data used to train government AI systems, for example, could be a good way to fight against biased algorithms.

These guidelines aren’t legally binding, but they could shape any future legislation drafted by the European Union. The EU has repeatedly said it wants to be a leader in ethical AI, and it has shown with GDPR that it’s willing to create far-reaching laws that protect digital rights.

But this role has been partly forced on the EU by circumstance. It can’t compete with America and China — the world’s leaders in AI — when it comes to investment and cutting-edge research, so it’s chosen ethics as its best bet to shape the technology’s future.

As part of that effort, today’s report includes what’s being called a “Trustworthy AI assessment list” — a list of questions that can help experts figure out any potential weak spots or dangers in AI software. This list includes questions like “Did you verify how your system behaves in unexpected situations and environments?” and “Did you assess the type and scope of data in your data set?”

These assessment lists are just preliminary, but the EU will be gathering feedback from companies in the coming years, with a final report on their utility due in 2020.

Fanny Hidvégi, a policy manager at digital rights group Access Now and an expert who helped write today’s guidelines, said the assessment list was the most important part of the report. “It provides a practical, forward-looking perspective” on how to mitigate potential harms of AI, Hidvégi told The Verge.

“In our view the EU has the potential and responsibility to be in the forefront of this work,” said Hidvégi. “But we do think that the European Union should not stop at ethics guidelines ... It can only come on top of legal compliance.”

Others are doubtful that the EU’s attempt to shape how global AI is developed through ethics research will have much of an effect.

“We are skeptical of the approach being taken, the idea that by creating a golden standard for ethical AI it will confirm the EU’s place in global AI development,” Eline Chivot, a senior policy analyst at the Center for Data Innovation think tank, told The Verge. “To be a leader in ethical AI you first have to lead in AI itself.”


" AI systems should be accountable, explainable, and unbiased, says EU"

Unlike the EU system then.

Actually the EU is accountable, explainable, and unbiased (of course depending on your definition of unbiased). Probably much more than your local political system..

I take it you understand the inner workings of the human mind then? This is a double standard… no one can truly explain why people do what they do, including EU members. No one can read minds effectively. We shouldn’t expect neural networks to be easily read. If they enforce this they will be left in the global AI dust.

Bullshit. This is an artificial ‘mind’ that is created and it can easily be documented.

Right… you can document the development of a child 24/7… but that’s not the same as being able to explain his day to day actions. If tech companies in the EU have to be able to explain every steering wheel input to an autonomous car, the’ll have to ditch neural networks and go with rudimentary statistical regressions or procedural programming.

You can probably take a guess about why the child punched the teacher, but it would be hard to be 100% certain. Luckily with computers with can iterate and retrain models… essentially regrowing the child in confinement to create different versions of him and see which ones would punch the teacher.

As an aside, revealing how you train your models could have serious IP and competitiveness issues.

If that’s what you think, then you have NO idea how most AI works. Neural networks are inherently non-explainable in all but trivial cases, since its internal weights are adjusted mathematically (or randomly with GAs) until it "works" to the desired accuracy level. The finished internal mappings normally have meanings/relationships which humans don’t consciously think in terms of (so how would one even start to "explain" them). Even complex systems with hand written rules can have emergent behavior, which can’t be easily predicted even by a person that exclusively wrote all its rules.

With most medicines, you can only do testing and statistically evaluate if they are safe/reliable, because we can’t [currently feasibly] model all the possible microscopic interactions with everything in the body and predict all potential results. Dealing with AI is very similar. It needs to be closely monitored (and discontinued/recalled if an unacceptable problem is found).

So basically same as the politicians. How’s that working out for us so far?

I vote for politicians be replaced by AI ASAP. And I don’t want AI’s decisions be overridable by human beings.

The EU seems to have a fundamental misunderstanding of how the machine learning systems work. Everyone would like to know how a result came to be from a machine learning algorithm, but the whole point is that a human didn’t come up with the algorithm, so only the machine understands how it got there.

If this is the EU’s idea of how to gain leadership in AI, prepare for the EU to be far, far behind the advancements of other nations in that area.

Same problem as with all credit decisions systems. While you can build a deep neural network and come up with a much better model than the typical regression models of today, since you can’t explain the decision, you can’t use it, unless you like discrimination lawsuits.

So maybe the regulations will eventually restrict neural networks/deep learning etc. for specific verticals like credit, housing advertising etc.

Actually, that is already the case for applications like credit risk assessment. The regulations require transparent models. So data scientists use simpler linear regression models, but choose features based on some inspiration from neural networks.

I suppose it all depends on how strict they are about what it is to explain. Because whether it’s a machine learning algorithm, or an employee, all I could really do is give an educated guess at their reasoning.

I see the conversation basically boiling down to:
"Our current system can’t tell us why it made a decision"
"So figure out a way to make the system know what it’s doing"
"But that’s too hard"
"Then don’t use a system that isn’t trustworthy"

I don’t see this as a problem. ML is inherently extremely powerful, but like all tools we need to understand its limitations. If we throw up our hands and say "it’s too complex, we can’t understand it" then we’re doing ourselves a disservice.

Look at all the biased algorithms that have been developed and used (like Amazon’s hiring AI). The algorithm wasn’t consciously discriminating, but the data used to train it was the result of existing bias, and so the algorithm included that bias in it’s function. The algorithm wasn’t trustworthy.

Also, remember that not all AI systems are exclusively based on Machine Learning. ML is just one method of creating an AI system.

Complexity of current ML models is quickly approaching or exceeding our understanding, and with future growth of processing power, a human operator will be delegated to an oversight functionality, not design. This is already a talked-about future trend at universities, and I’ve noticed this attitude with a lot of interns that I mentor.

Unbiased data is a misnomer in my opinion, as it can still be used to build discriminatory models, and its inherent lack of bias is not absolute and can be questioned. At the end of the day, any AI as envisioned by many in the near to mid-term future CANNOT be taught to be universally ethical or unbiased – it will have to develop those concepts on its own, hopefully with proper guidance.

Indeed and the thing is expert humans themselves often are unable to provide explanations about how they arrived at a conclusion (just as we immediately recognize faces without reasoning; we are all experts at face recognition). I would ask for data and method transparency instead of explainability.

that it didn’t override the objections of a human doctor

In time, AI will be better at making a diagnosis than humans, therefore why should they not override a human doctor (at that point)?.

While not AI in the actual definition of AI, in imaging for certain diseases, machine learning is becoming quite good at diagnosis, in some cases better than radiologists (where there is often disagreement between radiologists, as reading imaging results is part art-form).

I imagine there will be a hesitancy until there’s a way to make an AI accountable for its actions.

This is kind of the same argument behind fully autonomous vehicles and liability for choices it makes etc. Ie who is accountable?

The only way to make AI accountable, is to make whoever created it accountable for its actions.

Eventually those people will be dead, and at least with driving, the drivers can override the AI.

Then it’s useless. Until the AI can be trusted, it’s not actually making life any easier. Nobody gives a shit about a car that can """drive itself""" if you still need to keep your eyes on the road and your hands on the wheel and pay full attention to the actions of the car. You’re still driving it at that point.

Similarly, nobody cares about an AI that can make a diagnosis if you still need to send a doctor to college for a decade, pay them a quarter million a year, and they still have to arrive at their own independent diagnosis which happens to be the only diagnosis that actually matters.

Until AI can be trusted to work in an actually autonomous manner, it’s nothing more than an interesting piece of side-tech. At best, it’ll give a doctor an interesting idea, assuming they’re willing to admit that they weren’t very important to the process. (Hint: they won’t be)

It’s not useless, AI just shouldn’t be in charge until there’s a way to make it accountable. Take your second example, there’s value in AI making a diagnosis for a doctor to either approve or reject, because it could take work off of the doctor’s plate, and make them more efficient, and also possibly lead to a quicker diagnosis.

And who is accountable now? Boeing software downed 2 jets with more than 300 people onboard. Who is accountable? Do you really believe anybody would go to prison for murder or, at least, involuntary homicide? When cops make mistakes that cost people lives, are they hold accountable? Never, if that is indeed an honest mistake. AND mistakes DOES HAPPEN, in any human endeavor, in designing AI just as well. And even perfect AI would make mistakes sometimes, just as a live driver or doctor does. (Likely much less mistakes, BTW.)

So if people notice that robodoctors from Universal MedBots are making to manu mistakes, they will go to robodoctors from General Robotics. Or Huawei Medical. That’s all accountability you need.

There are two possibilities:

1. AI is being trusted to make decisions, and takes work off the doctor’s plate. The doctor looks at a diagnosis, says "well that’s not totally insane" and moves on without a second thought. At this point, AI is now doing the work, and if it’s trusted to that level, there’s no need for the doctor.

2. The AI isn’t being trusted to make decisions. The doctor still has to review the work of the AI, and simultaneously needs to come up with their own diagnosis, then figure out why they’re different if they are. At that point, they need to make a decision on whether to go with the AI’s guess or their own. At this point, the doctor is now doing more work, and the AI has simply become an expensive obstacle.

The latter sounds much more likely to me. Either we trust the machine to do the job it’s programmed to do and it’s helpful (after testing, of course), or we just throw the idea out.

Accountability is a strange one as it is clear that the accountability lands on whoever created the model just like any other tool we use. An important reminder, a model is a tool just like any other. Be careful humanizing these things or given them more agency than they deserve. They are a dumb tool that typically does one thing well and has no context outside of the dataset it was trained on. If a hammer falls of a shelf and lands on someone’s toe we don’t ask the hammer to be accountable for that. If it is so broken that the head flies off in normal usage… then we might. I think that is how we should think about this.

Explainable is hard but doable in some cases. Some models, like random forest, can be inspected to understand how it came to a decision but neural net models tend to be a bit of a black box. You can get some details out by running the model more than once and varying the input but this can be complex game. That said, I do think there are many processes and tools that we use day to day that are very challenging to explain as well. I don’t think this problem lives solely in the realm of ML.

Unbiased is interesting. A model only returns results based on the data it is given. So I’d argue that you need to make sure the data the model is trained on is unbiased. This is tricky though as the world tends to be a bit bias for all sorts of socioeconomic reasons. It is hard to be critical of a model for thinking it is statically more likely for a minority to be involved with a crime when that is in fact the ground truth in many places. If a model could understand that actually the driving factor here isn’t race but a huge number of economic, social and historical influences that would be ideal. That would also be an exceptionally hard model to train. I don’t really know the answer to this one. I think we have to strive for models that give the best results. A model that is bias will not produce good results due to that bias. I think people training these models strive to eliminate bias. I don’t know how you could enforce this though. That would be challenging…

View All Comments
Back to top ↑