Why AI & Ethics?

The following texts were created for the AI Campus as short lectures in the module on ethics and approach the topic of AI and ethics – or actually the topic of software and ethics – from a technical perspective. Ethical considerations are motivated, but deliberately not debated; because that is what ethicists have to do. Many of the ideas presented here have motivated the bidt projects on ethics in Agile Software Engineering and explanations in complex socio-technical systems.

---

Part I: What is AI, what is machine learning?

What does “problem” mean in computer science?

Artificial intelligence, quite a misnomer, refers to a set of procedures for solving problems. I use the term “problems” to refer to technical problems that computer scientists have and whose solution hopefully helps to solve problems in the analogue world. Technical problems we have in computer science are comparatively simple when compared to the problems we have in philosophy, political science or sociology. This is possibly because our problems have a solution.

For computer scientists, one problem is the assignment of outputs to inputs. The input “distance to the car ahead” maps the software in an adaptive cruise control system to the output “braking” or “accelerating”. In an air conditioning system, the software maps the inputs “actual temperature and set temperature” to the output “degree of opening of the valve for cooling liquid”. A pacemaker maps the input “current heartbeat and electrical resistance of the lung tissue” to voltage pulses. Google search maps the input “ethics and AI” to a list of websites. The Corona app maps Bluetooth contacts to random numbers. And so on. These are mathematical functions we know from school: f(x)= ^x2 calculates the square of the input x as the output.

Traditional programmes describe through a set of steps how these functions are calculated. These steps are very small, clearly defined in their order, and they must be absolutely precise. You have probably heard the word “algorithm” before. It is also such a sequence of steps. When an algorithm is specified and written down in such a way that it can be understood by a computer, we call it a “programme”. Several programmes together form software. We have presented this in detail in this article. For now, let’s keep in mind that a function and its computation, here by a program, are two different things.

What types of AI are there?

Artificial intelligence exists roughly in two forms. First, there is so-called symbolic AI. It works a bit like the programmes I just described. You write down explicitly and in detail everything you know and everything that is relevant. What are the individual steps in the programme are rules and properties of objects here: If it rains, it will get wet with a certain probability. If I pull an object on a string, it moves along the direction of pull. If the symptoms fever, cough and loss of taste are observed, there is possibly a corona infection. Based on the rules and available facts, one can then draw conclusions. What does not have to be determined by programmers in this case is the order in which the rules are applied: In a sense, the symbolic AI “guesses” these.

This sometimes works, but in the aggregate not always very well. If one states that pulling a string moves an object, then there will be situations in which one should also state that a string can be used to pull but not to push. And what else? That the string must not have been cut before? That it must be sufficiently robust? That the object must not be too heavy? Or, quite similarly: If you want to describe how a camera in a car recognises a pedestrian, then you have to talk about tall, short, fat and thin people; about people with umbrellas and shopping bags; about people who push bicycles and sit in wheelchairs; about people who wear chicken costumes at carnival. A person recognises all this immediately without thinking. But explicitly writing down the appropriate rules that could then be used in a pedestrian recognition programme is very difficult or impossible because there are so many situations to consider.

Machine learning as a solution to fuzzy problems

Let’s look at the problem of pedestrian detection. We are interested in a function f that gives the output “attention pedestrian” for input images with pedestrians and the output “no pedestrian” for images without pedestrians. Remember that a function and its calculation are two different things. Formulating the computation of the function f as a program, or as an explicit rule system, is difficult given the fuzzy problem, as we have seen.

That is why for certain classes of problems there is a variant of problem solving, machine learning. Instead of explicitly writing down the relationships, i.e. the rules or the program steps that the function f calculates to detect pedestrians, one lets it learn the calculation of this function by means of many examples. Because learning is done using a set of examples, we call them the training dates. Each training data is a pair consisting of an image and its description, i.e. “pedestrian” or “no pedestrian”. Machine learning then automatically creates a structure that represents the mapping of input image to output “pedestrian” or “no pedestrian” for the training data. The trick is that this structure does not only work for the training data. For these images we already know whether they contain a pedestrian – that would be boring. This structure now also makes it possible to calculate for new, unknown pictures whether they contain a pedestrian or not!

It works like this: When an unknown image is to be analysed, the images that are most similar to this unknown image are identified in the learned structure from the training data. For these most similar training images, one knows whether there is a pedestrian in them or not. If the majority of the training images most similar to the new image contain a pedestrian, it is assumed that the new image also contains a pedestrian. And vice versa. It is important that the learned structure itself does not describe any explicit correlations, but comes to a result solely through the analysis of similarities.

This works not only for functions whose output is one of two possibilities, such as “pedestrian” or “no pedestrian”. It also works for outputs that are continuous values, such as the degree of opening of a valve for refrigerant. There, for a new, unknown, input “actual and setpoint temperature”, the most similar values in the input part of the training data are searched again, and the average of the corresponding outputs of the training data is used instead of a majority, as we have seen before. The output can also consist of several such values, such as “pedestrian or no pedestrian” and also “distance to pedestrian”. The output can also be a whole picture or a movie – you may have seen Deep Fakes before. And it can also be a programme in the classical sense, i.e. a set of small steps that are then executed. That’s really crazy!

Let’s keep in mind: functions map inputs to outputs. The calculation of a function is distinct from the function. The calculation of the output given an input can be done by classical programs that consist of very precise small-particle steps. A similar type of computation is also performed in the symbolic variant of AI; here the individual steps are applications of very precise explicit rules for inference. The order of application, however, does not have to be specified by the human, but is, in a sense, guessed by the symbolic AI. In data-driven AI, machine learning, no explicit steps or rules are given. Instead, structures are learned from examples that later enable outputs to be determined even for previously unknown inputs, which were therefore not part of the training data. Incidentally, whether this structure has learned exactly the connections between concepts that we humans would consider relevant is usually not the case. We’ll look at that in the next part.

Part II: “Learning” from data

“Understanding” in machine learning

We humans also often learn by grasping contexts through examples. But grasping contexts is not the same as understanding contexts and being able to explain contexts! We have all had the experience: if a non-native speaker asks us whether the grammar of a certain sentence is correct, we can immediately say yes or no: we know because the sentence “sounds right”, because we have previously heard enough examples that are similar or not. But if we are then asked why the sentence is right or wrong, we cannot answer that. This is because we usually do not know the rules of grammar of our mother tongue.

Closely related to this is the following consideration, which I think is important. It is not even necessarily the case that the machine-learned structure represents the same contexts that we as humans would consider relevant. We humans might recognise the image of a pedestrian by the fact that it is an elongated being with features such as a head, torso, arms, legs, etc., that moves in a certain way and that does or does not do certain things in an expectable way. In our head, when we assign the mental to the objects of the real world, something happens that philosophy calls intentionality. The machine-learned structure that we introduced in the first part knows no such intentionality. It can even be assumed that it represents something completely different! This can be seen in practice by the fact that it will normally be possible to trick the machine by changing only one pixel in the image of a pedestrian: A correct mapping for the original image will lead to an incorrect mapping for the minimally modified image.

The natural sciences infer causal relationships from phenomena or data. In the course of time, mankind has developed incredibly powerful tools for this purpose, such as experimentation, modelling, hypothesis and its falsification. In machine learning, there is generally no explicit modelling of this kind, precisely because it often proves to be too complicated. Supposed correlations are derived from data alone. However, the AI does not recognise causalities, but statistical patterns in the data. That alone, without modelling, would be considered unscientific in many situations, because this procedure can lead to completely erroneous conclusions from today’s perspective. I am thinking, for example, of Paracelsus’ idea that you only have to keep sperm in warm horse manure for forty days to produce little humans. Or the idea of Samuel Hahnemann’s homeopathy comes to mind, that one could achieve effects with undetectable amounts of substances because certain poisons produce similar symptoms to certain diseases. The danger of such remarkable conclusions also exists in machine learning. However, and it is important to note this from a pragmatic perspective, machine learning is often simply incredibly successful in practice, and especially more successful than approaches involving explicit modelling to represent the interrelationships of effects. Hippocrates put it this way: He who heals is right.

When is machine learning useful?

We should certainly always keep this in mind. After this principled scientific-theoretical consideration, however, we also need to talk about the practice of using data. We have seen that machine learning is always appropriate when one does not or cannot know the exact relationships underlying a function to be calculated. Predictions about what customers will buy soon are difficult to describe as explicit rules. Predictions about which website is particularly relevant for a search query are also difficult to formulate explicitly. And the same is true for predicting when a mechanical part is likely to be defective; for weather forecasting; for diagnosing diseases, etc. It is important to remember that such predictions are precisely examples of the functions we introduced in the last part.

The limits of machine learning

The prerequisite for machine learning is that the training data is available in large numbers and that for each training datum not only the respective input but also the “correct” output is available. Sometimes you are in the fortunate situation that this is the case. Amazon, for example, knows what you have bought and what people similar to you have bought. Google, after all, knows which links have been clicked on by certain people for certain queries and also knows very well which people are similar to each other and will therefore possibly find the same web pages interesting.

In other situations, this is not so clear: in the case of medical diagnoses, for example, one does not always know what all the relevant factors are that have led to a diagnosis by a doctor – and the diagnosis can also be wrong. The underlying data itself can also be wrong. In the case of weather forecasts, it is also not entirely clear what all the factors determining the weather are, and how precisely and completely they have been measured is also not clear from the outset. And when predictions are made about whether your windscreen wiper motor is about to fail, the underlying data is also often not as clean, not as accurate and complete as it might be for search queries and websites that are perceived to be relevant.

We need to look more closely at two aspects here. One aspect is what data is used at what granularity to make a prediction. What are the factors that determine the creditworthiness of a customer? What are the factors that help predict academic success? What are the factors that help make the right diagnosis? This is not a technical problem, but a methodological one. For a given set of such data, machine learning can even help identify irrelevant data – but it cannot determine whether relevant data are missing! In any case, once you have decided what the supposedly relevant data is, the second aspect comes into play, the quality of the data. When data is measured and stored, errors happen for very different reasons, and the data is then incomplete or simply wrong. This happens in both the analogue and the digital world.

So training data can be incomplete and wrong. We will look at this in the next part.

Part III: Qualities of data

The problem area of training data

We concluded the last part with the observation that training data can be incomplete and wrong. Training data can be wrong in two ways: The input data can be wrong, and so can the outputs. Think of wrong lab values and wrong diagnoses.

Incomplete can be the training data in four manifestations: First, if they do not capture the relevant factors, they are simply inadequate. If we take pictures of pedestrians only in fine weather, we can really only hope that later pedestrians will also be recognised in rainy weather. If it is not understood that Helicobacter pylori is causal for peptic ulcers and the input data does not include information about the prevalence of the bacterium, we will obviously never be able to identify the link between Helicobacter and ulcers and may instead identify stress as the cause. So this relates to the selection of relevant factors as part of the input data.

Secondly, we have seen that to learn as part of the examples, we also need the relevant outputs in the training data. Amazon and Google know the purchase decisions or selected search results of their customers; these are the outputs. But often these outputs are not there at all! If we want to use machine learning to recognise pedestrians, we need many pictures where there are pedestrians and pictures where there are none, and in each case the information whether this is the case. But where does this information come from? This is often a structural problem: If a machine could create this information from given data, then we would no longer need machine learning, because obviously there is already a machine that has solved the problem. But if a human has to provide this information, then this becomes very time-consuming with large amounts of data – and humans can also make mistakes again. In the case of Amazon and Google, by the way, the case is somewhat different: Here, the outputs in the training data are created by observing the interaction of humans and machines, not by calculation.

Thirdly, too little data can be available, which then leads to drawing the wrong conclusions from too few examples. We humans also have this problem. We then call it prejudice. It is important to realise that the availability of sufficient amounts of data depends on the context: Amazon and Google find it comparatively easy to collect large amounts of complete, good quality data on purchasing and search behaviour. This is not so easy for measurements in the analogue world, and it is not so easy for the diagnosis of rare diseases.

Fourth, another difficulty arises from the problem of making training data representative of real-world contexts: If one applies a pedestrian detection system only in situations with nice weather, it may be acceptable to use only or almost only training data with nice weather. However, we would naturally want the system to work anyway in the exceptional case of bad weather; especially because we may have no idea that weather conditions have an impact on detection accuracy. In this case, however, it is likely that the quality of the detection system does depend on the weather. Recall that a machine-learned structure determines outputs by averaging, for an unknown input, the outputs of those training data whose input data are most similar to the unknown input. Now, if there are almost only images of fine weather, the most similar training data are probably not very similar to a rainy weather image. This can lead to wrong results. A famous example of this phenomenon is early versions of face recognition systems that recognised whites with very good accuracy, but blacks with poor accuracy because they were trained primarily with images of whites. When the results of automatic facial recognition are automatically assumed to be correct by the police, this can lead to consequences such as unwarranted arrests. This happened just last year in Michigan. Ethically, this is also problematic because we humans think that an AI will come to more “neutral”, “better” or “rational” conclusions because there is no human at work. But an AI cannot be better than the data it has been trained with.

Can an AI be discriminatory?

Statistical distributions of input data can therefore, for mathematical reasons by the way, lead in the practice of machine learning to the fact that rare input data are more likely to lead to incorrect outputs. It’s no different with humans: we tend to make mistakes in exceptional situations. Computer scientists are working on getting a grip on this problem. To conclude this part, however, I would like to briefly discuss another aspect of the distribution of input and output data. Since machine learning ultimately captures statistical relationships, it will also identify statistical relationships that we may find problematic. If members of a certain nationality and age are statistically identified as particularly likely to cause accidents; or if training for a certain age group is statistically identified as ineffective; or if a certain gender statistically leads to lower performance, then we may perceive that as discriminatory. For historical and cultural reasons, this plays a much greater role in the USA than it does here, but it is no less important here and is also prohibited in the Basic Law for certain characteristics. One can then simply omit these characteristics from the training data. This does not always solve the problem, however, if these primary characteristics in turn correlate with secondary characteristics and discrimination with regard to the secondary characteristics then automatically results in discrimination with regard to the primary characteristics.

Interim conclusion

Let us summarise: Firstly, machine learning involves learning correlations that we do not explicitly know. If we knew them, we would not need machine learning! Secondly, these connections do not necessarily represent real chains of effects: they are of a statistical nature, not ontological. Thirdly, the structures that are the result of the learning process do not contain any explicit rules that relate concepts to each other in a way that is obvious to humans, but ultimately function via statistical analysis of similarities. Fourth, for the reasons mentioned above, we do not know whether the data are suitable for representing real-world relationships; whether they work for all new input data; whether they work equally well for all subsets of input data; and whether they harbour undesirable discrimination potential. Fifth, we should not forget in our critical analysis, machine learning often works quite extraordinarily well in practice!

Part IV: Ethical considerations: Observations

Rules vs. data

We have already established in the first part that machine learning is always a good option when contexts cannot be grasped precisely or are not understood – because it is then very difficult to formulate the individual steps of a programme or to grasp the relevant rules cleanly and completely. But now we find that we have solved this problem by moving it! Instead of explicit rules, we need “good” data in many ways. It is the responsibility of the developers of AI-based systems to ensure that the data is sufficiently good to derive the relevant relationships from it. But determining whether this is really the case is usually impossible.

Real-world consequences of machine learning

I find all this very interesting! It also becomes relevant when calculations made by machines have consequences in the analogue world. Systems for recognising pedestrians are not an end in themselves, but are supposed to influence the behaviour of automated cars, such as braking or accelerating. Which results are presented to me for a search has an influence on which image of the world I gain. What kind of items are suggested to me for purchase has an influence on my buying behaviour. The colour of my skin can – and at least has in the past – lead to a miscalculation resulting in my arrest.

So calculations by machines have an effect in the real world. Otherwise they would also be irrelevant and would not need to be done at all. For my part, I think it is completely irrelevant whether these calculations were made with AI or traditional programmes – because both approaches have advantages and disadvantages, some of which we have already understood. “Ethics and AI” is not the central question, but ethics and software!

The problem of the term “decision

When effects occur in the analogue world and this is done on the basis of machine-generated data, it is often referred to as a “decision”. If one understands this not only as a linguistic analogy, but actually assumes that machines “decide” something, I think this is problematic. For me, decisions can only be made by humans, namely when they choose between options for action. I know that philosophy and neuroscience argue about whether free will exists: For me, there is! However, machines do “decide” in a certain way – “speeding up” or “not speeding up” can be understood as a decision. However, this machine “decision” is completely deterministic because of the underlying programme or the underlying learned structure and will always come to the same result under identical contextual factors. This is not the case for humans. In this sense, machines do not “decide” anything, at least in my world view, although I would like to repeat that for my part I have no problem at all with the word “decide” being used as an analogy. Machines are therefore not responsible or liable. It gets interesting when people use information calculated by machines as the basis for their decisions, and we’ll look at that in the next part.

Ethical consequences – the fallibility of the machine

We saw in the last part that the quality of the structures learned by machines to calculate a function depends directly on how good, representative, complete and correct the training data is. We have seen that undesirable discrimination can occur. We have also indicated that it is very difficult to be precise about these terms, especially completeness. In the last part, we will briefly look at what ethical consequences can arise if the outputs calculated by a machine-learned structure are wrong: When no pedestrian is seen where one is; when no vehicle ahead is seen where one is; when the wrong person is assigned to a face; or when a medical misdiagnosis is made.

There are many examples of such cases: in 2016, a Tesla in Florida crashed into an unrecognised crossing semi-truck, killing the passenger or driver. in 2018, a pedestrian in Arizona was not recognised by an autonomous Uber car and fatally injured. in 2020, a Tesla in Taiwan crashed into an overturned truck because it failed to detect the object; no one was seriously injured here. Also in 2020, Robert Williams of Michigan was arrested because facial recognition software misidentified him as a suspect.

These are extreme examples that we will hear about with increasing frequency. As tragic as these events are: In the public perception, for my taste, they lead to overreactions and also to fear-mongering. We have a tendency to demand much more from machines than from humans: Digital signatures must “be able” to do much more than human signatures. It is not only drunk drivers who endanger themselves and others on a massive scale around the world every day, which Elon Musk has used as an argument for the ultimately higher safety of automated vehicles. Police officers make mistakes in recognising people. There are estimates that in the US, 5% of all outpatient diagnoses are misdiagnoses – that affects 12 million people – and that 40,000 to 80,000 patients die each year because of misdiagnoses. This is not to excuse anything, please do not misunderstand me. But disappointment is a function of expectation, and that is why we have to formulate expectations realistically. I would like to reiterate that, from my perspective, it is initially completely irrelevant whether the misbehaviour results from machine-learned structures or from programmes written by humans. The problem is not that AI makes “wrong decisions”. The problem is rather that software systems can, for very different reasons, provide wrong output, which either a machine or a human then continues to use, which can then lead to problems.

There is a difference between my examples: In the case of facial recognition, humans made mistakes because they relied on the data from the facial recognition software. In the case of Uber and Tesla, the machine misbehaved with disastrous consequences. In the three cases with semi-autonomous cars, however, a safety driver or the driver of the car was on board, who should and could have braked in time, at least from a legal perspective. Today, aircraft can also take off and land autonomously; however, a pilot can always intervene. Whether humans, in case of doubt and statistically more often than machines, take the more adequate actions is not at all clear to me. On the one hand, we rightly prefer to rely on our ABS than on our own braking skills. On the other hand, examples such as that of the spectacular 2009 birdstrike landing on the Hudson River by a heroic pilot rightly make us wonder whether software would have solved the problem with similar success.

Part V: Ethical considerations: Challenges

The principle of correctness

There is no doubt that we want to prevent misbehaviour of technical and socio-technical systems as much as possible. In purely technical systems, we have various ways of checking hardware and software and their interaction for correctness. Correctness is a relative concept that relates actual behaviour to target behaviour. There is indeed a difference between traditional software systems and those based on AI: In the traditional world, at least in the automotive, avionics, railway, medical and pharmaceutical sectors, there is a good understanding of what all goes wrong and how to avoid it – and there are design, analysis and quality assurance techniques, standards and certifications, the use of which ultimately leads to the fact that the corresponding technical systems really do work quite excellently in the aggregate. An important step in this process is always the specification that the target behaviour must be described precisely, and that there are detailed steps in the development process that serve to compare the actual and target behaviour.

With machine-learned systems, however, we cannot define the target behaviour precisely. This is exactly why we use machine learning and not traditional programmes! This makes it necessary to develop new testing methods that not only analyse the learned structure, but in particular also the training data. Computer scientists around the world are working hard on appropriate quality criteria and testing techniques for data and systems.

In these purely technical systems, a software system directly influences the analogue world. One example is braking to detect pedestrians. Other examples are fully automated decisions about granting credit, about advancing in an application process, or about suggestions for further training for the unemployed. All this is already done today with machine-learning systems.

Socio-technical systems

In socio-technical systems, another aspect comes into play. There, a human being makes a decision on the basis of data provided by a machine: A machine-generated medical diagnosis is reviewed by a doctor; a machine-generated judicial verdict by a judge. It is speculation, but I think it likely that a software system will normally make more objective and probably more appropriate suggestions than a human could. However, especially in special cases, this may not be the case. That is why we have already discussed the problem of the representativeness of training data. Then the question arises whether a human being, in this case a doctor, a judge, has the self-confidence to disregard the suggestion of the machine when he/she knows that the machine is perhaps right in 90% of the cases and a human being statistically only in 80% of the cases.

How can people in socio-technical systems be empowered to override the suggestions of machines? One possibility is that not only the final suggestion is presented to the human, but also a justification or an explanation or plausibilisation of how the machine arrived at the suggestion. Of course, this explanation must then be able to be understood by humans. This closes the loop to an observation made at the beginning: when machine learning is used, there is usually no explicit representation of the rules applied or of connections between human-comprehensible concepts, precisely because of the absence of intentionality! Then, by definition, it becomes difficult to provide human-understandable explanations that, for example, a doctor or a judge could use to examine the proposal. Computer scientists are working hard on this under the keyword “explainable AI” – which, by the way, is easier for the aforementioned symbolic procedures of AI than for machine learning because of the explicit representation of human-comprehensible contexts.

The question of responsibility: human vs. machine

Finally, I would like to briefly address the question of responsibility. If a doctor receives diagnostic and therapeutic suggestions from a machine, checks them, changes them or does not change them and then presents them to a patient, one can argue that the responsibility for a wrong diagnosis lies with the doctor. But if the machine is almost always right, I think it is at least understandable if the doctor relies on the machine rather than his or her own judgement. For me, the responsibility clearly lies with the doctor, because I find the idea of assigning responsibility to a machine absurd. However, from a legal and moral and liability perspective, as a layperson I think it is appropriate to include the influence of technology here: Humans may be responsible, but at least they are not solely “to blame” for a wrong decision.

Now, of course, we have to be careful not to diffuse responsibility. If the doctor is not fully responsible and the machine cannot be, who is? The developers of the machine? The data collectors? The certifiers of the machine? The operators of the machine? The users of the machine? For me, all of them together, and then we have exactly diffusion of responsibility. From a legal perspective, i.e. from a liability perspective, I find it very positive that intensive deliberations are currently taking place here on strict liability, which of course must be weighed up against considerations of innovative capacity

Obviously, there are many interesting and highly relevant considerations about ethics not only for AI, but for software in general. We are taking up one aspect of responsibility, that of the responsibility of developers, in the interdisciplinary bidt project on ethics in agile software development. We are working on questions concerning the further development of AI models and their explainability in the project on human-machine partnerships.

The guest articles published by bidt reflect the views of the authors; they do not reflect the position of the institute as a whole.

---