Introduction Lecture 2017

Lecture Slides: Critical Algorithm Studies 2017

Topics and Dates:

  • April, 27th: What is an Algorithm?
  • May, 4th: Embedded Values and Biases in Algorithms
  • May, 11th: Culture, Society and Algorithms
  • May, 18th: Methods and approaches for studying algorithmic systems
  • June, 1st: Algorithmic Fairness
  • June, 8th: Ethics and Algorithms
  • June, 22nd: Surveillance, Privacy and Data in the Smart City
  • June, 29th: Accountability, Policy and Law

All the participants should read the core papers (can be found in the owncloud) for each topic, the presenters should read all the papers and find a supplemental paper.


New Lecture in 2017 at TU Wien

In the summer term of 2017 the seminar lecture “Critical Algorithm Studies” will be held again at TU Wien. Subscription to this lecture will be possible starting from the 4th of March via TISS. A new reading list with a new topic selection will be provided in the first session.

The seminar will introduce participants to the emerging field of Critical Algorithm Studies. Blocked weekly discussions of assigned reading material will focus on interdependencies between society, culture and algorithms, and critical reflections of their ethics and politics.

Preliminary list of topics:

  • What constitutes an algorithm? What do users/developers/society understand about algorithms?
  • How and why can algorithmic systems have embedded values and biases?
  • How can we conceptualize algorithmic fairness, think about ethics in algorithmic systems and deal with accountability in complex algorithmic assemblages made of developers, users, management, law, code, computers, and many others?
  • How does culture and society influence the creation of algorithms and vica versa?
  • How can algorithmic management foster erasure of human judgement through increasing rationalization and automation? What are benefits and issues?
  • What methods and approaches are available to study algorithmic systems?
  • What possible futures are currently being imagined?

Many topics in this seminar are very much discussed at the moment. We provide some correspondence as a motivational introduction:

Lecture Submissions

All participants of the seminar were asked to create a final assignment focusing on a topic of interest in the context of “Algorithm Studies”. The following list contains all contributions:

Sixth Lecture – “What users understand about algorithms?” and “Methods to study algorithmic systems”

The last session dealt with two topics:

Discussion (protocol):
We talked about different approaches to understanding algorithms. One possibility is to recode an algorithm to see if one understands the problem. Another would be to look at different creative representations and explanations (e.g. dancing a sort algorithm). Possibilities for methods to study algorithms and the understanding of users of algorithms is related, since one first needs to really know what an algorithmic systems does (sometimes difficult due to complexity or type of method, e.g. machine learning) in order to present it. Many algorithms are also not open source (sometimes even considered trade secrets). Therefore, efforts from companies are necessary so users and policy makers are able to understand what is happening. Currently most companies do not explain their algorithms, persuasion and policy is needed. The open sourcing of code is not enough, since not everyone understands code, companies should also be required to explain very important algorithms (e.g. sorting on facebook).

Next we discussed differences between human and algorithmic work in the context of algorithmic aversion. We discussed benefits of human doctors compared to online alternatives (e.g. WebMD):

  • Accountability is easier to determine with humans, because algorithmic systems are usually complex assemblages (black box) and in turn the accountability is hard to trace.
  • Good care for people requires “human connection”.
  • Algorithms are perceived as objective, in contrast to doctors, but algorithmic decisions are also based on data which could contain problematic bias.

We asked ourself then: “Do doctors actually want to use (algorithmic) decision support systems”? Beside from the reasons mentioned above, also some of the superiority/prestige associated with being a doctor would get lost if algorithms supported or replaced them. A study showed that doctors like to take advice much less from lower ranking people than pilots. Next we to came up with examples of algorithmic aversion in other businesses.

According to a study on the course reading list, many people strongly believe in the benefits of automated plagiarism checks. They are seen as objective and replace work many humans do not really want to do and are therefore accepted. A counter example is AI composed music. The creators could not find good musicians which wanted to play it, because it “had no soul”. Music is more than sound. It is very context depended and intertwined with culture. It is connected to social class, stories, cultural identities, …

Then we talked about an algorithmic system in the Austrian context, ELGA, a system to centralize personal medical data in Austria. There has been a lot of resistance from medical personal and from some data protection activists.

Pro arguments:

  • More transparency  for patients and law makers over the activities of doctors
  • Permanently stored and online accessible list of medications available to patients. May enable people to extract their own data for more evidence based and personalized therapy.
  • Aggregated medical data is useful for basic research
  • It may help with identifying spending issues and thereby potentially save money

Con arguments:

  • Fear of disclosing health information to other doctors (especially if associated with corporations)
  • The collection of data usually leads to secondary uses. In this case insurance companies could be interested in the data.
  • Every computer system is also susceptible to hacks and other online attacks.
  • The online-Interface to control data will only be accessible to some people (Digital Divide).

Another issue is that ELGA is opt-out and most people do not actually know about it. What would people need to know about ELGA to make it an informed consent? The overall process and consequences would be useful, but not too much depth – would probably be too much information. For the interested people it should be possible to dig deeper. Risk assessment should be communicated to people, e.g. What is recorded? Who has access? How can I control it?
In conclusion knowing the effects is more important then the inner-workings of algorithms, but it is very hard to determine effects of things, especially complex algorithms. An interdisciplinary approach would be necessary.

What policies would be necessary for companies to ensure informed consent of users?

  • Algorithms which are used by or have consequences for many people need much oversight, and algorithms that are not used by or have consequences for not many people need little oversight.
  • Oversight by experts over algorithms should be required (consumer protection). Maybe also through audit of a presentation on inner workings. Audits could be done by specialized companies or governmental organisations. A system of “checks and balances” is needed for algorithms.
  • Inform users on consequences and what could be done with personal data
  • End user licence agreements should be available in accessible language

We concluded with a short discussion about possible futures for algorithms in society. Many promises concerned with the replacement of human labour with machines and algorithms are visible in popular discourse. An example are an add by Nurses United against robotic  replacements:
In the discussion we came to the conclusion that in the far future (everything will still take some time) there will probably be still jobs left for humans, but they will be different jobs. The trend indicates a stronger divide (inequality) between very well educated and the rest. Skills like repairing could be lost for good, due to cheap availability of new products, but jobs concerned with personal pleasure like in the tourism sector could become more important. We need to ask questions such as: “What do we want to have automated?”, “What are human jobs?”,  “What are human jobs?” In such futures algorithms should support people. Humans should be responsible and decide in the end (“Humans in the loop”).

Currently a lot of technology that is built has no oversight and centralises power and decision making. Many new technologies are only understood by tech experts and not enough time is spent on people who are not able and have to live with it, due to social and economic pressure (e.g. smart home, smart city, automated care). More discussion by public is needed on this developments in order to ensure democratic and inclusive developments.

Discussions about a possible Singularity seem far fetched at the moment are mostly done by very privileged people not dealing so much with systemic problems such as sexism, racism, and so on. Attention should be focused more on these basic social problems we have now as a society and not deal with problems in a far away future. We should look at the people who profiting the least from a system to know what is wrong with it and how to improve it.

Fifth Lecture – Surveillance, Privacy and Data

This meeting was on: Surveillance, Privacy and Data

We talked about predictive policying and discriminatory algorithmic bias in such systems. How to deal with survailance. In China a big survailance system is already established. Many people are aware that smartphones do a lot of tracking but still continue using them, because they are convenient and deliver a lot of benefits. The usage of alternatives like TOR (for browsing on PCs) may make one seem suspicios and actually attract attention. Different popular sites also discourage the use of tools like TOR.

In Germany and Austria politicians talk about using a ‘Bundes Trojaner’. One does not know if survailance is really happening, or if it just looks like it’s happening. What are the actual benefits for people of doing survailance? Studies show that the impact on crime detection is neglectable. One does not know for what the data will be used in the future, people and governments change. What if totalitarian governments in the past had our current technology? The storage of big amounts of data is also problematic and dangerous. How to secure the data? What about data beeing moved? As a person one does not know where one’s data moves around. There are also possibilities of misinterpretation of data. People can be stopped from traveling because of ‘joke’ tweet (there have been already reported cases). What about a future which extends more and more in our lives?

Survailance exists also outside of governments, e.g. a shopping card tracking your purchases and giving you bonuses in return. In these cases survailance is usually done for capitalistic reaons, like predictions of what a person is buying.

The biggest problem of survailance is that it is usually invisible. One does not know what processes are executed and what the results are. The public only notices incidents or big discussions. Is for instance the data agreement of Google visible and understandable enough for most people? Currently only very few people have actually read it. We asked in our group what tools for “selfprotection” do people use: Ghostery, AdBlock+, NoScript. A lot of privacy enhancing tools are not that convinient, so many people think: “I know I should, but I don’t do it usually”.

Survailance often categorizies people, objects and events. This categorization is also not visible and users are usually labeled without knowing about it. For example automatic pregnancy detection through previous purchases and based on that the creation of recommendation of products. Another example is the monitoring of women and their toilet routines by their companies in order to detect pregnancy early. This predictions can then be used to fire them before they are in motherhood protection (by officially telling the company).

There is a difference in the actions if people know they are beeing watched (Panopticonism), e.g. persecution of homosexuality. For example: A woman posts private photos on Instagram without her hijab and gets in trouble because of that. In private this is ok, but actually the photos are entering the public realm. The distinctions between public and private are not so easy to see in the digital world.

In the neutral net everybody can host everything. Net neutrality is also important for a fair market. Getting rid of neutrality would result in changes in power structures on the web. It has already happened that ISPs not only want to ‘balance’ traffic but also completely make a certain service unsuable (e.g Netflix) unless one subscribes to a higher rate.

In the future more technology will be part of our lives (e.g. Internet of Things) and in turn more survailance is possible. The responsibility of technlogy creators needs to be a greater focus of this discussions. At the same time complexity has risen so much that it’s not really possible anymore to completly understand all deployed systems. Nowadays careing about privacy becomes a competitive advantage (e.g. Apple). Having a public discussion on privacy is of great importance.

Fourth Lecture – “Algorithmic Culture” and “Erasure of human judgement through rationalization and automation”

In this meeting we had one presentation:

and a short introduction to “culture”, a short summary of the readings and an “Engaging Activity” to Algorithms in our daily lives:

Culture is a term that is very difficult to grasp. It could maybe be desribed as aquired cognitive and symoblic aspects of human existence. Culture isn’t bounded, i.e. one can not define a “circle” around it (e.g. “Austrian Culture”: What is it actually? Does it stop at national borders? …). Many algorithms tend to fragment culture/publics through sorting/filtering/… (e.g. geolocalised search results). Culture does not only influence people, people also influence culture. This maybe is analogous to languages and how people speak them.

“The Lives of Bots” describes Wikipedia bots that take part in editing. In this system they are actors similar to human beings. An alternative stance would be to think of them as cultural artifacts, “anything created by humans which gives information about the culture of its creator and users” (  The article “The Cathedral of Computation” was focused on the culture around algorithms. It is centered on technological determinism, i.e. with the belief that technology ultimately advances “society” and is able to solve important social problems. It also stated that people tend to view algorithms as god-given, unchangeable and objective, which is a problem since they are changeable.

Next we took part in an activity with the intention to have a recent, conscious experience related to the topic in order to fuel reflection and dicussion. The presenter of “Algorithmic Culture” chose to  use this method instead of the usual presentation with slides. We were told to search for a recent article in Austria on a specific algorithm influencing our daily lives. We talked about our experiences and the results we got from the different search engines we used. The various plattforms featured different paraemters for the search. Participants using the same plattform and search query got different results due to personalization algorithms. We talked about how ranking by “relevance” is very context dependent and that something like “relevance” can not just be calculated. It is also a very poltical concept, since these rankings are a hierachy of importance which is acknowledged by users. A lot of personalization seems to be language and country dependent which reproduces legal borders between states in the digital realm.

The last hour of our session was again dedicated to discussion. We talked about a paper stating that self-tracking healthy eating apps do not really structurally change the culture of unhealthy eating. The responsiblity is with the companies and marketing departments selling unhealthy products and not really emphasising the risks involved. These apps are predomanitkly used by people with smartphones and more wealth, since one needs time and money to really use these apps. Many unhealthy foods are also cheaper in comparision to healthy alternatives. The article “algorithmic self” raised the issue of people writing posts not only with the intent to influence how other people will see them, but also how the algorithm sees them. For example some news publisher write titles for their articles in ways that algorithms will rank them higher and more people will click them. Later on we talked about Social Media and its influence on people. The used medium governs how people communicate with each other and in turn social norms develop around structures build into the system, e.g. “can you friend me on facebook” is the new “can you give me your phone number”. It is important to make data available for discussion and research, so different groups can inspect the systems to see if they adhere the ethical principles. It would also be important to teach algorithmic literacy in school. Global Trending Topics on Twitter work as a way to unify different Filter Bubbles. Trends are still fragmented (by country or others). One can choose which fragment one desires, but still on twitter not many people do it. Creating filter bubbles protects from the outside and create a closed spaces where fewer things have to be discussed and less criticism is possible. Algorithmic systems are also becoming increasingly important in political discourse with Facebook beeing accused of manipulating it’s streams in the U.S. against Trump and Hillary Clinton having various issues with her emails.

Third Lecture – Ethics and Accountability

This meeting featured two presentations:

After the presentations we had a discussion:
We talked about how accountability is at the core of ethics. Ethics are not really measurable, there is no way to tell which is the best, most ethical, way of doing something. We agreed that the context in which the algorithm acts is important and that technologies may develop in unpredictable ways. Therefore intentions of designers may not be met in the long run or in different contexts. New problems and dynamics emerge. We reflected upon: How do we define the relationships between networks of machines, people and more? How do we get ethics? What about the invisibility of algorithms and how to open up the black box (i.e. understand  complex algorithm)?

We talked about designing vs developing and concluded that design has people in mind, is sometimes general (i.e. not so much about details), thinks about the context, ethics, what “it” looks and feels like and deals with the formulation of problems. It is situated and explorative. No perfect solutions exist, only design decisions with consequences. In contrast development is about solving a given problem, i.e. finding a correct “solution” (positivism). It is sometimes used as a defence to circumvent making moral choices. But during development small design decisions are done all the time (such as the default value of a variable in a program). In turn development also requires a thoughtful design approach.

The next topic of interest was “Algorithmic Regulations”. They should be about defining a set of goals, such as no gender/race/sex/.. bias or not trying to kill people, and ways of measuring these goals. According to one of the papers, accountability is about:  “How to break up the algorithm?” (deconstruction) and “governance of the algorithm” (“How can we establish accountability?” and “How can we enforce accountability?”).

We talked about algorithms being used in court, advertised as more fair than judges, and ones used by military to make life and death decisions. What does the probability of being a terrorist actually mean? What are the factors? Who chooses the factors? How are the factors fed into the system? What about correlation vs. causality/errors? What does a “like” or a visit to website really indicate about a person? “Fairness” is a social construct and thereby has different meanings in different contexts, like e.g. in different countries. Is there really a way to calculate if someone is guilty or should be killed? Is there a right or wrong / better or worse here? Why do we need an algorithms? Is it because they are more efficient? Algorithms are biased due to the development in a specific context and the political viewpoints of their builders. In this case very important decision processes are suddenly influenced and maybe even solely understood by engineers. The algorithms base their calculations on data collected through algorithms again developed by engineers. It is wrong to assume algorithms would come to “fairer” decisions, but it is still done since they are preceived as “objective”. Algorithmic judgments are executed by people but who is actually making the decisions? Who is responsible? What did the algorithm decide? How did it reach a conclusion? Was it under the control of humans all the time? Modern, complex algorithms (specially many machine learning algorithms) can only be understood to a certain point and complexity is ever increasing. Transparency is usually not a sufficient solution and limited by complexity. It is better have people/developers/users/… do the right things (i.e. be ethical). People should ask themselves constantly: “What is it the right thing to do?”.  Companies/Organisations should have an Ethos. The value of ethics in society needs increase.

Last we talked about different methods of self-protection against algorithmic governance. We came to the conclusion that this option is limited for a number of reasons such as resources of organisations developing countermeasures are usually small compared to organisation developing technology for profit and one needs to be knowledgeable about technology to use the state of the art tooling.