Augmented Reality: Ethics, Perception, Metaphysics

 

A Royal Society of Edinburgh Research Network Project

November 2021 – November 2023

Investigators: Professor Fiona Macpherson, Professor Ben Colburn, Dr Derek Brown, and Dr Neil McDonnell

Research Assistant: Laura Fearnley

Funder: The Royal Society of Edinburgh

Description of the Research

Augmented Reality (AR) technology allows us to see the world around us and experience virtual objects overlaid, or inserted into, our field of view. For example, you might experience virtual directions on the pavement guiding you to your destination, or a recreation of how your location in a city looked in the past or might look in the future. AR contrasts with Virtual Reality (VR) technology, which stops us from seeing the world, and immerses us wholly in the virtual. AR technology is in its infancy but is beginning to be widely available. Today, you can see what a new sofa would look like in your sitting room using your phone and you can experience virtual art works in galleries. Future AR technology will likely involve people wearing special glasses, contact lenses or implants. We can identify possible uses of the technology based on its present and likely future capabilities.

Four philosophers based at the Centre for the Study of Perceptual Experience (CSPE) at the University of Glasgow (Macpherson, Brown, McDonnell and Colburn) who, between them, have expertise in philosophy of perception, metaphysics, and moral and political philosophy, will build a network of scholars, through a series of online workshops, to consider philosophical and practical questions about AR. They will also organise a KE workshop on the nature and capabilities of AR with our business partners, Sublime (details below), and another KE workshop on AR policy and implementation with civil servants and lawyers. The project will culminate with an in-person conference at the CSPE that brings the participants together, showcasing our research, weaving the various strands of the research together, agreeing concrete guidance for regulating this new technology, and setting the agenda for future research, including applying for further grants.

Our work on AR will begin by examining the experiences had using AR and then, in the second stage, examining the nature of the virtual objects we seem to be aware of using AR. We will subsequently consider the ethical issues surrounding AR and, finally, compose guidelines for its ethical use. The rationale behind this approach is that only by fully understanding the technology, in the first and second stages, will we be able to assess the ethical implications of it in stage three, and then, at stage four, make policy recommendations.

More detail of each of these four stages now follows:

(1) AR raises important questions about perceptual experience. AR can provide extra information about our environment. For example, people’s names might appear beside them. But AR can also provide misleading experiences. How people look might be altered: they might be made to look younger, more beautiful, or to be of a certain race. Slums might be erased from view, replaced with beautiful landscaping. We normally think of perception as a trustworthy source of information. Would it remain so if AR becomes ubiquitous? Could perception be undermined as a source of knowledge by extensive AR usage? Moreover, what account can we give of the blended nature of normal (typically veridical) and AR (often non-veridical) elements of experience? Do extant accounts of illusion and hallucination explain this blend?

(2) We will consider the nature of the virtual objects we seem to see using AR. Do virtual objects exist? If so, are they immaterial, computational, emergent, functional? Can they possess value: aesthetic, financial, moral? Can they be bought, sold, or stolen (as courts in the Netherlands have recently considered)?

(3) Armed with an account of AR experience and virtual objects, we will turn to ethical questions. The introduction of the internet and other technologies led to inequalities that did not previously exist. When no one had the internet, people were not disadvantaged by not having access to it, whereas today, much of our society is organised assuming that people can access the web. Will the same become true of AR, since it can add greatly to our knowledge and capabilities?

AR can create misleading appearances; thus, it has the potential to give us false beliefs, prevent our access to true belief, and bias what we attend to. Thus, people who have the power to determine what AR experience is like have the potential to manipulate people by altering their experiences. This concern is made all the more pressing given future AR might involve wearing contact lenses or implants that cannot be (easily) removed combined with the potential future necessity of using AR to participate in society, mentioned above. We intend to map the sorts of harms that can accrue in this fashion, such as affecting what people think exists, altering people’s political views, skewing people’s choices, affecting their autonomy, and so on.

A particular form of these worries arises from the fact that there are typically just a few real statues in city squares, and laws concerning trespass, property, and access are built around that—whilst in AR there could be as many statues as there are users. Should there be limits on who can place which virtual objects where? Can a city legitimately stop a political group placing virtual statues in their city squares? Can a gallery owner be harmed by an artist placing virtual objects in their gallery?

(4) Finally, in response to these ethical problems, our fourth topic will be what sort of moral and legal guidelines societies ought to have in place to counter them. Who should have the power to make AR content and place it at locations? What sort of content is harmless and what not? What information should people be given about how AR is manipulating their experience? For example, if I am told that the AR will make people more beautiful, ought I to be told if that affects the apparent race of people? Would such information allow me to understand how my world view might be affected? Can we always know what all the consequences of changing the appearance of things will be? What kind of consent is optimal before using AR? What kinds of motive for creating AR experience are acceptable and unacceptable?

Impact

The research will improve our understanding of AR, allowing those who create and consume AR to appreciate the nature of AR experiences and virtual objects. It will affect whether, or in what form, people seek copyright, legal claims to ownership, and understand the value of AR products. Moreover, our exploration of the ethical impact of AR and the guidelines that we develop to govern its use will allow consumers to understand the pitfalls of the technology and seek full information on how what AR experience adds, alters, and deletes so that they understand how their worldview is being changed. Our guidelines, shaped by engagement with policymakers, disseminated on our website, and hopefully taken up by those policymakers, will help inform future regulation. Our work will also allow those making AR experiences, maximise the technology’s benefits while avoiding its potential ethical pitfalls around manipulation and consent.

Project Events

We will host a series of online workshops on the topics outlined above. The workshops are numerous, but each will be short: lasting one afternoon. Each workshop will feature a number of speakers and will also allow time for group discussion. The last five workshops on policy development will feature fewer speakers and will devote time to virtual breakout rooms where participants will consider policies and guidance and work together to draft regulation.

We encourage all participants to attend as many of the workshops as they can, with a view to building up a sustained collective body of expertise to carry out future research and policy work.

Where speakers are agreeable, we will record talks and put them on this website together with copies of written papers or handouts. We will also put drafts of the policies on the website as we are developing them.

Mailing List

To receive news and updates about the workshops, join our mailing list by writing to laura.fearnley@glasgow.ac.uk

Workshop 1: The Nature of Augmented Reality (27th January 2022)

Our opening workshop lays the groundwork for our network by focussing on the technology of Augmented Reality itself. Speakers from industry and academia will outline the state of the technology today, and its future trajectory. How the technology works – the processing, the optics, the software, and the interfaces – is highly salient to a range of philosophical and policy considerations. This workshop’s main aim is to establish a common understanding and language around the technology for the benefit of the project as a whole. 

Speakers

Neil McDonnell (University of Glasgow)
Steve Holmes (University of Glasgow, formerly Meta)
Julie Williamson (University of Glasgow)
Mark McGill (University of Glasgow)

Abstracts and Talks

Neil McDonnell - VR and AR and Philosophy
VR and AR and Philosophy 

Steve Holmes - Meta and AR

The purpose of this paper is to outline the technology that will likely be used in a compelling, glasses form-factor, high-end, socially-acceptable, all-day-wearable, consumer-based, augmented reality (AR) headset. General system specifications will be summarised and some potential use cases considered. The links between use cases and the associated technological challenges, along with potential solutions, will be discussed. Some ethical issues that result from the use cases (or cases of misuse) will be considered. These will be limited to a sub-set of those that are already being considered to some extent by the organisations developing this technology.

The definition of the type of AR under consideration is important. This type of AR system is not a head-up display from the avionics or automotive industry, nor is it a single line of text hovering just below the user’s eyeline, nor is it tablet/smart-phone based. This is the type of compelling AR system to which a consumer-focused tech giant would be happy to apply their logo. It will allow full interaction with the real world via vision, hearing, and touch whilst superimposing on the real world convincing virtual objects, sounds, and text.

The question of what a system of this type will be able to do for its stakeholders remains open, some new use cases will develop, and some will arise that we cannot even begin to appreciate. Some relatively obvious use cases have already been surfaced, such as:
For consumers:

  • SuperPowers
  • Telepresenc
  • Emulation of existing tech
  • Context-aware personal assistants 

For suppliers both of the platform and in the more general sense:

  • Focused advertising
  • Direct selling
  • Users’ data access

Each of these use cases already exist in some form on existing platforms. When they are applied in widespread AR they create the potential for additional ethical issues. Some known examples of these include:

  • Skin-tone rendering inequalities
  • Identification and use of users’ characteristics of characteristics unknown to the user
  • Privacy violations.

Fortunately, AR of this type requires technology breakthroughs that do not yet exist. We may be 10 years away from this type of fully integrated AR system and so there is some time to consider the wider societal and ethical issues.


Meta and AR

 

Julie Williamson - Social Dimensions of AR Use

Immersive technologies make it possible to develop new forms of entertainment, productivity, and social experiences. Advances in graphics, tracking systems, and input devices mean that VR research can now focus on the challenges of interaction in real world contexts. My research focuses on new applications of immersive technologies for social and public settings. Even though head-mounted displays like Oculus Quest technically work in settings like a bar or restaurant, an office, or during travel, the social acceptability of such interactions is still an open challenge. In this talk I will discuss my current research in public and social settings, how immersive content can be a social experience, and the open challenges for a future where we have “always on” immersive devices.


Social Dimensions of AR Use

 

Mark McGill - Bystanders and Privacy

Bystanders and Privacy

 

 

Workshop 2: AR and Philosophy of Perception 1 – Illusion and Hallucination in AR (31st March 2022)

The workshop focuses on the kinds of perceptual experiences facilitated by AR. AR experiences are inherently about a mixture of both virtual and physical objects and properties. This generates challenging questions around: the differences and similarities between experiences of the virtual and experiences of the physical; the ways in which these types of experiences might interact with one another; and the extent to which these types of experiences are accurate as opposed to illusory or hallucinatory. A key practical issue concerns the ways in which AR experiences might be used to manipulate perceivers, whether to positive or negative effect. This workshop will examine these issues with an aim to establishing a common understanding and language around AR perceptual experiences and the means by which they might manipulate perceivers.   

Speakers

Dr Derek Brown (University of Glasgow)
Professor Katalin Farkas (Central European University) 
Professor David Chalmers (New York University)  

Abstracts, Talks and Q&As

Derek Brown: AR, Indirect Perception & Illusion

Indirect perception is perceiving something by virtue of perceiving another thing. AR perceptual scenes involve overlapping virtual and physical objects. By virtue of this, such scenes involve indirect perception. I survey different types of indirect perception found throughout our everyday perceptual lives. I then show various ways these can generate illusions. Following this I consider the AR case, where all forms of indirect perception can apply. The opportunities for perceptual illusions are vast. I consider several possible cases, though the limits of these possibilities are largely unknown at present. Of particular interest are broad scene distortions that adjust the overall “look” of physical things. On the positive side, these might be used to help combat some mood disorders, to undo perceptual biases, and to generally make our perceptual lives more interesting and useful. On the negative side, broad scene distortions might draw our attention away from important aspects of physical reality or hide them from us entirely. Given the scope and kinds of perceptual manipulations that will be possible with AR contexts, AR designers, researchers and policymakers should be attuned to these issues. 

AR, Indirect Perception & Illusion

 

Katalin Farkas: Illusion and Hallucination in Virtual Reality

In my earlier work, I defended a "constructivist" view of perceptual experience. When our sensory experiences come in a certain crossmodally coherent and predictable order, they give rise to the idea of an object that causes clusters of these sensations - and if there is indeed an object that causes them, this is the object we perceive. For example, when we hear and see a train arriving, the familiar changes in the auditory and visual sensations as the train approaches cluster around a certain object: the train. It is tempting to think that on this view, in virtual and augmented reality, we perceive virtual objects. After all, we have sensations that form a certain structure, and there is something that is responsible for this structure: lines in the computer program. However, I will argue that data structures don’t always possess the kind of unity that is required from an object of perception, and hence they are not actually perceived. Much of virtual reality is an illusion or hallucination, even if we are not taken in by these deceptions. 

Illusion and Hallucination in Virtual Reality

 

David Chalmers: Perception, Illusion, and Hallucination in Virtual and Augmented Reality

I will argue that perception in virtual reality and augmented reality need not involve illusion or hallucination. In sophisticated users of VR and AR, perception typically presents things as they are.

Perception, Illusion, and Hallucination in Virtual and Augmented Reality

Workshop 3: AR and Philosophy of Perception 2 – Epistemology (26th May 2022)

In this workshop, we turn our attention to epistemological issues in augmented reality. Epistemology is the study of knowledge. Rene Descartes famously raised the possibility that our experiences—and our beliefs based on them, such as that trees are green—might not reflect the way the world really is. Perhaps an evil daemon has always been feeding us erroneous experiences of a world that doesn’t exist, and there are no green trees in the world. A modern version of this idea is that perhaps we might have always been in a completely virtual world generated by a computer. These scenarios raise the sceptical worry: can we trust our experience to give us knowledge about the world? The response of some philosophers to this problem of global scepticism has been to argue that if we were in such a situation, and had been for all of our lives, then we really wouldn’t be being deceived, for our experiences—and our beliefs generated by them—would really be about the virtual world and not about the “real” world lying behind that. And those experiences and beliefs would be accurate or true of that virtual world – there would be green trees in it. That sort of response is put under pressure by thinking of scenarios in which we sometimes experience the real world as it actually is and sometimes experience elements of a virtual world—but in which we can’t tell which is which. This scenario will be realised when augmented reality technology is perfected. Such a scenario seems to raise new sceptical worries. Would we then have undermined our perceptual knowledge of the world by introducing augmented reality experiences?

This workshop examines these, and related, problems with talks from three of our esteemed colleagues in epistemology at the University of Glasgow. We start with Prof Jack Lyons giving us an overview of epistemological theories and the problems augmented reality raises for them. We move on to hear from Dr J. Adam Carter who argues that the sceptical problems people have to date claimed augmented reality raises are not so serious. However, he raises a new, more worrying, sceptical problem that augmented reality engenders. Finally, we hear from Dr Emma Gordon who considers reasons we may have for resisting pharmacological cognitive bio-enhancement – that is taking drugs to gain new knowledge – and whether those reasons should also make us resist cognitive enhancement by means of augmented reality.

Speakers

Prof Jack Lyons (University of Glasgow)
Dr J. Adam Carter (University of Glasgow)
Dr Emma Gordon (University of Glasgow)

Abstracts, Talks and Q&As

Jack Lysons - 'Introduction to the Epistemology of Augmented Reality'

What are the epistemological implications of augmented reality? I examine the most salient options, discussing the likely epistemological impact of AR, given various kinds of AR and various background epistemological theories. Throughout, I will focus on perception and address questions of both knowledge and justification.

 
 

J. Adam Carter - 'Augmented Reality and Scepticism'

Some recent epistemologists have expressed the worry that augmented reality generates ‘sceptical problems’, and they do so by pointing to our claimed inability to differentiate between the virtual and the real in AR space. One strategy that would nip such arguments in the bud would be to contest received thinking about the difference between the virtual and the real (Chalmers 2017). But how else can might we respond to such worries, without disputing this difference? This talk has two aims. First I’ll show why extant arguments for ‘augmented scepticism’ overstate their case. The sense in which we are unable to discriminate the virtual and the real in VR space is not nearly as sceptically disastrous as some epistemologists have thought. Even so – and this is the second aim – I’ll suggest that there is a type of AR-based sceptical problem in the neighbourhood, one which turns on considerations to do with ontic occlusion rather than indistinguishability, which is comparatively more serious but thus far overlooked.

 
 

Emma Gordon - 'Augmented Reality and Intellectual Enhancement'

A central topic of research in recent bioethics concerns cognitive bioenhancement – the use of the latest science and medicine aimed at improving cognitive functioning to make us ‘better than well’. Even though cognitive enhancement offers a more expedient route to acquire epistemic goods such as true beliefs and knowledge, bioconservative philosophers maintain that we should forego enhancement on the basis of arguments that appeal to (among other things) (i) the alleged ‘cheapened’ value of our (enhanced) cognitive achievements; and (ii) the idea that relying on enhancements to gain knowledge undermines our intellectual authenticity. Such arguments have focused principally on pharmacologically mediated cognitive enhancements – such as ‘smart drugs’, and have yet to be extended to technologically mediated intellectual enhancement via augmented reality. In this talk, I will outline the achievement- and authenticity-based arguments bioconservatives have raised against pharmacological cognitive enhancement and consider what versions of these arguments look like when applied to cognitive enhancement via AR. Ultimately, it will be shown, neither argument offers a compelling case to refrain from enhancement via AR. 

 
 

Workshop 4: AR and Metaphysics 1 - Do Virtual Objects Exists? (July 2022)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Workshop 5: AR and Metaphysics 2 - What Properties Can Virtual Objects Have? (September 2022)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Workshop 6: Knowledge Exchange – How can theory inform practise and vice versa? – in partnership with company Sublime (October 2022)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Workshop 7: AR and Ethics 1 – Harms from what others acquire (November 2022)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Workshop 8: AR and Ethics 2 – Identifying harmful content (January 2023)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Workshop 9: AR and Policy Development 1 – Identifying harms from harmful content (March 2023)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Workshop 10: AR and Policy Development 2 – Consent and AR use (May 2023)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Workshop 11: AR and Policy Development 3 – Drafting responsible AR guidelines (July 2023)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Workshop 12: AR and Policy Development 4 – Redrafting responsible AR guidelines (September 2023)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Workshop 13: Knowledge Exchange – AR guidelines in practice – in partnership with civil servants and the legal profession (October 2023)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.

Final in-person conference (November 2023)

Workshop details including speakers, talk abstracts and the schedule will be announced closer to the workshop date.