All content hosted on the Knowledge Base site is shared under a Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license, unless otherwise stated. A human-readable summary of the full license is provided below. Additional licences may apply to non-hosted content (e.g. links out or embedded media).

You are free to:

• Share — copy and redistribute the material in any medium or format

• Adapt — remix, transform, and build upon the material for any purpose, even commercially

• The licensor cannot revoke these freedoms as long as you follow the license terms.

Under the following terms:

• Attribution — You must give , provide a link to the license, and . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

• ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the as the original.

• No additional restrictions

XR hardware are physical computer hardware components used to access XR content. Contemporary XR hardware typically consists of a set of interconnected off-the-shelf hardware and software components, which we will refer to as an XR system.

Assessing XR Hardware

• What kind of computer system is needed to run the software?

  • How flexible is the specification of the hardware?

  • Is a dedicated GPU required? If so, does it require one with a specific feature set?

• How is sound played back? Is this played back via the HMD or is additional hardware required to support this? e.g. audio interface, speakers, headphones.

• What kind of interactivity is supported? What kind of hardware is needed to support this?

  • Does the experience require the use of a controller or other human interface device (HID)?

  • If yes to above, can this be fulfilled by a generic piece of hardware or is a specific or custom-made device required?

• Does the experience involve sound? How is this played back?

  • Is utilised and if so how is it played back? e.g. via positional speakers or head-related transfer function (HRTF) through an or headphones.

  • Is subtitling or captioning used, or might they be needed?

• Are there any technical or conceptual reasons that specific XR hardware (e.g. ) is needed to access the XR content?

  • Dependency relationships with specific hardware can be baked into software when it is created (e.g. by using certain manufacturer-specific plugins).

  • A creator might prefer certain hardware for its characteristics (e.g. technical, visual, conceptual).

• How many users are supported simultaneously? e.g. single or multiple stations providing access, discrete or shared experience.

• Does the software require internet access? If so, what for and does this need to be maintained?

Acquisition Checklist

If specific or custom hardware is an important component of the XR material you are caring for, the following measures may help you maintain access and prepare for long-term preservation:

• Collecting multiple examples of exemplar hardware as a means of providing access in the short-term.

• Create disk images of any internal storage media of computers.

• Gathering documentation of the hardware and its components.

The Preserving Immersive Media Knowledge Base is a resource created to help share information between members of the digital preservation community who are caring for virtual reality (VR), augmented reality (AR), mixed reality (MR), 360 video, real-time 3D software and other similar materials. This site was born out of with additional funding from the .

The Knowledge Base is designed as a flexible and collaborative space for sharing information and materials that may be work-in-progress or based on best available knowledge. As such, it is constantly evolving and pages can never be considered final or authoritative. If you are new here, these pages might be useful places to start:

• with immersive media preservation.

• of frequently encountered immersive media and preservation terminology.

• that the community is trying to answer.

For a longer read, the published as an output of Tate's immersive media research is a useful place to start learning about preserving VR.

Preserving Immersive Media Group

The Preserving Immersive Media Group (PIMG) is a community and email list for those interested in collecting, preserving and stabilising artworks that utilise immersive media, including 360 video, real-time 3D, virtual reality and augmented reality. This group was born out of the ongoing Preserving Immersive Media project at Tate. We encourage all with an interest in these topics to join the PIMG email list on Groups.io. Members are welcome participate in discussion and to share any relevant information via the list, providing they agree to the PIMG Code of Conduct.

PIMG also runs regular events, recordings and outputs from which are made available online.

List of Immersive Media Preservation Projects

List of Immersive Media Artworks

The purpose of this list is to show the variety of artworks and technologies used, but also to foster exchange between conservators. The listed artworks are not necessarily well documented case studies. In the list below not only VR- or AR-based artworks are included, but also other interactive artworks that are produced with a game engine and hence require similar preservation strategies.

The preservation of immersive media is an emerging topic in digital preservation and presents many new challenges and opportunities for research. This page is a place for tracking questions and prompts that we have arrived at through research, discussion and daydreaming. Do you think you can contribute an answer to any these or do you have any questions of your own? We'd love your contribution!

VR

• What are the most effective approaches to consistently capturing video documentation of VR experiences? Including field-of-view capture, video etc.

• What does the process of adding OpenXR support to an existing VR experience look like? Are there any changes that result and how might these be managed?

• Diagram and technical specs for underlying technology of headsets

3D File Formats

• Which 3D data file formats are most suitable/sustainable for preservation purposes? Are there any which are not?

Tools

• Gap analysis of preservation tools - what is available, what could be developed?

• Calibration tools- comparison with game brightness etc

• Capture tools

Acquisitions

• A complexity matrix- tool to enable advocacy for institutional resources

Documentation

• How can we navigate the relationship between the artwork and the technology, to what extent is the experience determined by an individual’s engagement with the peripheral devices, and to what extent might this be understood and managed over time?

• What non-technical frameworks exist to help us understand user experience? e.g. oral histories, historical context of technologies

• What combination of technical tools and non-technical frameworks might we employ to try to create a sustainable catalogue of user experience, and how might this feed into the historical records of an artwork?

360 Video

• What are the most appropriate metadata standards and how can they be applied?

PIM Knowledge Base

• How do we identify and deal with broken links in the future? GitBook doesn't offer this service, but could we use the GitHub repo (e.g. )?

• How should we run/support the knowledge base in terms of encouraging contributions? How might this play out in the long-term?

We are gathering a curated list of publications and other resources on the topic of immersive media and its preservation. You will find references to external resources embedded throughout the Knowledge Base, while an extensive bibliography of references can be found in our public Zotero library:

https://www.zotero.org/groups/4453604/preserving_immersive_media/library

If you want to add something to the bibliography, drop us an email so we can invite you to the Zotero group or add it for you: tom [at] tomensom.com

The Preserving Immersive Media Group (PIMG) runs occasional workshops, webinars and other events - find link to recordings of presentations, slides and other materials relating to them here. Event recordings from PIMG events can be found on the PIMG YouTube channel.

Knowledge Building Sessions

A series of collaborative events held during 2023-24, focused on knowledge building activities around particular topics in immersive media preservation. Sessions invite participants to share experiences and collectively work towards building knowledge in this field, with a focus on possible contributions to the knowledge base.

The first took place on Tuesday 12th December 2023 and collaborative notes from this session can be found here:

Documenting the Interactive Documentary Webinar (6 November 2020)

Recordings of presentations are available online via a YouTube playlist:

Preserving Immersive Media Workshop (27 March 2020)

Recordings of presentations are available online via a YouTube playlist:

Slides, Notes and other Documents

iPRES 2019 Hackathon

The iPRES 2019 hackathon was an effort to better understanding the variability of virtual reality artworks e.g. what differs between artworks depending on the choices made by artists; and what happens when we make changes to elements of the work.

A full abstract can be found on the iPRES website:

Detailed notes from the event can be found on the collaborative notepad hosted by Rhizome:

Preserving Immersive Media Workshop (8 March 2019)

Slides, Notes and other Documents

This glossary is an effort to gather definitions of frequently encountered immersive media terminology and support a common vocabulary in collaboration between disciplines. The was compiled during iPRES 2019 by participants in the VR Hackathon event, using definitions sourced from existing online glossaries:

• Unity:

Immersive media (XR) is a term used to describe a set of related technologies that aim to extend our physical reality in various ways. You might be familiar with terms like Virtual Reality (VR) and Augmented Reality (AR), both of which are kinds of immersive media. In this section of the Knowledge Base, you will find information to help you get started with immersive media preservation and steer you towards resources appropriate to your context.

The recommended route through these pages is to start by reviewing the prompts in the Initial Assessment section, which will help you characterise the XR material/experience you are working with and then point you in the direction of relevant sections of the Knowledge Base.

Document History

The first version of this Code of Conduct is based on that used by the . Future updates or alterations to the Code of Conduct should be noted here along with the date and actioner.

Code of Conduct

The Preserving Immersive Media Group is dedicated to providing a harassment-free environment for everyone. We do not tolerate harassment of participants in any form.

This code of conduct applies to all (online and offline) Preserving Immersive Media Group spaces, including including mailing lists, online meetings, in-person meetings, the GitBook space, the GitHub repository. Anyone who violates this code of conduct may be sanctioned or expelled from these spaces at the discretion of the response team.

Some Preserving Immersive Media Group spaces may have additional rules in place, which will be made clearly available to participants. Participants are responsible for knowing and abiding by these rules.

Harassment includes:

• Offensive comments related to gender, gender identity and expression, sexual orientation, disability, mental illness, neuro(a)typicality, physical appearance, body size, age, race, or religion.

• Unwelcome comments regarding a person’s lifestyle choices and practices, including those related to food, health, parenting, drugs, and employment.

• Deliberate misgendering or use of ‘dead’ or rejected names.

The Preserving Immersive Media Group prioritizes marginalized people’s safety over privileged people’s comfort. The response team reserves the right not to act on complaints regarding:

• ‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’

• Reasonable communication of boundaries, such as “leave me alone,” “go away,” or “I’m not discussing this with you.”

• Communicating in a ‘tone’ you don’t find congenial

Reporting

If you are being harassed by a member of Preserving Immersive Media Group, notice that someone else is being harassed, or have any other concerns, please contact a member of the . If the person who is harassing you is on the team, they will recuse themselves from handling your incident. We will respond as promptly as we can.

This code of conduct applies to Preserving Immersive Media Group spaces, but if you are being harassed by a member of Preserving Immersive Media Group outside our spaces, we still want to know about it. We will take all good-faith reports of harassment by Preserving Immersive Media Group members seriously. This includes harassment outside our spaces and harassment that took place at any point in time. The response team reserves the right to exclude people from Preserving Immersive Media Group based on their past behavior, including behavior outside Preserving Immersive Media Group spaces and behavior towards people who are not in Preserving Immersive Media Group.

In order to protect volunteers from abuse and burnout, we reserve the right to reject any report we believe to have been made in bad faith. Reports intended to silence legitimate criticism may be deleted without response.

We will respect confidentiality requests for the purpose of protecting victims of abuse. At our discretion, we may publicly name a person about whom we’ve received harassment complaints, or privately warn third parties about them, if we believe that doing so will increase the safety of Preserving Immersive Media Group members or the general public. We will not name harassment victims without their affirmative consent.

Consequences

Participants asked to stop any harassing behavior are expected to comply immediately.

If a participant engages in harassing behavior, the response team may take any action they deem appropriate, up to and including expulsion from all Preserving Immersive Media Group spaces and identification of the participant as a harasser to other Preserving Immersive Media Group members or the general public.

Response Team

Tom Ensom: tom [at] tomensom.com

Jack McConchie: jack.mcconchie [at] tate.org.uk

The Knowledge Base in a collaborative and community-driven space for sharing anything from publications, to guides or workflows, to technical notes and work-in-progress. We welcome contributions big or small!

As a contributor, you agree to the Code of Conduct and that any content you create on this GitBook site will be shared according to our Licence terms.

If you contribute, please do add your name to the list of contributors!

How to Contribute

If you are interested in contributing, you can join the team as an Editor by following this link:

Once you have logged in and joined the PIMKB group on GitBook, you will be able to make edits to pages:

1. You can start editing by clicking on the 'Edit' button in the top right corner. Changes you make in Edit mode will exist independently from the live version until you submit them for review.

2. Add a brief description of the changes made during your edit by filling in the 'Describe your changes...' field at the top of the page. This will help the reviewer understand the rationale behind your changes.

3. Once you are happy with your edit, click the 'Submit for review' button in the top right corner. Your edit will then be checked by an administrator before going live.

For more information on GitBook and how to use it, check out the .

Need Help Contributing?

If you'd like to help or support with contributing or using the GitBook platform, or just want to chat about an idea for a contribution, please do get in touch with the site admins:

• Tom Ensom: tom [at] tomensom.com

• Jack McConchie: jack.mcconchie [at] tate.org.uk

Virtual Reality Fundamentals

• Google. Google VR: Fundamental Concepts. URL: https://developers.google.com/vr/discover/fundamentals

• Brown VR Software Wiki. URL:

Virtual Reality Preservation Research

• Campbell, S. (2017). A Rift in our Practices, Toward Preserving Virtual Reality [Master’s Thesis, New York University].

• Campbell, S., & Hellar, M. (n.d.). From Immersion to Acquisition: An Overview Of Virtual Reality For Time Based Media Conservators. Electronic Media Review, Six: 2019-2020. Retrieved October 7, 2021, from

• Cranmer, C. (2017). Preserving the emerging: virtual reality and 360-degree video, an internship research report. Undefined.

Getting started with the long-term care of immersive media (XR) can be overwhelming. Multifaceted production processes, complex components and challenging access requirements can all confound applying established digital preservation practice. This section will help you work through an initial assessment, identifying the characteristics of the XR experience you are trying to preserve and pointing you in the direction of resources where you can learn more.

Context

This section considers where an XR experience has come from, including established networks of care, and the institutional frameworks which will need to navigated.

Production and Display History

• How was the experience created? What conditions/context informed this process?

• Who created and/or has cared for the experience up until now? What will their roles be as long-term care continues?

• How has the experience been presented in the past? Can you gather documentation to support transmitting this to future stakeholders?

Organisational

• Can this media be collected in accordance with your organisations preservation policy? The longevity of an XR experiences is difficult to predict and there are few guarantees when it comes to long-term access.

• What are your overall goals in terms of preservation i.e. what 'level' of preservation do you hope to achieve? e.g.

  • of the media?

Experiential Characteristics

Immersive Media Type

Immersive media technologies can be used to create experiences that combine elements of physical and virtual environments in different ways. The following terms are frequently used to characterise types of immersive media:

• Virtual Reality (VR) refers to experiences where a user is fully immersed in a virtual environment. All elements of the environment experienced are virtual (typically through an ) and the user is unable to see the physical environment they are occupying.

• Augmented Reality (AR) refers to experiences which add virtual elements to a real-world physical environment. The environment experienced combines virtual and real-world elements, the former often being superimposed onto a camera feed of the latter.

• Mixed Reality (MR) combines elements of VR and AR.

Together, VR, AR and other related terms, are sometimes referred to using the umbrella label XR.

Interactivity

The extent to which the user can interact with an XR experience can vary in several ways. One is the extent to which the user has control over their spatial position in the virtual environment during the experience:

• In a fixed-position experience, the user views the environment from a fixed position. Only rotational tracking, known as three degrees of freedom or 3DoF, is used in these experiences.

• In an on-rails experience, the user moves through the environment along a predetermined path. Only rotational tracking, known as three degrees of freedom or 3DoF, is used in these experiences.

• In a fully interactive experience

Elements of the virtual environment may also be interactive. This is dependent on the way in which the VR content has been produced. 360 video content (see ) is typically not interactive, as the moving image sequence was predetermined when the video was produced. Real-time 3D software content (see ) is more likely to have interactive elements as the moving image sequence is generated on-the-fly and so can respond to user input.

Technical Characteristics

Identifying Key Components

Technical characteristics depend on the kinds of components the experience makes use of. Immersive media content is the stored digital material that is created to be experienced through XR hardware. This generally fits into one of two categories, which significantly impact the approach taken to preservation. Identify what type of immersive media content you are working with and consult the relevant subsections of the Knowledge Base:

• 3D Software is immersive media content produced or stored as real-time 3D software (see ). This is typically produced using .

• 360 Video is immersive media content produced or stored as video (see ).

Then consider what hardware or other physical components are needed to provide access to this content:

• XR Hardware is the physical computer and human-interfaces devices required to access the immersive media content (see ).

• Physical Components are other (non-electronic) physical components that form part of the immersive media experience (see ). This might include props, flooring, seating or other environment characteristics.

Condition Assessment

Having identified the components, consider their condition and the potential implications for long-term care. This can help identify potential vulnerabilities, inform preventative actions and highlight immediate treatment needs. The following questions may help guide this process:

• If hardware has been received, is this functional? If not, what needs to happen to make it functional?

• Are any of the components used obsolete (i.e. neither sold nor supported anymore)? If so, are these components integral to the work and can they be repaired?

• Given responses to the above questions, would it be worth intervening now to stabilise the condition of the components?

Thank you to all those who have generously given their time and knowledge to help shape the Knowledge Base:

• sasha arden

• Rasa Boycte

• Jack McConchie

• Claudia Roeck

• Samantha Rowe

• Jesse de Vos

• WinZs

If you have contributed to the Knowledge Base, please do add your name here (and optionally, a hyperlink to an appropriate URL) and .

OpenXR is an open, royalty-free standard for APIs that provide XR applications with access to XR platforms and devices. This is implemented in the XR runtime software supplied by the manufacturer of XR hardware. Application support for OpenXR is potentially useful for preservation purposes — as it is a open standard, which will make keeping software available that

OpenXR is developed by a working group managed by the Khronos Group consortium, who describe it as follows:

OpenXR is an API (Application Programming Interface) for XR applications. XR refers to a continuum of real-and-virtual combined environments generated by computers through human-machine interaction and is inclusive of the technologies associated with virtual reality (VR), augmented reality (AR) and mixed reality (MR). OpenXR is the interface between an application and an in-process or out-of-process "XR runtime system", or just "runtime" hereafter. The runtime may handle such functionality as frame composition, peripheral management, and raw tracking information.

Optionally, a runtime may support device layer plugins which allow access to a variety of hardware across a commonly defined interface.

—

Up until the arrival of OpenXR, support for each manufacturers API would have to be built into the XR applications if they were to be used.OpenXR attempts to solve the problem of compatibility between XR applications and XR hardware. Image source: .

Using OpenXR

In order to make use of OpenXR, you need to:

1. Develop software which supports — see Engine Implementations below.

2. Make use of an XR platform which supports it — see XR Runtime Implementations below.

XR Runtime Implementations

The that the following XR runtimes/platforms are compliant with OpenXR:

• Acer:

• ByteDance:

• Canon:

Engine Implementations

Documentation provides an important resource, accompanying, or even standing in for, an immersive media (XR) experience as it ages. As a starting point for documenting an XR experience, the following prompts may be useful:

• Recording key information gained during initial assessment of the characteristics of an XR experience e.g. using the templates on this page.

• Creating video documentation of XR content using video recording tools.

  • What is the user experience like?

• Creating or recording personal narratives/accounts of works (i.e. oral histories).

  • From whose perspectives are these recorded? e.g.

    • User/viewer

• Could your documentation become ''?

  • Documentation created while the experience is accessible can become a useful preservation pathway later, by transmitting some understanding of the original experience.

  • How do you measure the success of your documentation? e.g. completeness, multiplicity, accuracy...

• Consider outside-in and inside-out perspectives on documentation - see referenced below for more info:

  • "An outside-in approach takes the technical characteristics of hardware and software as its starting point, where the material properties determine how content is rendered and experienced."

  • "By contrast, an inside-out approach is user-centered, where experience of the content reveals significant properties that guide decisions as to how content is rendered."

Templates

Acquisition Information Templates

These documents were designed to guide information gathering and discussion during the early stages of the acquisition of virtual reality (VR) or augmented reality (AR) artworks, primarily with conservation and long-term preservation in mind. They are designed to be completed by or in close collaboration with an artist prior to receiving media from the artist.

AR Templates

VR Templates

• (March 2019) developed by Jack McConchie and Tom Ensom during Tate's Preserving Immersive Media project:

• (May 2019) developed as an extension of this template by Savannah Campbell and Mark Hellar:

Publications

Augmenting Our Approach to Preservation: Documentation of Experience for Immersive Media

Written by sasha arden during their 2021–22 graduate internship at Tate, this paper considers the experiential dimension of immersive media (IM) and its significant role in preservation and future access.

PDF available via Tate website:

Documentation Infographic for artists and makers

The infographic was created by Lieve Baetens in 2023, and provides artists and makers information about the importance of documenting their work. The infographic answers questions about why it is important for artists and makers to document their work and how they can document their own work.

Extending the service life of hardware through repair could support preservation efforts if a particular XR system is required. These hardware systems are meant to be used, and in an exhibition context they can be subjected to serious wear and tear.

The consumer market for XR hardware systems and mobile phones is built on planned obsolescence; products are superseded by new models, manufacturers only provide limited warranty periods for repair or replacement, and parts are not commonly available. Purchasing backup equipment or sourcing equipment to be used for parts may be part of a preservation plan.

iFixit is a US-based company that advocates for repairability of consumer devices. The Preserving Immersive Media Group has no affiliation with iFixit. The guides linked here provide step-by-step instructions with photos, as well as tools and in some cases replacement parts. The teardown reviews are another resource should you have equipment that is intended to be used for parts. Popular XR systems and Samsung Gear VR are linked here, but many other device guides can be found on iFixit.

Valve Index:

Meta/Oculus Devices:

HTC Vive Devices:

Samsung Gear VR:

Computer hardware is difficult to preserve in the very long-term as its maintenance is dependent on access to suitable components and specialist knowledge, both of which can be lost with time. XR hardware provides a particularly challenging case study given its complexity. However, repair guides may be able to help maintain it in the near term.

Introduction to Modern XR Systems

Modern XR systems are unified sets of hardware, with supporting software, which are used to provide end users with the capability of accessing immersive media content like XR-enabled 3D software and 360 video. Common components of XR systems include:

• Head-mounted display (HMD) which is placed over the users head, providing the screens which create the visual illusion of being in a virtual environment.

• Human Input Device (HID) which allows the user to interact with elements of the virtual experience (e.g. by pressing buttons or moving a joystick).

• Tracking system which translates user movement in the physical environment into movement in the virtual environment using camera and/or sensor data. In practice this is often integrated, at least partially, into HMD and/or HID devices

• Computer system to run the XR software, which can be integrated into the headset (e.g. in the Oculus Quest series) or run standalone (for tethered headsets like the Oculus Rift S or Valve Index).

• which provides interfaces between these hardware components and XR software.

Head-Mounted Displays

The simplest type of XR system uses a handheld mobile phone mount with lenses that help to focus on the screen content and create the illusion of an immersive view. Google Cardboard is an example of such a handheld device of this type. The next step in features and complexity involves a more sophisticated that is hands-free and provides a better illusion of immersion.

Further interactivity and immersion is achieved with tracking systems, which provide input for the XR content and/or information about the user's position in space. Tracking systems can be built into the HMD (e.g. Oculus Quest series) or hardware incorporating sensors is used along with an HMD (e.g. Oculus Rift or HTC Vive).

Another category among XR systems is standalone types and tethered types. Standalone HMDs (e.g. Oculus Quest, HTC Vive Focus) offer greater freedom of motion and are less expensive overall, while tethered systems (e.g. Oculus Rift S, HTC Vive Pro) use a connected computer to offer better performance in terms of graphics and location processing.

The commercial marketplace has strongly influenced development of XR systems since the early 2010's when a "second wave" of immersive technologies started. Most companies produce tiered feature sets at varying price points. Releases of product models might follow incremental improvements in technology, resulting in very similar systems with only one or two differences between them -- or releases could represent a big jump in technology or hardware design from year to year. It can be challenging to identify defining features and technical specifications for a given XR system because of advertising styles (many hard-to-prove claims are made) and the proprietary, competitive nature of development.

Comparison of XR HMDs

A head-mounted display (HMD) is a display device, worn on the head or as part of a helmet, that mounts either a single screen in front of both of the users eyes (monocular HMD) or a separate screen in front of each eye (binocular HMD). An extensive summary of the properties of VR HMDs is available on the wiki or on . HMDs are generally either "tethered" (such as Oculus Rift) to a PC that monitors tracking and undertakes the rendering, or "untethered" (such as Oculus Quest) where the unit is standalone.

The table below is meant to gather information for the purpose of identifying the features of popular XR systems. It could be useful for considering compatibility in the case of a potential hardware replacement or migration. For example if the XR content uses 6DOF interactivity, an XR system that is only capable of supporting 3DOF would not be suitable.

HMD-Runtime Compatibility

Supporting software is required to connect XR software (e.g. a VR application or 360 video player) to XR hardware. This is typically a software package provided by the manufacturer of an XR hardware system. This may include a user-facing application with a storefront for purchasing content (e.g. Steam or Oculus) but also includes back-end software components which serve a technical role.

The most important component of this package from a preservation perspective is an XR runtime, which provides an API and driver to support an interface between XR applications and XR devices. It can also provide features that modify the behaviour of XR applications (e.g. improving performance of image quality). Examples include the Oculus runtime and the SteamVR runtime. An XR runtime can provide XR applications with a variety of interfaces with which to interact with them, which might conform to a standards such as or OpenVR.

3D software involves the use of software to dynamically generate a moving image sequence from 3D data and code. The real-time nature of the process means that the moving image sequence can be dynamic and responsive to user interaction. This can be contrasted with video, where frames are encoded and played back linearly. In practice, 3D software technologies used in XR production are often very similar to those involved in the creation of video games e.g. game engines, 3D modelling, texturing.

Existing software frameworks and tools are typically used as a starting point for development of 3D software, from which a distributable form of the software is generated. Development for desktop or mobile applications is typically using a . This can result in builds supporting different platforms e.g. desktop, mobile, web. Development for the web may involve the use of web frameworks like and .

An XR runtime is a piece of software that support communication between XR software and hardware. This page describes the process for extracting an XR runtime as a contained unit of software, so that it can be archived independently of a computer system.

XR runtime are often distributed using front-end tools that carry out downloading and installation in the background (e.g. SteamVR is downloaded through Steam). This makes it harder to extract and archive them for reuse in the future. This page describes the process of extracting different XR runtimes for independent archiving.

Oculus (Windows)

Legacy Oculus Runtime 0.8.0 (dating from 2017) is currently available to download from the Oculus website: .

For later versions you are dependent on the to install and manage the runtime. After installation, the runtime can be found in C:\Program Files\Oculus. Testing is needed to ascertain whether this directory can be copied to a new machine.

SteamVR (Windows)

These instructions are adapted from Valve's of SteamVR. While this method allows you to extract a standalone copy of the SteamVR runtime, the version available through Steam cannot be controlled.

1. Either a) Install and open the on a PC with full internet access or b) Access a computer which already has the appropriate version of Steam installed.

2. In the Steam Client, open the Library section and find the part of it labeled "Tools".

3. Find the entry "SteamVR" and install it.

4. Right-click on the entry "SteamVR" and in the resulting popup menu click on the entry "Properties".

5. A new window with multiple tabs will open. Select the tab "LOCAL FILES" and click on the button labeled "BROWSE LOCAL FILES".

6. The directory containing the SteamVR Runtime will open. Copying this entire directory will capture the files required to run SteamVR on another computer. From this directory, SteamVR can be launched by running the "vrstartup.exe" executable file in "\SteamVR\bin\win64".

Unity is a game engine that was initially released in 2005. Released with the aim of democratizing video game development, it was intended to be an affordable engine aimed primarily at independent and amateur game developers. Unity can create both 2D and 3D applications. The engine typically has a major version update every year with multiple smaller updates throughout the year. Unity is free for developers that bring in under $100,000 in revenue in a 12-month period. After that, there are 2 different pricing models. Though originally designed to build games, today Unity is used across multiple industries including automotive, architecture, engineering, construction, film, education, and retail.

This page is designed to help you understand what Unity is and how it can be used by creators. The Unity Engine was initially released in 2005 and was designed to provide affordable game development tools aimed mainly at amateur game developers. Unity can create both 2D and 3D applications. The engine typically has a major version update every year with multiple smaller updates throughout the year. Unity is free for developers that bring in under $100,000 in revenue in a 12-month period. After that, there are two different pricing models. Though originally designed to build games, today Unity is used across multiple industries including automotive, architecture, engineering, construction, film, education, and retail.

Anatomy of a Unity Project Folder

When a new project is created, Unity creates several folders as a part of the project. Over the years and versions of Unity, this default folder structure has changed somewhat, but it now generally contains the following folders:

Assets

This is the folder where all game resources are stored, including scripts, textures, sound, and custom editor scripts. The organization of the folder can vary greatly from one project to another and from one organization to another. There can be numerous subfolders in the Assets folder, depending on how the project is organized. For example, there may be a folder for scenes, one for scripts, one for audio, or one for sprites. There is no limit to how deep the organization can be.

As a best practice, subfolders in the Assets folder should be created within Unity and not the computer's local file system. Likewise, assets should be added directly to the folders in the Unity Editor and not the file explorer as seen above. This is particularly important due to the way Unity constructs metadata for assets.

Unity reserves several special folder names under the asset folder. These folders are Editor, Editor Default Resources, Gizmos, Resources, Standard Assets, and Streaming Assets. Not every project will have all of these folders.

Assets/Editor — This folder is for custom editor scripts that extend the functionality of the Unity editor. These scripts will run in the editor but not in the project at runtime. Multiple Editor folders can exist in the Assets Folder. The execution of the editor scripts varies depending on where in the folder structure the editor file exists.

Assets/Editor Default Resources — This is where asset files used by Editor scripts are stored. There can only be one such folder and it must be placed in the Assets folder root. There can be subfolders in this folder.

Assets/Gizmos — Gizmos are graphics in the scene view which can help to visualize design details. This folder stores images used for gizmos. There can only be one folder and it must be placed in the Assets folder root. Gizmo examples are seen below with the red squares around them. The one on the left is the main camera’s position and rotation. The one on the right represents a light source. These two are Unity built in gizmos:

Assets/Resources — This folder stores resources so that they can be loaded on demand in a Unity project. There can be multiple resource folders. On demand loading is helpful for dynamically loading game objects that don’t have instances created by designers during design time. In other words, these resources may not have corresponding game objects placed in the scene at design time and can be loaded dynamically at run time.

Assets/Standard Assets — This folder stores any standard asset packages that have been imported into a project. There can be only one standard assets folder. Standard assets are free assets maintained by Unity.

Assets/Streaming Assets — This folder is for assets that will remain in their original format and later be streamed into the Unity application, instead of directly incorporating them into the project’s build. An example would be a video file from the filesystem. There can only be 1 streaming assets folder in the project.

Library

Moving on from the Assets folder, the next folder is Library. This is a local cache used by Unity for imported assets. It can be deleted and will be regenerated by unity automatically. All that is needed to recreate the folder are source assets and .meta files. If this folder is deleted, Unity will reimport all asset information and regenerate the folder the next time the project is opened in the editor. This folder should not be included in Version Control. These imported assets are used by Unity to save time when Unity is running.

Of special note in the Library folder is the package cache folder. This contains information about all the packages installed with the current project. Though this can be regenerated by Unity like the other items in the Library folder, for archival purposes, it is important that this file not be deleted. This is because it may be helpful to be able to see what packages are included in the project without having to regenerate the cache, which would require opening the project in its appropriate editor version.

Packages

This folder contains the manifest file in JSON format used to maintain the dependencies link between packages. The manifest file is used to regenerate the package cache in the library folder. It also contains a file listing the individual packages installed with the project. These are used by the Unity Package Manager. The package manager was added in Unity 2018.1. Prior versions of Unity will not contain the package manager and the packages folder will not exist in those cases.

Project Settings

This folder contains all project settings from the project settings menu. It also includes editor version number and build settings and a slew of other settings used by Unity systems. Editor version number, as a standalone file, was not added until Unity 5. For any version before that, the editor version number can be found in the project settings file.

Preservation Considerations

Access and Licencing

Export Formats

Editor Dependencies

Build Dependencies

Game engines are software development tools for creating interactive software. They package together libraries and software which simplify the development of interactive software. Game engines are a widely used tool in the creation of real-time 3D VR software, and many engines support VR production workflows out-of-the-box.

A modern game engine will typically include:

• A 3D or 2D renderer, which supports the rendering of a moving image sequence in real-time.

Unreal Engine 4 (UE4) is a game engine developed by Epic Games. The first version of Unreal Engine was created during the development of the 1998 game Unreal, at which point Epic Games started licencing the engine to other developers. Version 4.0 was released in 2014, and has been followed by 27 subversions (the latest is 4.27). Unreal Engine 5, announced in 2020, is expected to supersede UE4 at some point in the near future. UE4 is not an open-source engine (see the EULA), but the engine source code is freely available via their GitHub repository.

This page is designed to help you understand what Unreal Engine 4 is and how it can be used by creators.

One way of preparing for the preservation of an application made in Unity 5, is to archive the project files associated with it. Executable software is created from a Unity project by exporting an application that supports the target platform. By archiving an Unity project, we aim to gather together all the materials required to carry out to repeat this build process. This opens up options for incremental migration to new versions of Unity, and the modification of code to support other hardware and software platforms.

To build an Unity project, you need the following components, guidance on the archiving of which is provided on this page:

• the collection of custom Unity content and project files

• : software which allows you to open and edit a Unity project folder

• : any additional software not included with the engine binaries or project by default e.g. libraries, modules

Project Folder

#

Editor and Modules

The Unity Editor software can be installed using the Unity Hub software or from the intallers distributed via the .

Before proceeding, you will need to identify the version of the Unity Editor the project was created with. To do so, navigate to the ProjectSettings folder within the project folder and open the file named ProjectVersion.txt (tested in 2018.3.9).

Using Unity Hub

1. Install Unity Hub on a suitable computer and navigate to the Installs tab.

2. Click Install Editor and select the appropriate editor version. If the appropriate version is not available, you will need to install it use the Unity Download Archive method.

You may also wish to install additional modules to improve build support.

3. Return to the previous Installs tab, click on the cog icon next to the Editor entry you wish to archive, and click on Show in Explorer.

4. In the Window which opens, you will be looking inside the application directory for the Unity Editor version. This folder can be archived and used to access the Editor independently of the Unity Hub installer.

Using Unity Download Archive

Dependencies

There are two common ways of extending the functionality of Unity which you may find have been used: Modules and Packages.

Modules extend core features, and includes options to build for non-Windows platforms. Unfortunately, if any non-standard modules have been used by the project your only option is to use the Unity Hub application to download and install these.

Packages are handled by the Unity package manager module and are included in the project folder once they have been added.

Godot Engine is a free open source 3D game engine for Android, Linux, Mac and Windows. A WebEditor is also available.

Some information with relevance to XR preservation, quote from the :

XR support

This page describes the process for archiving an Unreal Engine 4 project and its dependencies, so that it can be archived independently of a computer system.

One way of preparing for the preservation of software made in Unreal Engine 4 (UE4), is to archive the project files used to create it. While it is the executable form of the software (or 'build') which is used to run it, archiving the source form of the software opens up more preservation options such as:

• Creation of new builds which support different platforms and environments;

Mobile platforms and app services are part of preserving and re-exhibiting AR content and mobile phone-based XR content. This page gathers published research and resources on these topics.

“Towards a preservation workflow for mobile apps” (24 February 2021)

“A Race Against Time: Preserving iOS App-Based Artworks” (2019-20)

“Considerations on the Acquisition and Preservation of Mobile eBook Apps” (2019)

Smartphones within the changing landscape of digital preservation (2017)

It may be desirable to explore independent preservation pathways for 3D data the forms a part of a real-time 3D immersive media experience. On this page you can find links to initiatives/workgroups, events and publications on the topic of developing standards for data packaging, file types, metadata, documentation, normalization, migration, storage and access of 3D data.

International Image Interoperability Framework (IIIF) Community 3D Interest Group

CS3DP (Community Standards for 3D Data Preservation)

Council on Library and Information Resources (CLIR): 3D/VR in the Academic Library: Emerging Practices and Trends

Developing Library Strategy for 3D and Virtual Reality Collection Development and Reuse (LIB3DVR)

The software tools listed here support preservation activities for compiled, executable files or project files. Some are proprietary and require a license or developer account, and some are open source. Not all may be currently supported or compatible with your files or hardware.

Preservation strategies are applied to XR content to guide it into the future. In an ideal world, full functionality and the same look and feel would be a preservation goal. However, in the real world technical limitations and limited resources may apply. Stakeholders need to agree on acceptable degrees of loss or change to the artwork and develop the preservation strategy accordingly.

Another consideration in developing a preservation strategy is its sustainability. If possible, a preservation strategy should reduce dependencies and stabilises the software so that the frequency interventive preservation measures is reduced.

3D software used in AR and VR can be highly dependent on specific hardware, particularly the HMD with its sensors, optics and electronics. To make a XR material more widely compatible, it can be made headset-independent. Currently this is supported by building in OpenXR support, an open standard for XR software. If this cannot be used, alternative migration approaches or emulation can be used. These various strategies are discussed in more detail in the sections below.

Preparation

In order to select and successfully apply a preservation strategy, you need to understand what it is you are trying to preserve. A good starting point for this is our page. Working through these questions, ideally in consultation with the stakeholders of the XR material, you are aiming to identify:

• Which components or characteristics of the software are core to the XR experience? e.g. functionality, look and feel, user experience, quantitative / technical parameters; physical integrity.

• Which are variable or less important? Perfect replication may not be necessary to realise the XR experience authentically.

• Are they measurable or documentable in some way? Documentation can be very helpful as a reference for transmitting these characteristics e.g.

This approach will help you determine what can and cannot change during the implementation of a preservation strategy, and develop a framework by which you can assess the success of a particular strategy. It is possible that compromises will have to be accepted because of financial or technical limitations. Therefore, applying a documentation strategy (see sub-section below) can complement one which priorities technical intervention.

Documentation Strategies

A game engine can be used to output a 360 degree video, which can be preserved independently of the software as a video. Capturing the game engine output in this way is a type of documentation strategy. A 360 degree video is much easier to preserve than a game engine project with all its software dependencies. In addition, it does not depend on any specific XR hardware for continued access. To output a 360 degree video, usually the game project is necessary as most tools target specific game engine editors. However, it might be possible to capture the output from running the executable using .

(2022) advocates for the inside-out documentation approach, where the impact that the experience has on the user is central. This kind of documentation can become more relevant than the technical documentation of the artwork, if the functionality of the artwork cannot be preserved due to technical or financial constraints. This approach to documentation is practiced in performance art where the video documentation of the performance can replace the actual performance (for instance, Charlotte Moore performing Nam June Paik's TV cello in 1976).

However, if the capturing strategy is applied to interactive XR experiences, where the experience reacts to user input (other than the head movement), capturing the output leads to a total loss of that functionality. If this is not acceptable, the game engine project must be migrated to a version that supports newer XR hardware

Migration Strategies

A migration strategy involves updating the source code/project to run on newer computer or XR hardware. This requires access to the game engine source code/project (not just the executable) as well as the game engine editor software.

Different approaches to migration can be taken e.g.

• Updating the game engine project to the newest LTS game engine version, and build the project for the newest runtime of the same headset brand.

• Updating the game engine project to the newest LTS game engine version and compiling the project for the OpenXR runtime. OpenXR has open specifications and "aims to

Migration Case Studies

Case study applying an incremental migration strategy: LIMA (2021). A Practical Research into Preservation Strategies for VR artworks on the basis of Justin Zijlstra’s 100 Jaar Vrouwenkiesrecht. URL:

Emulation Strategies

An emulation strategy (and allied virtualization strategy) involves using a contemporary computer system to simulate a historical computer system. If there is only an executable, and the source code/project is not available, emulation can become an especially important preservation strategy for 3D software. Through emulation, the software environment is preserved as a whole, including the 3D software and any dependencies (e.g. the XR runtime).

Emulation is mentioned at the end of this list, as it is often not expedient in the case of XR material. Gaining access to XR hardware (e.g. an HMD) from an emulated environment has not been successfully achieved in a preservation context, as far as we are aware. However, emulation can become an option if there is no HMD used, but other immersive displayed equipment (e.g. video projectors or monitors) are used instead.

Video documentation can be used to record aspects of an immersive media (IM) experience. This can take two complementary forms:

• Physical capture: Video recording of the real-world physical actions of a user interacting with an IM experience.

• Virtual capture: Video recording of the virtual actions of a user interacting with an IM experience, as would be sent to a display device.

Virtual Video Capture

Virtual video capture can produce two kinds of video:

• Fixed-perspective video: video which representing a fixed perspective on the virtual environment; the standard form of video designed for non-interactive viewing.

• : video representing a 360 degree view from a central point, therefore allowing a level of interactivity via rotational tracking.

Capturing Video from an Application

Hardware and software tools can be used to capture video from a real-time 3D application. This can be fixed-perspective or 360 video.

Software Tools

Rendering Fixed-Perspective Video in the UE4 Editor

The Unreal Engine 4 editor has built-in tools which allow export of video sequences. The actions of these sequences can be scripted or recorded from user interaction within the editor. Using them therefore requires access to the source project and a level of engagement with the editor software. This will may involve modification of the source project, so you may wish to work with a copy or version control system.

Render Formats and Settings

Video export format depends on the options you select in the Render Movie Settings dialogue (info derived from MediaInfo output of a rendered video):

• With 'Use Compression' unchecked: RGBA (8 bits-per-channel) in AVI with OpenDML extension container.

• With 'Use Compression checked: MJPG video (YUV, 4:2:2, 8 bits-per-channel), Interlaced, Top Field First) in AVI container.

There are some configurable options, including output video resolution and framerate.

The uncompressed video option yields the best quality output but the video files produced are VERY large - in our test 8 sec of capture (1080p at 24fps) yielded a 1.5 GB file. Take care that you have enough storage space to capture in this format if you are going to use this setting, at which point you can convert to lossless compression format like FFV1 for storage.

Workflow

In order to generate a video from a UE4 project using the editor tools, you need to first create a sequence: a scripted or recorded set of events occurring within the virtual environment. Some projects may already use a sequence to choreograph the actions that occur within the IM experience (e.g. an 'on-rails' experience). In these cases, locate the sequence in the Content Browser and open it. In the toolbar you should see a clapperboard icon, which will open up the Render Movie Settings dialogue.

Where there is not an existing sequence, you can create a sequence by scripting or interaction in the engine.

Rendering 360 Video in the UE4 Editor

360 video can be created with a workflow that utilizes plugins enabling export of stereoscopic frames from UE4, which are then assembled into a video sequence with software like Adobe After Effects (which has support for VR and 360 video).

The following workflow uses , a free plugin included with UE4.

Note that there are some caveats to using Panoramic Capture Tool:

• It does not export audio.

  • Audio could theoretically be pulled from fixed-perspective video of an "on-rails" experience, but another strategy would be required for interactive content.

• It is an Experimental feature in UE4, and is not as actively developed or supported as other UE features.

There are other 360 export tools that can be purchased in the Unreal marketplace.

UE4 Panoramic Capture Tool Workflow

[workflow to be added here]

Physical Video Capture

[To be added]

360° video uses video formats to encode moving image sequences that surround the user in virtual space. The user is able to freely rotate their head, determining the viewing angle. The viewing position may be a fixed position, may follow a pre-determined path (on rails), or may offer some degree of positional interactivity through "portals" which jump the viewer to another piece of 360 video content.

360° video can be created in a number of different ways:

• Captured by a camera or array of camera lenses;

• Generated as an export from 3D rendering software (e.g. Blender);

• Generated from a real-time 3D . The 360° video exported from a game engine can be the artistic end product (s. for instance "" by Studer / van den Berg) or the documentation of a real-time 3D artwork.

Assessing 360° Video

• How was the 360° video created?

  • Was it created through camera capture? If so, is the camera type and output format known?

  • What is created using a 3D software tool e.g. 3D renderer or game engine? Is the capture and output format of the video known?

Acquisition Checklist

The following measures can help ensure long-term access to 360° video:

• Ensuring metadata has been captured describing the projection format and mono/stereoscopic format used.

• Ensuring the video file received is the highest quality available.

• Considering whether the source video files (pre-stitching) should also be acquired.

Monoscopic vs Stereoscopic

360° video can be either monoscopic or stereoscopic. Monoscopic video supports what is perceived as a 2D representation of the scene i.e. there is no perception of depth. Stereoscopic video supports a 3D representation of the scene with a perception of depth.

• Monoscopic 360° video contains video captured from a single viewpoint within the scene.

• Stereoscopic 360° video contains video captured from two viewpoints within the scene. These viewpoints can be different point of view for each eye. These are packed into a single video file where they can be arranged side-by-side or top-bottom.

Video File Format

Codecs: h.264/AVC, h.265/HEVC, VP9

Wrappers: MP4, Mastroska, WebM

Significant variables in choice of video file format:

• Achievable bitrate / compression

• Metadata container options

• ...?

Projection Format

Projection format refers to the way in which data representing a 360° or spherical field of view is mapped to a flat image when it encoded. It is similar to the way in which a map of Earth is a flat representation of the spherical surface of the planet.

Some common projection formats include:

• Equirectangular

• Cubemap

• Equi-angular cubemap

Significant variables in choice of projection format:

• Pixel density

• Tool support (encoding, decoding)

• Requirements of video streaming platforms (e.g. YouTube)

Players and Playback Frameworks

Transcoding

360-degree has specialised transcoding requirements if properties such as projection format and stereoscopy to be properly managed. At this point the tools listed here have not been tested by us and inclusion should not be taken as a recommendation.

List of tools:

• ffmpeg,

• Headjack VRencoder,

frameworks, spatial audio plugins , ambisonic and more

Frameworks

Wwise is an audio framework for MacOs and Windows to create audio interactive content for UnReal and Unity Engines

Spatial Audio

IEM Plug-in Suite

The is a free and Open-Source audio plugin suite including Ambisonic plug-ins up to 7th order for Linux, MacOs and Windows.

VST2, VST3 and standalone app.

StereoEncoder, RoomEncoder, EnergyVisualiser, BinauralDecoder, MultiBandCompressor and more....

Due to the increased complexity and number of variables in 360 video, there is a need for metadata standards to accommodate this. Google has a suite of standards and metadata injection tools on GitHub under the title of "spatial media" here.

In order to upload 360 video to YouTube and have it recognised and played back properly, it is required that spatial metadata be injected using the Google tool. This supports MP4 and WebM (video only) containers. You can find specification information on GitHub for the support elements for video and audio.

It supports:

• Projection type

• Stereoscopic mode

• Projection pose (yaw, pitch, roll) - presumably for initiation position?

• [Add audio]

Spatial Media Metadata Injector

is a open source software that can be used to "convert standard" videos (equirectangular for example) to inject 360° / stereoscopic 3D (top/bottom) and spatial audio (ambix / sn3df) datas