Mark your calendars for IEEE ICIP and ACM ICMR 2021

While the pandemic has mostly had devastating effects on the international conference circuit, there’s also one positive consequence: There are interactive virtual editions of basically all the bigger events now. This means, of course, that you can partake in meetings in exotic locations that would have been hard to get to before – like the ones we recommend in this post:

September 19-22: IEEE ICIP

IEEE stands for the Institute of Electrical and Electronics Engineers (the event’s host), and ICIP is short for International Conference on Image Processing (“Imaging without borders”). It’s a very big, very technical conference with a focus on image processing (obviously) as well as video processing and computer vision. In a world without Corona, this year’s instalment would have taken place in Anchorage (USA). To directly register for IEEE ICIP, follow this link. The official conference website has more info, including a detailed schedule of the technical program. You can also follow the conference on Twitter via the #ICIP2021 hashtag.

November 16-19: ACM ICRM

ACM ICRM refers to the Association for Computing Machinery’s International Conference on Multimedia Retrieval, a scientific conference that covers everything from multimedia content retrieval/understanding to content-based indexing and human-computer interaction issues. This year’s edition was originally intended to take place in Taipei (Taiwan). Registration info via service.icmr2021@gmail.com, details on the ACM ICRM program here.

We hope you get to enjoy IEEE ICIP or ACM ICRM or both. Stay tuned for more event notes.


Photo by Gabriel Benoit

Meet the XR4DRAMA Consortium (III): Nico Heise and Deutsche Welle (DW)

Deutsche Welle in Berlin (photo by DW) and Nico Heise (photo by Nico Heise)

In this series of blog interviews, we further introduce the people and organizations behind XR4DRAMA by asking them about their work and their particular interest in the project. Our third interview partner is Nico Heise of DW, Germany’s international broadcaster.

1) When did you first get in touch with the concept of situation awareness and XR technology?

Our team at Deutsche Welle, Research and Cooperation Projects, works on cutting edge technologies and explores how they might become relevant for media production or storytelling. We had a couple of projects in the past that dealt with extended reality. There is great potential and we have decided to pursue this path. The concept of situation awareness was introduced by Vanessa Oberin, a PHD student who worked in our team when we wrote the proposal. She contributed a lot of the theoretical background that drove our approach.

2) Who is in your team – and what are your colleagues working on at the moment?

The team that currently works on XR4DRAMA consists of Farina Hamann, Axel Primavesi, Alexander Plaum and myself. Farina is an expert on designing immersive interactions and has a lot of experience with immersive media in general. She will work on UI and UX design regarding the second pilot use case (media production planning). Axel is a Deutsche Welle journalist and producer who has successfully planned and carried out various feature productions for Deutsche Welle and other broadcasters. He will bring his experience in production planning to the project. Alexander has developed the project’s identity and dissemination strategy. He supervises the XR4DRAMA website and blog.

3) From your point of view: What are the most interesting aspects about XR4DRAMA?

In the past, we have worked on AR and VR solutions in media storytelling to create immersive experiences for end-users. In XR4DRAMA, it is fascinating to explore the potential of extended reality for the production process itself. We know from experience that pre-production planning is crucial for a successful media production. It will be very interesting to see whether the technologies that will be developed within the project will indeed increase situation awareness of production staff to a level that makes pre-production easier and more efficient.

4) What could be a challenge for the consortium?

Apart from the Covid pandemic that creates difficulties for all European projects, it will be challenging to really meet the requirements that the users have set out when the project started. Immersive experiences have the potential to create a feeling for a specific location – even from a remote position. However, many people are not (yet?) used to this kind of experience and might feel overwhelmed. This could ultimately compromise their judgement. It will be challenging to find the right mix of immersion and other forms of representation to avoid this.

5) When the project is over in late 2022, what kind of outcome do you expect?

We hope and strive for a tool that indeed increases situation awareness for production staff that is not on location. The Covid pandemic with all its travel restrictions has even increased the need for solutions that support remote planning (not only in media production). It would be great if the XR4DRAMA technologies could become part of this solution.

Dear Nico, thank you for your time!

Our next interview partner will be Yash Shekhawat of Nurogames.

Flood Management in Vicenza, World Heritage Documentation on Corfu: The XR4DRAMA Pilot Use Cases

The XR4DRAMA project is driven by two pilot use cases: The first one focuses on disaster management, the second one is about media production planning. The two user partners, Alto Adriatico Water Authority (AAWA, Italy) and Deutsche Welle (DW, Germany), have spent the first months of the project to specify the use cases and define user requirements.

The disaster management use case

The first use case focuses on a scenario where clear and comprehensive information is vital: the management of a (natural) disaster. AAWA is an institution that is – among other things – responsible for flood prevention, management, and mitigation.

In XR4DRAMA, the focus is on on Vicenza, a city located in north-eastern Italy. The city is situated at the confluence of no less than four rivers and has experienced severe flooding in the past. Due to this risk level, city officials and AAWA have already developed alert systems, advanced emergency action plans, and programs for civil protection volunteers. However, managing emergencies remains highly challenging, especially when it has to be done from a distant control room where there’s relatively little awareness of the current circumstances in the disaster area. AAWA’s main purpose in XR4DRAMA is therefore to set up a system that supports disaster managers in remote locatations with first responders in the field.

The pilot is split into two operational phases: The preparation for a potential crisis – and the time during the emergency.

The pre-emergency phase begins with an official warning that informs about an alarming rise of water levels. At this point, the staff inside the control room should be able to access various relevant information through the XR4DRAMA system. This information includes map data, satellite images as well as water level reports and delivers a comprehensive overview of the situation. This will allow the management team to initiate first measures to minimize the impact of the upcoming disaster.

During the emergency itself, the team in the control room depends on information from the field, either reported by first responders or by citizens on-site. This information will be communicated through an XR4DRAMA mobile app. It undergoes automated textual, visual and audio analysis to provide a new information layer in the XR4DRAMA system. This layer, in conjunction with AR and VR technology will improve situation awareness for disaster managers and help them make the right call from their positions. In addition, the control room staff will be able to assign tasks and monitor their execution in real time. The project will also use sensors that allow managers to continuously assess the physical status of first responders and potential risks to their wellbeing.

The main purpose of the XR4DRAMA system is therefore to support disaster managers who are working from remote positions and to connect them with the first responders in the field. The whole scenario is split into two operational phases: the preparation for a potential crisis and the time during the emergency.  

The pre-emergency phase begins with an official warning that informs about an alarming rise of water levels. At this point, the staff inside the control room should be able to access various relevant information through the XR4DRAMA system. This information includes map data, satellite images and water level reports and delivers a comprehensive overview of the situation. This will allow the management team to initiate first measures to minimise the fallout of the upcoming disaster. 

During the emergency itself, the team in the control room depends on information from the field, either reported by first responders or by citizens on-site. This information is supposed to be communicated through the XR4DRAMA mobile app. It undergoes automated textual, visual and audio analysis to provide a new information layer in the XR4DRAMA system. Together with this layer visualisations in virtual and augmented reality improve situation awareness for disaster managers and help them to make the right decisions consciously and efficiently even from their remote position. Additionally, the control room staff is able to assign tasks and to monitor their execution in real time. Furthermore, the project uses physiological and environmental sensors to allow the management to continuously assess the physical status of first responders and potential risks for their wellbeing.

The media production planning use case

The second use case isn’t concerned with matters of life and death – but that doesn’t make it less interesting. Media production management is a complex process that can always be improved. For example when it comes to planning and preparations – which can be quite difficult when managers have not visited and experienced a designated shooting location. Their lack of grasping the conditions on-site is likely to bring down production quality.

In order to find out how situation awareness, digital models, multimodal data and XR renderings can improve film making, DW and its partners will enlist the XR4DRAMA system in the production of a (fictitious) documentary about the Old Town of Corfu, a UNESCO World Heritage site since 2007.

Throughout the centuries, the island was ruled by various powers and often subject to unrest and war. Large fortifications and ancient ruins bear witness to this – and make perfect 3D models. The documentary intends to cover the changeful history of the island as well as the life of its inhabitants today.

The media production planning workflow will be split into three steps.

DW and the consortium will start by using the XR4DRAMA system to receive basic information about the shooting location. The results of these automated queries are then analysed, organised and presented on a screen.

In a second step, the production management assigns a location scout to further investigate. Their task is to verify, adjust and add information such as images, videos and audio recordings. Once processed by the system, this data will give the (remote) production staff a better overview and feeling for the location.

In a third and final phase the XR4DRAMA system will display more sophisticated immersive environments and also simulate camera movements, shot angles and various lighting conditions. The remote production management team will thus be able to make exceedingly informed decisions with regard to cinematography, logistics, and so on. In the end, XR4DRAMA should have a significant impact on saving resources, optimizing the productions and boosting the creative outcome.

Both in the case of disaster management and media production planning it will be interesting to see how professional users deal with digital models, extra data, and immersive environments – and how all of this improves situation awareness and the quality of their work. We’ll keep you posted!

  


Photos by
giem36 via Pixabay
Erroscia, CC BY-SA 3.0, via Wikimedia Commons
Martin Falbisoner, CC BY-SA 4.0, via Wikimedia Commons

Meet the XR4DRAMA Consortium (II): Sotiris Diplaris and the Centre for Research and Technology – Hellas (CERTH)

4 out 5 team members: Sotiris Diplaris (2nd from the left) with his colleagues Stefanos Vrochidis, Ioannis Kompatsiaris, and Anastasios Karakostas (from left to right)

In this series of blog interviews, we further introduce the people and organizations behind XR4DRAMA by asking them about their work and their particular interest in the project. Our second interview partner is Dr Sotiris Diplaris of ITI/CERTH, the Information Technology Institute at one of Greece’s (and Europe’s) leading research centres.

1) When did you first get in touch with the concept of situation awareness and XR technology?

It’s hard to say when exactly, but we’ve worked on SA- and XR-related projects for quite a while now. For instance, there was beAWARE, which focused on decision making in emergency situations and also featured AAWA as a partner in disaster management. beAWARE led to the idea of employing XR tech for real-time situation monitoring and collaboration between field teams and control room staff. But we’ve also had several talks with DW, who told us that media professionals would really benefit from having detailed, immersive models of a filming location – before going there with a production team. As for XR tech, we’ve explored that in projects like MindSpaces, SO-CLOSE, and V4Design (the latter also together with DW).

2) Who is in your team – and what are your colleagues working on at the moment?

Dr Stefanos Vrochidis does overall coordination. He’s the head of M4D; that’s the Multimodal Data Fusion and Analytics Group of the Multimedia Knowledge and Social Media Analytics Lab (MKLab) of ITI-CERTH. Stefanos leads a project management team that consists of three more people: Dr. Anastasios Karakostas supports the coordination of XR4DRAMA by supervising research activities and use cases.  Spyridon Symeonidis conducts the daily technical management and the implementation of the web data collection mechanisms. As for me, my duties are the scientific and technical management of XR4DRAMA. Finally, there’s Dr. Ioannis Kompatsiaris. As head of MKLab, he’s involved in the high level supervision of the project.

…and the team member missing in the group photo (hi there, Pandemic): Spyridon Symeonidis

3) From your point of view: What are the most interesting aspects about XR4DRAMA?

The most exciting part of this project is that we use several technologies to improve areas of work and life that are really important and depend on situation awareness and decision making. Hopefully, we’ll be able to show how realistic VR or semantically enriched AR representations of environments can improve the way professionals work together. The concepts and applications we’re developing will add a whole new dimension to workflows that has never been explored before. 

4) What could be a challenge for the consortium?

The main challenge lies in combining all kinds of data sources and technologies that usually belong to different research fields. We have multimedia web content, drones and satellites, and sensors that enrich our repositories with environmental and physiological information. There’s analysis on a text, audio, video and sensor level, and interlinked results created via semantic technologies. 3D reconstruction techniques generate the elements of immersive environments while geographic information systems (GIS) are used to properly place the content in 3D space. It’s easy to understand that it won’t be trivial to make optimal use of all these components. At the same time, our designated users will only benefit if we put it all together really well.

5) When the project is over in late 2022, what kind of outcome do you expect?

We hope that the end result of XR4DRAMA will be a complete set of XR software that has been evaluated in two real world use cases, namely disaster management and media production planning. The infrastructure we have will be versatile and suitable for other scenarios and domains. Front-end tools will include a desktop and a mobile application as well as an AR and VR environment. We’ll equip professionals with a powerful solution that makes their work easier and safer. And of course, depending on the use case, the impact of XR4DRAMA may go well beyond professionals in the field: For example in the event of a flood, better decision making and disaster management will have positive effects for all people in an affected community.

Dear Sotiris, thank you for you time!

Our next interview partner will be Nicolaus Heise of DW.

CEM, HCI, SAR? – A Brief Guide to the XR4DRAMA Alphabet Soup

While writing concepts, technical documentations and tweets, we’ve realized that the XR4DRAMA consortium uses a lot of abbreviations and acronyms, many of which aren’t necessarily known to a non-expert audience. And even within the team, nobody has heard of all of them, as it’s almost impossible to be an expert in disaster management and media planning and data processing and computer linguistics and smart clothes and immersive technologies.

So in order to everybody a good overview, we’ve compiled this handy glossary  – which also defines/explains some of the more exotic or difficult terms: 

  • AR: Augmented reality (a type of extended reality where a users see their “real” environment, but with digital overlays)
  • CEM: Certified emergency manager
  • COWM: Citizen Observatories for natural hazards and Water Management
  • DP: Disaster Preparedness (coordinated actions taken to prepare for disaster, prevent them, or mitigate their impact)
  • DRR: Disaster Risk Reduction (coordinated actions that aim to reduce the damage caused by natural hazards, via an ethic of prevention)
  • ECCA: European Climate Change Adaptation Conference
  • ECG: Electrocardiogram (the process of recording the electrical activity of the heart, usually to check for cardiac abnormalities)
  • EGU: European Geosciences Union
  • EO data: Earth observation data
  • GIS: geographic information system (a conceptualized framework that provides the ability to capture and analyze spatial and geographic data)
  • GNSS: Global navigation satellite system
  • H2020: Horizon 2020 (an EU program for research and technological development – and the funding source of XR4DRAMA)
  • HCI: Human-computer interface (basically any device that lets human interact with a machine, e.g. a keyboard, a touchscreen, a dataglove)
  • IA: Innovation action (a certain type of R&D project; XR4DRAMA is an IA)
  • ISCRAM community: an international community of people working in the field of Information Systems for Crisis Response and Management
  • IFAFRI: The International Forum to Advance First Responder Innovation
  • LIDAR: Light detection and ranging (also: laser imaging, detection, and ranging): a remote sensing method that uses light (a pulsed laser) to measure variable distances
  • MR: Mixed reality (like AR, but with interactive virtual objects anchored in the real world)
  • NLP/NLProc: Natural language processing (a subfield of computer science/artifical intelligence and linguistics that is focused on creating programs to handle and analyze large amounts of natural – i.e. human – languages.
  • NLU: Natural language understanding (a subfield of NLP/NLProc that is focused on creating program to comprehend what has been collected and processed)
  • PPE: Personal protective equipment
  • R&D: Research & development
  • SAR: Synthetic Aperture Radar (a type of radar used to create 2D or 3D reconstructions of landscapes and objects)
  • SVI mapping: Social vulnerability index mapping (efforts to visualize/map U.S. Census data that determines the social vulnerability of specific geographic regions)
  • VR: Virtual reality (a type of extended reality where users don’t see their “real environment”, but are rather fully immersed in a digital sphere)
  • WWS: Wearable wellness system (a body worn system designed to monitor all kinds of
    physiological parameters)
  • XR: Extended reality (AR + MR + VR + all other forms of immersive media)

Have we missed a term that’s important in the context of XR4DRAMA?
Send us an email: consortium@xr4drama.eu