Project Creates Higher Quality Immersive Internet Videos Without Lag in Streaming
Of all the internet has brought, 360-degree videos that let users see around a fixed point are among the coolest. Seeing the entire setting, as if the user is standing there, is an immersive experience.
At its most basic level, a 360-degree video is data just like anything else sent over the wires and cell towers of the internet to our devices. But, a 360-degree video is a lot of data. In fact, it is difficult at the rate data travels over computer networks, called bandwidth, for the entire high-quality video to arrive perfectly at the outset.
As the user views the video sphere, there is often a lag between what comes from the internet and what the users wants to see.
In virtual reality and augmented reality formats, which hold the most exciting applications of 360-video, this lag can cause motion sickness as viewers’ brains and eyes move faster than the video can load, or stream, off the internet.
“It’s too slow because there is a lot of data that your network bandwidth cannot support,” said Dr. Jacob Chakareski, assistant professor of electrical and computer engineering at The University of Alabama.
In his lab, Chakareski is developing a method to smooth this hang up by predicting where users will view, sending a smaller, high-quality spatial segment of the 360-degree video panorama for the areas likely to be viewed and a low-quality version for the parts of the video unlikely to be navigated by the user.
“As you are exploring the content on a virtual reality, head-mounted display, we only send you the present 360 panorama segment of interest in high quality, and the rest is streamed in low quality, and that prevents motion sickness and enhances the quality of immersion experience for the user,” Chakareski said.
The work received the attention of Adobe, which gave Chakareski the Data Science Research award in fall 2017 and invited him to its upcoming Digital Experience Research Symposium at the company’s headquarters near Silicon Valley in September.
Currently, gaming, education and training, and entertainment are the main applications for virtual or augmented reality (VR/AR), but it is anticipated that other industries will adopt the technology. Environmental and weather sciences, disaster relief, transportation, search and rescue, and remote sensing for a range of applications are potential adopters.
Expanding the use of 360-video, though, will require overcoming the delay and insufficient network bandwidth when streaming to a VR headset, especially in the wireless setting, he said.
“The lag and inadequate streaming data rate cause this motion sickness and low visual quality, and this is the typical problem with VR,” Chakareski said. “To avoid the former, rather than sending only what you really need at a specific time, current practices send everything because it’s unpredictable where you will view next.”
Sending the whole video at once, though, does not work well because network bandwidth, which can vary widely depending on a host of factors, means it arrives piecemeal. This creates an excessive lag, even at multiple seconds in length, between what the user wants to view and what is on-screen. Also, the problem of insufficient network bandwidth is compounded because data is sent, but not used, he said.
Chakareski is creating intelligent methods to separate and stream the 360-degree video content, only sending high-quality video where the user looks at the moment and sending the rest of the video in a low-quality format. This is achieved via a rigorous analysis of the required data rate across the 360-degree video panorama to maximize the delivered video quality, while meeting the present network bandwidth limitations.
The area sent in high quality is what the user sees plus a border area around the view. Chakareski and his students built machine learning software that uses data from experiments with users viewing 360-degree videos on VR headsets. It can accurately predict where a user is likely to turn to view the video panorama next, sending only that spatial slice of the video in high quality.
The streaming of the video is dynamically adapting to the user’s head movements, and, even if a user strays from the prediction, a low-quality version of the remaining portions of the video panorama is always available. Since that low quality video uses less data, the navigation experience is more seamless for the user, reducing the chance for motion sickness from lag, according to Chakareski’s work.
In other words, it is better to see a slightly lower quality video immediately than for the mind to wait for a clear video to load.
“If the prediction is not correct, you will observe lower quality shortly, while still making sure that the streaming continues,” he said. “It will catch back up quickly once you settle and the system realizes the prediction was not correct.”
The method shows a 50 percent savings in data sent to the user and noticeable improvement in experienced visual quality, while also providing considerably reduction in lag, which further enhances the immersion quality of experience for the user.