Platform Latency of WebSocket and WebRTC

These are some notes about latency in my tilt-and-pan system.

The tilt-and-pan WebRTC Streaming system (link) relies on an underlying Network Architecture (link) which makes it all work over the internet.

There are two main network elements:

• WebSockets (routed from Pi through intermediate host to PC)
• WebRTC (direct from Pi to PC)

Architecture

Excerpted from a prior post is this diagram of the network setup:

Basic Premise

When moving the Cardboard viewer, orientation data in the phone is sent via WebSockets to the Pi, which operates motors which control the direction of the camera.

The camera video is constantly being streamed back to the phone.

There is latency between the actual movement of the phone and the corresponding video image appearing to move.

Scenarios to Investigate

To narrow down where there is latency, I decided to take measurements in two deployments:

• LAN
• Internet

The LAN deployment runs the RDVP Server on the Pi, and all comms are local LAN.  The idea being to eliminate public internet uncertainty from measurements.

The Internet deployment is more "real-world" use case for future projects.

In all cases the Pi remains on the LAN.

A note about the LAN

During testing, it became clear that the LAN should be further split into wired and wireless.

That is, connections to the local LAN can be on Ethernet or WiFi.

Ethernet does not have to contest with wireless congestion.  WiFi does.

For the tests below, the Pi is always on Ethernet, and the Laptop is tested on both Ethernet and 2.4GHz WiFi.  And the WiFi is congested as a result of many apartments nearby.

A note about the Internet

The RDVP Server lives on the internet.  But not just anywhere.

My apartment is located in NJ.  The RDVP Server is running in Canada right near Montreal.

So traffic via the internet will have to travel approx 320 miles in each direction (as the crow flies).  This will likely be a fair amount longer in actual network links.

Areas to Investigate

Network Latency - Ping times and general RTT
WebSocket Latency - WebSocket messaging over given Networks
WebRTC Latency - Attempt to measure video latency over given Networks


For the WebSocket Latency, I am measuring the RTT from a message sent from my Laptop to the Pi and back again to the Laptop.


For the WebRTC Latency, I'm pointing the webcam at a high-res clock on the computer and then comparing the received video image to the clock.

LAN - Network Latency

Here I am pinging from the Laptop to the PI on the local LAN.

	Pi Ethernet
Laptop Ethernet	sub-ms
Laptop WiFi (2.4GHz)	32ms (highs in hundreds and high hundreds)

Internet - Network Latency

Here I am simply pinging the intermediate RDVP Server.  Cannot ping the Pi directly of course.

	RDVP Server
Laptop Ethernet	22ms
Laptop WiFi (2.4GHz)	36ms

LAN - WebSocket Latency

	Pi Ethernet
Laptop Ethernet	12ms
Laptop WiFi (2.4GHz)	14ms

Internet - WebSocket Latency

	RDVP Server	Pi Ethernet via RDVP
Laptop Ethernet	27ms	53ms
Laptop WiFi (2.4GHz)	38ms	65ms
Laptop LTE Phone Tether (cable)	65ms	99ms

LAN - WebRTC Latency

	Pi Ethernet
Laptop Ethernet	516ms 527ms

Internet - WebRTC Latency

	Pi Ethernet
Laptop LTE Phone Tether (cable)	466ms 430ms

Observations

Firstly, WiFi congestion has a pretty dramatic effect.  It makes for extremely erratic RTT on ping and WebSocket traffic, and hugely increases the average latency.

Otherwise, if you consider 1/2 RTT to be the time the Pi receives a command, then the following tables show the latency there isn't so bad.

LAN - WebSocket Comms

	Ethernet	WiFi
RTT	12ms	14ms
½ RTT	6ms	7ms

Internet - WebSocket Comms

	Ethernet	WiFi	Cellular
RTT	27ms	38ms	99ms
½ RTT	14ms	19ms	50ms

The rest of the visual latency is due to the WebRTC Stream.

Observations about WebRTC

In short, I think the framerate is part of the root problem.  The other part is possibly software (UV4L) queuing on the Pi side.

The video stream, when looking at Chrome's internal WebRTC monitoring system (chrome://webrtc-internals), shows many stats about the connection.

Here is one of the stats about frame rate over time.  It's very low.  I've been able to up this number with various tweaks to upwards of 10 frames/sec.  Not much higher.



At 10 frames/sec, the duration between each frame is 100ms.

That is still well short of the total visual latency of the approx 500ms latency seen. This could be software queuing, as it doesn't appear to be bandwidth limitations or CPU issues on the Laptop side.

A future release (link) of UV4L claims to vastly improve the framerate and nearly eliminate latency. It will be worth measuring again once that is released.

Google Cardboard Controlling Raspberry Pi Streaming Tilt-and-Pan Streaming Video

Background

In a prior post (link), I demonstrated a setup which:

• A Raspberry Pi streams WebRTC video to anywhere on the internet.
• Can pan horizontally using an additional WebSocket control channel, controlled by a Chrome page which works on Desktop and Mobile.

I have extended that setup so that now:

• I can both tilt and pan (X and Y axis, using a different WebSocket for each)
• Uses a phone's orientation data to dictate the angle each axis should rotate to.
• Exposes a fullscreen WebRTC video which still relays the orientation data.
• Exposes a fullscreen Google Cardboard display mode which also still relays orientation data.

The basic network and communication setup didn't change, and and really only required an additional servo for the Y axis, as well as the incorporation of client-side javascript libraries.  The client in this case is Chrome for mobile (Android in my case).

Videos

External view:

Internal view:

Screenshots

Below I have a few pictures of the setup, some video of the setup in action, and an explanation of how I extended Client-side functionality.

My home-brew tilt-and-pan rig.  Just two servos control it, one for X, one for Y.


A close-up shot of the Y-axis servo.  Only the highest quality rubber bands and cardboard will do.



Here is a screenshot of the webpage control.  This happens to be on Desktop for screenshotting.
It's a prototyping page, and so a number of controls are exposed:

• The top is some interactive controls to connect and disconnect my WebRTC, and the two WebSocket control channels.
• The video below it is the WebRTC video stream applied to a <video> tag.
• The "Camera Control" box below it lets me switch between manual servo control and Orientation Sensor mode.  Full-screen when touched.
• The Cardboard View is another element (more on this later) which displays the split screen view required for viewing the video in Google Cardboard.  Full-screen when touched.

Here is the Cardboard View element full-screen on Chrome on my phone.  It's displaying the WebRTC video feed in a split-screen.
















And that view now in my Google Cardboard viewer (ViewMaster).


And a close-up of the Google Cardboard viewer once closed up and ready to look at.

Client-side functionality

There are a few client-side enhancements that needed to be made to make this possible:

• Reading the Orientation Sensor data
• Creating the Cardboard view

Reading the Orientation Sensor data

I had played with the Orientation API (link)  as seen around on the web in the past, but found it confusing.  Maybe I'm just not understanding it correctly.

The reason I found it confusing is because the Orientation API is returning Euler Angles (link), which don't directly give me the data I want.  And what it does give me I found difficult to turn into what I want.

What I really want is to get some readings which are more like what a person would intuitively think of as orientation.

That is, if I'm standing up, and I rotate in-place to the right, that could be expressed as a rotation in positive degrees.  Negative if I turn left.  Let's say that's the X axis.

If I look upward, or downward, that could also be expressed in terms of positive and negative degrees.  Let's say that's the Y axis.

In both cases, I have mapped the Orientation Sensor data to X and Y axis rotations to positive and negative degrees, ranging from 90 to -90 in either direction.  Useful since that's 180 degrees total, which is basically the range of a servo.

The software running on the Pi is a very simple program which expects to be told a number in the range 90 to -90, and map that to the appropriate PWM calls to move the servo to that location.

Getting the Orientation mapping to work was a challenging task for me and involved actually having to leverage some graphics libraries (Three.js) for some conversion away from Euler Angles (which are apparently used in graphics) and subsequently do some trig.  I'm probably doing it wrong.

Creating the Cardboard view

The Google Cardboard viewer requires a split-screen display with each image aligned to the center of the lens for each eye.

To accomplish this, I additionally use Three.js and its StereoEffect plugin.

The basic application in javascript is:

• Create a Three scene
• Create a Three renderer (WebGL)
• Wrap the renderer in the StereoEffect effect
• Create a HTML canvas element
• Create a Three texture from the canvas
• Create a 2d plane and apply a Three Mesh to it which has the canvas-backed texture mapped onto it
• When animating
• Get the current video frame from the WebRTC-backed <video> tag, and paint it onto the canvas.
• Indicate to the Three texture that the canvas data has changed.

This leads to a split-screen render where there is a 2D plane which appears to be a video playing in front of each eye.

Final Notes

There are several areas which need to be improved about the whole setup.

Latency

There is a bit of delay (sub-second) in movement of the Cardboard viewer and the actual perceived movement of the image in the video.

This is going to be a combination of several factors, such as:

• Pure network latency.  I'm in NJ, the Pi is in NJ, and I communicate to it via Montreal when using the Internet (as opposed to LAN-only).  That's a one-way 720 mile journey.
• On the other hand, even on the LAN the latency isn't much better.
• Latency on the Python program running on the Pi of actually processing messages and moving the servo.
• Latency of the video streaming software (UV4L) pushing out new frames and subsequently being received and displayed on the browser.

I will investigate this in a future post.  Latency is not a concern in this prototype but it'd be nice to see where it's all coming from.

The actual tilt-and-pan rig

It's pretty make-shift at the moment.  A co-worker pointed me at a properly-built tilt-and-pan rig for less than $9 on Deal Extreme (link).  I put in an order and will wait to see how it works.

Not a huge priority actually but probably more stable.  Plus the rig comes with two servos so not bad.