Build WebRTC stream viewer¶

This article provides guidance on creating a minimal WebRTC client to access and display a video stream from Robovision AI cameras.

Overview¶

To display the camera stream, we'll:

Establish a WebSocket connection to the signaling server.
Create a WebRTC peer connection (RTCPeerConnection).
Handle signaling messages (SDP offers/answers and ICE candidates).
Receive the media stream and display it in a video element.

Key WebRTC concepts¶

Before diving into the code, let's briefly explain the essential WebRTC concepts involved:

WebRTC (Web Real-Time Communication): An open-source technology that enables real-time peer-to-peer audio, video, and data communication between web browsers without plugins.
Signaling server: Facilitates the exchange of connection information (SDP and ICE candidates) between peers.
SDP (Session Description Protocol): Describes multimedia sessions for establishing connections.
- SDP offer: Is sent by a peer to initiate a connection.
- SDP answer: The response to an SDP offer.
ICE (Interactive Connectivity Establishment): Helps peers discover network paths for communication.
- ICE candidates: Possible network addresses (IP and port) that can be used to establish the connection.
STUN/TURN servers: For NAT traversal, STUN servers help discover the public IP addresses, and TURN servers relay media when direct peer-to-peer communication isn't possible. In the following example, we're using a public Google STUN server.
RTCPeerConnection: The primary WebRTC interface for managing peer-to-peer connections.
Media streams and tracks: Represents the audio and video data transmitted between peers.

Prerequisites¶

WebRTC must be enabled during Robovision AI installation.
You must have an active recording session or camera inference to view the stream.

Step-by-step implementation¶

Add the following to your site’s Javascript or make a new script.js page.

1. Set up the WebSocket connection¶

Establish a WebSocket connection to the signaling server provided by your deployment. Replace the path.

const SOCKET_URL = "wss://deploymentname.clustername.example.com/webrtc"; 

// Create a WebSocket connection to the signaling server 
const socketChannel = new WebSocket(SOCKET_URL); 

// Handle WebSocket connection errors 
socketChannel.onerror = (e) => { 
    console.error("Failed to connect to socket:", e); 
};

2. Create the RTCPeerConnection¶

Initialize a new RTCPeerConnection with the appropriate configuration.

// Configuration for the RTCPeerConnection 
const WEBRTC_CONFIG = { 
    iceServers: [
        { 
            urls: "stun:stun.l.google.com:19302" 
        }
    ] // STUN server for NAT traversal 
}; 

// Create the RTCPeerConnection 
const peerConnection = new RTCPeerConnection(WEBRTC_CONFIG);

3. Handle signaling messages¶

Listen for messages from the signaling server and handle them accordingly.

In this example, it will select the first available stream.

// Helper function to send messages to the signaling server 
const sendSignalingMessage = (data) => 
    socketChannel.send(JSON.stringify(data)); 

// Handle incoming messages from the signaling server 
socketChannel.onmessage = async (event) => { 
    const msg = JSON.parse(event.data); 

    switch (msg.type) { case "welcome": 
        // Set our peer status to "listener" 
        sendSignalingMessage({ 
            type: "setPeerStatus", 
            roles: ["listener"] 
        }); 
        break; 

        case "peerStatusChanged": 
            // Request the list of available producers (camera streams) 
            sendSignalingMessage({ type: "list" }); 
            break; 

        case "list":
            // Get the ID of the first available producer 
            const peerId = msg.producers?.[0]?.id; 
            if (peerId) { 
                // Start a session with the selected producer 
                sendSignalingMessage({ 
                    type: "startSession", 
                    peerId }
                ); 
            } 
            break; 

        case "sessionStarted": 
            // Store the session ID for future communication 
            const sessionId = msg.sessionId; break; case "peer": 
            if (msg.sdp) { 
                // Handle SDP offer 
                await handleOffer(msg.sdp, sessionId); 
            } else if (msg.ice) { 
                // Handle ICE candidate 
                const candidate = new RTCIceCandidate(msg.ice); 
                await peerConnection.addIceCandidate(candidate); 
            } 
            break; 

        case "error": 
            console.error("Signaling error:", msg); 
            break; 

        default: 
            console.warn("Unknown message type:", msg.type); 
    } 
};

4. Handle the SDP offer¶

When an SDP offer is received, set it as the remote description and create an SDP answer.

async function handleOffer(offer, sessionId) { 
    // Set the remote description with the received SDP offer 
    await peerConnection.setRemoteDescription(new RTCSessionDescription(offer)); 

    // Create an SDP answer 
    const answer = await peerConnection.createAnswer(); 

    // Set the local description with the SDP answer 
    await peerConnection.setLocalDescription(answer); 

    // Send the SDP answer back to the signaling server 
    sendSignalingMessage({ 
        type: "peer", 
        sessionId, 
        sdp: peerConnection.localDescription.toJSON() 
    }); 
}

5. Handle ICE candidates¶

Exchange ICE candidates to establish the best possible connection path.

// Send local ICE candidates to the signaling server 
peerConnection.onicecandidate = (event) => { 
    if (event.candidate) { 
        sendSignalingMessage({ 
            type: "peer", 
            sessionId, 
            ice: event.candidate.toJSON() 
        }); 
    } 
};

6. Receive and display the media stream¶

Attach the received media stream to a video element to display it.

// Handle incoming media streams 
peerConnection.ontrack = (event) => { 
    const [stream] = event.streams; 

    if (stream) { 
        const videoElement = document.getElementById("videoStream"); 

        if (videoElement) { 
            videoElement.srcObject = stream; 

            videoElement.play().catch((err) => { 
                console.error("Error playing video:", err); 
            }); 
        } 
    } 
};

7. Putting it all together¶

Here's the complete script.js file:

// Replace with your deployment's WebSocket URL
const SOCKET_URL = "wss://deploymentname.clustername.example.com/webrtc";

// Configuration for the RTCPeerConnection
const WEBRTC_CONFIG = {
    iceServers: [{ urls: "stun:stun.l.google.com:19302" }]
};

const socketChannel = new WebSocket(SOCKET_URL);
const peerConnection = new RTCPeerConnection(WEBRTC_CONFIG);
let sessionId;

// Helper function to send messages to the signaling server
const sendSignalingMessage = (data) => socketChannel.send(JSON.stringify(data));

socketChannel.onerror = (e) => {
    console.error("Failed to connect to socket:", e);
};

socketChannel.onmessage = async (event) => {
    const msg = JSON.parse(event.data);

    switch (msg.type) {
        case "welcome":
            sendSignalingMessage({ type: "setPeerStatus", roles: ["listener"] });
            break;
        case "peerStatusChanged":
            sendSignalingMessage({ type: "list" });
            break;
        case "list":
            const peerId = msg.producers?.[0]?.id;
            if (peerId) {
                sendSignalingMessage({ type: "startSession", peerId });
            }
            break;
        case "sessionStarted":
            sessionId = msg.sessionId;
            break;
        case "peer":
            if (msg.sdp) {
                await handleOffer(msg.sdp, sessionId);
            } else if (msg.ice) {
                const candidate = new RTCIceCandidate(msg.ice);
                await peerConnection.addIceCandidate(candidate);
            }
            break;
        case "error":
            console.error("Signaling error:", msg);
            break;
        default:
            console.warn("Unknown message type:", msg.type);
    }
};

async function handleOffer(offer, sessionId) {
    await peerConnection.setRemoteDescription(new RTCSessionDescription(offer));
    const answer = await peerConnection.createAnswer();
    await peerConnection.setLocalDescription(answer);
    sendSignalingMessage({
        type: "peer",
        sessionId,
        sdp: peerConnection.localDescription.toJSON()
    });
}

peerConnection.onicecandidate = (event) => {
    if (event.candidate) {
        sendSignalingMessage({
            type: "peer",
            sessionId,
            ice: event.candidate.toJSON()
        });
    }
};

peerConnection.ontrack = (event) => {
    const [stream] = event.streams;
    if (stream) {
        const videoElement = document.getElementById("videoStream");
        if (videoElement) {
            videoElement.srcObject = stream;
            videoElement.play().catch((err) => {
                console.error("Error playing video:", err);
            });
        }
    }
};

8. A simple HTML page¶

A simple HTML page with a video element to display the stream. The WebRTC stream is automatically rescaled to a resolution of 1280 × 720 pixels (720p).

<!DOCTYPE html>
<html>
    <head>
        <title>WebRTC Stream Viewer</title>
    </head>
    <body>
        <!-- Video element to display the stream -->
        <video id="videoStream" autoplay playsinline controls></video>
        <!-- Include your JavaScript code -->
        <script src="script.js"></script>
    </body>
</html>

Diagram¶

sequenceDiagram
    participant C as Peer 1 - stream Consumer
    participant SS as Signaling Server
    participant P as Peer 2 - stream Producer
    participant ST as STUN Server

    C->>SS: WebSocket Connect
    SS-->>C: welcome

    C->>SS: setPeerStatus(roles=["listener"])
    C->>SS: list
    SS-->>C: list(producers=[{id: "camera1"}])

    C->>SS: startSession(peerId="camera1")
    SS->>P: notifySessionStart(peerId="camera1")
    SS-->>C: sessionStarted(sessionId="1234", peerId="camera1")
    P->>P: createOffer()
    P->>P: setLocalDescription(offer)
    P->>SS: peer(sessionId="1234", offer sdp)
    SS-->>C: peer(sessionId="1234", offer sdp)

    C->>C: setRemoteDescription(offer sdp)
    C->>C: createAnswer()
    C->>C: setLocalDescription(answer)
    C->>SS: peer(sessionId="1234", answer sdp)
    SS->>P: peer(sessionId="1234", answer sdp)
    P->>P: setRemoteDescription(answer sdp)

    loop ICE Candidate Exchange
        P->>ST: Gather ICE Candidates
        C->>ST: Gather ICE Candidates
        P->>SS: peer(sessionId="1234", ice)
        SS->>C: peer(sessionId="1234", ice)
        C->>SS: peer(sessionId="1234", ice)
        SS->>P: peer(sessionId="1234", ice)
    end

    C->>P: Media Stream Setup
    P->>C: Send Media Stream
    C->>C: ontrack(event)

Troubleshooting¶

Issue: WebRTC stream fails to start

The WebRTC stream may fail to start due to issues with the firewall on Windows. Specifically, the firewall blocks the first ICE candidate exchange, causing the initial connection attempt to fail.

Solution: Configure the firewall

Properly configure the firewall settings to allow WebRTC connections. Ensure that UDP traffic on the STUN server ports is allowed. The most common STUN server ports are:
- 3478 (port for the internal Robovision AI STUN server)
- 5349
- 19302
Alternatively, disable the firewall completely if it is safe to do so in your environment.