Chasing Dropped Calls and a 3 fps Webcam

Title: Chasing dropped calls and a 3 fps webcam
Date: 21 September 2026
Author: Sivert Gullberg Hansen
Keywords: WebRTC, bandwidth estimation, keyframes, SFU, WebSocket, Cloudflare tunnel

Abstract

A webcam in our community sent 1080p at 3 to 7 frames a second on a 74 Mbps upload, and the app now and then lost its connection to every server at once. We rebuilt both in a test bed: a throwaway server and SFU with two headless Chrome guests and a relay that could cap the rate, plus a harness that held calls and connections up for four hours. The webcam came down to four bugs. The SFU asked for a keyframe every two seconds and never sent Chrome a round-trip time, the microphone was allowed 510 kbps, the app swapped the camera for itself four times a second, and a camera turned off and on took the screen share's slot. Through a 700 kbps cap the camera went from 4 and 8 fps to 29 and 30 fps once those were fixed. The drops had four causes: Cloudflare restarting its proxy, a reconnect that switched video off, our own releases, and a router that drops IPv6 every 5 hours 8 minutes. The fixes are in SFU 1.0.69 and apps 1.11.33 to 1.11.35. A few things are still open, listed in Section 5.2.

1 Introduction

Someone in our community had a webcam that wouldn't behave. It was sending 1080p at 3 to 7 frames a second, and Gryt's video debug overlay blamed bandwidth. Their upload was 74 Mbps, and Discord had no trouble with the same webcam.

Around the same time, the app would now and then lose its connection to every server at once. Nothing in the logs said why.

Neither turned out to be one bug. This post answers two questions:

1. Why did the camera get so little bandwidth on a line that fast?
2. Why did connections drop, and which drops can Gryt do something about?

2 Background

The overlay's Avail out number is Chrome's own estimate of how much it can safely send right now. On the bad sessions it sat at about 559 kbps, and a 1080p camera doesn't fit in that. So the question was why Chrome's estimate stayed that low.

Our SFU is the server that passes everyone's audio and video along. Chrome builds its estimate partly from feedback the SFU sends back. The first guess was that the SFU never sent that feedback. It turned out it sends about ten feedback packets a second, so the cause was somewhere else.

A keyframe is a whole picture. The frames between keyframes only carry what changed, so a keyframe is a lot bigger. At 559 kbps, a 1080p keyframe takes seconds to send.

3 Method

3.1 The video test bed

I ran a throwaway server and SFU, with two guests in a voice channel, each in headless Chrome. The camera was a 1080p video file with noise in it, noisier than a real camera would be. Between the camera and the SFU sat a relay. It could add delay, jitter and packet loss, cap the rate, and count what came back. Both guests read Chrome's stats once a second.

To reproduce the stuck estimate, I capped the camera's upload at 700 kbps for the first 20 seconds, with about 60 ms of round-trip time, then lifted the cap. As a control, Chrome also sent straight to Chrome through the same 700 kbps cap, without Gryt in between.

3.2 The soak harness

For the drops I wrote a harness [5] that keeps calls and connections up for hours and writes down everything that happens to them. It runs headless browsers with a fake camera and microphone, sitting in a call. Next to them are probes: connections to the server and the SFU, from Node and from a browser page, and a health check every 10 seconds. It logs all of it with timestamps. Afterwards, each drop gets lined up with what the server, the SFU and the Cloudflare tunnel logged in the same minute.

The probes went three ways at once: through a Cloudflare tunnel to a test server, straight to the same server over the local network, and to a server on the same machine as the harness. The calls ran on that last one. The main run lasted four hours.

4 Results

4.1 Too many keyframes, and no round-trip time

The SFU asked every camera and screen share for a keyframe every two seconds, on a timer of its own. In a two-minute test call the camera got 60 of those requests, and the person watching had asked for one. The stats from the webcam we were chasing fit that too: 21 requests in 238 frames, or one every 1.6 to 3.8 seconds.

The SFU also reports back to the sender on what arrived. Those reports never had a round-trip time in them, because the SFU never read the sender's own reports. So for the whole call, Chrome got no round-trip time from us at all.

Table 1 shows the capped test from Section 3.1 with and without the fixes.

Table 1: Capped at 700 kbps for 20 seconds, then uncapped.

SFU	Runs that got stuck	Estimate after the cap	Camera
Old	2 of 3	Stayed at exactly 548 kbps for the rest of the call	30 kbps floor, 4 to 5 fps at 480x270, nearly every frame a keyframe
Either fix, or both	None	Passed 2 Mbps within two seconds	Up to 2.5 Mbps, 30 fps, 1080p

The SFU now only passes on the keyframe requests from people watching, and its reports carry a round-trip time [1]. Dropping the timer didn't slow down people who join in the middle of a stream, because their own request gets passed on. Their first frame came 252 to 303 ms after connecting, against 217 to 430 ms with the timer. Both changes shipped in SFU 1.0.69.

4.2 The microphone got most of a small budget

Unless a channel set its own bitrate, the app allowed the microphone up to 510 kbps, the most the Opus audio codec can do. Chrome splits its estimate between everything you send, and it gave the microphone all it was allowed. With a 548 kbps estimate, the microphone got 510 and the camera 30. The microphone only used 74 to 84 kbps of it.

The default is 64 kbps now [2]. A channel's own bitrate still wins, and eSports mode stays at 128.

4.3 The camera was swapped for itself four times a second

This one was a plain bug in the app. It re-added your camera to the call over and over: 485 times in a two-minute call, about four times a second. Each time, it swapped the camera out for itself.

Every swap cost a keyframe. It also reset how Chrome scales the camera. So on a thin link, Chrome dropped frames instead of lowering the resolution. In the control from Section 3.1, the camera ran at 640x360 and 31 fps. Swapping it for itself every 250 ms held it at 1920x1080 and 7 fps.

Table 2 has the app through the same 700 kbps cap, twice without the microphone fix and this one, and twice with both.

Table 2: The app through a 700 kbps cap, two runs each.

	Frame rate	Resolution	Keyframes
Before	4 and 8 fps	480x270	55% and 99%
After	29 and 30 fps	640x360 to 960x540	2%

That last 2% came from the SFU's two-second timer. On a good connection, the camera now starts at a lower resolution and Chrome raises it from there. It used to go out at 1080p straight away, because the swaps kept Chrome from scaling it. Both fixes are in app 1.11.33 [2].

4.4 Turning the camera off and on broke screen sharing

This turned up while I was testing something else. If you turned your camera on, off and on again, then shared your screen, everyone else was told your screen wasn't coming through. If you shared first and then turned the camera off and on, the camera never came back.

The SFU gives each connection four slots to receive on: microphone, camera, screen and screen audio. Turning the camera off took it out of its slot. And a slot that's been used once can't be used again. So when the camera came back, it took the next free video slot. That was the screen's, so a screen share had nowhere to go.

Now the camera pauses in its slot and picks up there again [3]. That's in 1.11.33 too.

4.5 The drops

Table 3 has the four-hour run from Section 3.2.

Table 3: Drops in the four-hour run, by path.

Path	Drops
Through the Cloudflare tunnel	6
Straight to the server over the local network	3
On the harness's own machine	0

The calls on the harness's machine lost no audio packets. The drops came from four causes and two I can't fully explain.

Cloudflare restarting its proxy ended the call. Twice, Cloudflare closed an SFU connection with code 1001, "CloudFlare WebSocket proxy restarting". A probe just reconnected, in two or three seconds. A call didn't. The app read a clean 1001 as the call ending on purpose, so it never reconnected. Meanwhile the SFU dropped the person and told the server, and eleven seconds later the server took them out of voice. When a connection just died, with no close code, the app did reconnect, and the call was back in 1.6 seconds. Our SFU never closes a connection with 1001 itself, so a 1001 always means something in between went away. Now only a normal close ends the call. A 1001 reconnects like any other drop, and the call is back in 1.6 to 2 seconds without anyone leaving it. The fix is in app 1.11.34 [6].

A reconnected call came back with audio only. When the app reconnected a call on its own, every camera and screen share had been switched off, and everyone's tiles waited for video that never came. The app turned your camera off whenever the call wasn't connected, and a call also counts as not connected for the second or two a reconnect takes. Now the camera and screen share only stop once the call has really ended. With the fix, both test guests saw both videos again 3.7 seconds after the drop. It's fixed in 1.11.35 [7].

Our own releases. Two of the drops were our own deploys. Connections drop and are back within 3 to 8 seconds. Our deploys only replace the SFU when nobody's in a call on it. A call carries on through a server restart, and in a local test the audio and video never stopped. The video tiles do go blank for about six seconds while the server connection is down, though.

My router drops IPv6 every 5 hours 8 minutes. One drop hit only the probes going through the tunnel. My router at home drops its IPv6 route on a fixed timer, every 5 hours, 8 minutes and 34 seconds, and puts it back about two and a half seconds later. Everything in the house talks to Cloudflare over IPv6 when it can, so every Gryt connection from the house dies at that moment. The tunnel talks to Cloudflare over IPv4, so it didn't notice, and cloudflared logged nothing. The apps noticed at their next ping timeout, 16 seconds later, and were back in under 2 seconds. The same timer lines up with earlier drops in our servers' logs on 16 and 20 September.

Unexplained. In one drop, both ends lost the connection within 18 ms of each other, so it was closed somewhere between them, in Cloudflare or on the tunnel. It was back in 0.7 seconds, and the app doesn't show a drop that short. The other was my machine losing its path to the server over the local network, and no real client connects that way.

5 Discussion

5.1 If you host behind a Cloudflare tunnel

• Keep the app updated. 1.11.34 stops a Cloudflare restart from ending the call, and 1.11.35 brings cameras and screen shares back after a reconnect. The desktop app updates itself.
• Update the server to 1.10.19 and the SFU to 1.0.69. With Docker Compose that's docker compose pull and then docker compose up -d, as the docs say [4]. The Windows and Linux 1.10.19 bundles come with the new SFU, and if you host from the desktop app, 1.11.33 has both.
• If UDP isn't reliable where cloudflared runs, think about running it over HTTP/2. Twice in two nights our whole tunnel went down, for 141 and then 103 seconds. Both times our internet connection had lost UDP but still carried HTTPS. Our connectors were on QUIC, which runs over UDP, and they didn't switch to HTTP/2 by themselves either time. HTTP/2 runs over TCP, so it would probably have stayed up. Set TUNNEL_TRANSPORT_PROTOCOL=http2 in cloudflared's environment, or pass --protocol http2. Ours have run that way since 21 September, so it's too early to say it fixed it.
• If all your connections from one place drop at the same moment every few hours, check whether your router is dropping IPv6. Gryt can't keep those connections, because the network under them has gone.

5.2 Limitations and what's still open

• The calls ran on the harness's own machine. Calls through the tunnel haven't had their four hours yet, because the harness had no invite for the test server.
• Video tiles go blank for about six seconds while the server connection is down, even though the call carries on.
• The server logs nothing when it stops for a release, so a release looks like a network drop in the logs.
• Whether a camera should start at a higher resolution on a good connection.

6 Conclusion

Why did the camera get so little bandwidth? Chrome's estimate got stuck low because the SFU never sent it a round-trip time and kept asking for keyframes. The microphone then took 510 kbps of what was left, and the app swapping the camera for itself four times a second kept Chrome from lowering the resolution instead of the frame rate. With all of that fixed, the capped camera went from 4 and 8 fps to 29 and 30.

Why did connections drop? Cloudflare restarting its proxy, our own releases, and a router dropping IPv6 on a timer. Gryt can't prevent those. What it can do is recover: a 1001 now reconnects instead of ending the call, and video comes back after a reconnect.

References

1. sfu#47: the keyframe timer and the round-trip time.
2. voice#64 and client#626: the microphone's ceiling, and the camera swapped for itself.
3. voice#63 and client#625: the camera taking the screen share's slot.
4. Gryt docs. Upgrading with Docker Compose.
5. client#633: the soak harness, in the client repository under e2e/soak.
6. voice#65: Cloudflare's 1001.
7. client#635: video after a reconnect.

The pull requests have the full measurements.