Learning WebRTC From the Ground Up

I started with the hard part

Most people would tell you to build the easy stuff first, get the UI working, set up authentication, design the chat system, and bolt on voice later. I did the opposite. When I decided I wanted to build a communication platform, WebRTC was the first thing I tackled.

The reasoning was simple: voice is what makes or breaks a platform like this. You can have the most beautiful UI in the world, the most polished text chat and the best role system, but if your voice doesn't work reliably, none of it matters. I'd seen enough open source projects build fantastic features around text and community, only to struggle when it came to real-time voice communication. I didn't want that to be Gryt's story. If I couldn't solve voice, there was no point building the rest.

The pragmatic move would have been to grab a WebRTC-as-a-service SDK, wire it up, and call it done. Plenty of good ones exist. But I had a nagging feeling that if I was going to build a voice platform I could stand behind, I needed to understand what was actually happening on the wire.

That decision turned what could have been a weekend integration into a nearly two-year deep dive. This post is about what I learned along the way.

Nobody around me could help

One of the first things I did when I committed to learning WebRTC was ask around. I work as a developer, surrounded by experienced colleagues, so surely someone had dealt with this before? I asked every single developer at my workplace. Not a single person had any real experience with WebRTC or could tell me how it worked. A few who had worked at NRK, Norway's public broadcaster, and had been involved in live streaming infrastructure had heard of it, but when I pressed for details, it was clear they had zero idea how the protocol actually worked under the hood.

That was a sobering moment. WebRTC powers every video call on the modern web, including Google Meet, Discord and Zoom in the browser, and yet among dozens of professional developers, nobody could explain how ICE negotiation works or what DTLS-SRTP actually does. A lot of people use it. Very few seem to know what it is doing. There were no mentors to lean on, no colleagues to pair with. It was going to be me, the RFCs, and a lot of trial and error.

The WebRTC stack is deeper than you think

Most tutorials show you the "happy path": create an RTCPeerConnection, exchange SDP offers and answers through a signaling server, and audio starts flowing. It looks simple. It is not.

Beneath that API sits a tower of protocols:

• ICE (Interactive Connectivity Establishment) figures out how two peers can actually reach each other through NATs and firewalls
• STUN helps peers discover their public-facing IP addresses
• TURN relays media when direct connections fail, which is more common than you'd hope
• DTLS (Datagram TLS) establishes an encrypted channel over UDP
• SRTP (Secure Real-time Transport Protocol) carries the actual encrypted media packets
• SDP (Session Description Protocol) is the text-based format describing what each peer supports

Each of these has its own RFC, its own edge cases, and its own failure modes. Understanding them changed how I thought about the entire system.

ICE: the hardest "simple" problem

ICE negotiation is where most WebRTC implementations silently break. The protocol gathers network candidates (host, server-reflexive, relay), prioritizes them, and runs connectivity checks to find the best path. Sounds straightforward until you encounter:

• Symmetric NATs that defeat STUN and require TURN fallback
• Candidates trickling in after the initial offer/answer exchange
• Network changes mid-call (Wi-Fi to mobile data)
• Firewall rules that block UDP entirely, forcing TCP or TURN-over-TLS

I spent weeks debugging ICE failures that only happened on specific ISPs or corporate networks. The lesson: your local dev environment will always connect perfectly. The real world is a different story.

Security: DTLS-SRTP and why it matters

One of the things that drew me deeper into WebRTC was the security model. Unlike most web protocols, WebRTC encrypts media end-to-end by default using DTLS-SRTP. This isn't optional. Browsers won't transmit unencrypted media.

The handshake works like this:

1. During ICE connectivity checks, peers establish a UDP path
2. DTLS runs over that UDP path to exchange keys
3. Those keys derive SRTP encryption parameters
4. All subsequent audio/video packets are SRTP-encrypted

What makes this interesting from a security perspective is the fingerprint verification. Each peer includes a DTLS certificate fingerprint in their SDP. If you control the signaling server, you can verify these fingerprints match, creating a chain of trust from signaling to media.

For Gryt, this was critical. I wanted users who self-host to have confidence that their voice data is encrypted in transit, even when passing through the SFU. The SFU can route packets without decrypting them (in forwarding mode), preserving the encryption guarantees.

From peer-to-peer to an SFU

The first version of Gryt was pure peer-to-peer: every participant sent their audio to every other participant. This works fine for 2-3 people but falls apart quickly:

• With N participants, each person uploads N-1 audio streams
• CPU and bandwidth scale quadratically
• Any single weak link degrades everyone's experience

The solution is a Selective Forwarding Unit (SFU). Instead of sending to every peer, each participant sends one stream to the SFU, and the SFU forwards it to everyone else. Upload is constant regardless of room size.

My first attempt at building an SFU was with Mediasoup, a well-regarded C++ SFU with Node.js bindings. Mediasoup is a serious, production-grade library. Projects like Spacebar and Sharkord use it for their voice systems, and for good reason. But for me, at the stage I was at, it was too much abstraction over concepts I didn't yet understand. The C++ core was opaque, the abstractions assumed knowledge I didn't have, and when things went wrong I couldn't trace why. Mediasoup is an excellent library. I was struggling because I didn't understand the protocols underneath it well enough to use it effectively.

That realization is what led me to Pion WebRTC. Pion is a pure Go implementation of the WebRTC stack, with no C dependencies, no CGo, nothing hidden. Paradoxically, going lower-level made everything easier for me, because I could finally see what was actually happening:

• Read and step through the actual protocol implementation
• Understand exactly how DTLS handshakes, SRTP encryption, and RTP packet routing work
• Customize behavior at a level that wouldn't be possible with opaque SDKs

Building on Pion was one of the best learning decisions I made. When something broke, I could trace it from the application layer all the way down to individual RTP packets. The detour through Mediasoup taught me what I didn't know, which pointed me toward what I needed to learn.

What I'd tell someone starting today

If you're thinking about building something with WebRTC, here's what I wish I'd known:

Start with the signaling server

Get SDP exchange working reliably before worrying about anything else. Use WebSockets (Socket.IO is fine). Make sure you handle the offer/answer dance correctly, including ICE candidate trickling.

Set up TURN early

Don't assume direct connections will work. Run a TURN server (coturn is the standard) from day one. You'll save yourself weeks of debugging "it works on my machine" issues.

Read the RFCs (selectively)

You don't need to memorize RFC 8445 (ICE), but reading the overview sections of the key RFCs gives you a mental model that no tutorial provides. The WebRTC for the Curious book is an excellent starting point.

Log everything

WebRTC failures are notoriously hard to diagnose. Log ICE states, DTLS handshake progress, and connection state changes. Chrome's chrome://webrtc-internals is invaluable for client-side debugging.

Understand the security model

Don't just trust that "it's encrypted." Understand how DTLS-SRTP works, what the fingerprint in the SDP means, and what an SFU can and can't see. If you're building something people rely on for private communication, you owe them that understanding.

The payoff

Two years is a long time to spend learning a protocol. There were plenty of moments where I questioned whether it was worth it, and whether I should have just used an off-the-shelf SFU and spent those years building features instead.

But that investment shows up everywhere in Gryt today:

• The SFU handles edge cases gracefully because I understand the underlying state machines
• Security decisions are deliberate, rather than copied from examples
• When something breaks in production, I can diagnose it from packet captures instead of guessing
• I can explain to users exactly what happens to their audio data and why they should trust it

Looking at the broader landscape of open source communication platforms, I'm especially grateful I made this choice. There are brilliant projects out there, like Stoat, Spacebar and Element, doing strong work on community features, UI polish, federation, and encryption. Some of them have been at it longer than I have, and they've built things I genuinely admire. But real-time voice communication remains one of the hardest problems in this space, and it's where many projects hit a wall. By starting with the hardest problem first, I made sure Gryt's foundation was solid before building everything else on top of it. Doing it the other way around, features first and voice later, might have left me with a polished platform that couldn't reliably carry a conversation.

Surface-level WebRTC knowledge is enough for a demo. I wanted enough to ship something people would rely on, and that took the longer path.

---

If you want to dig into the Gryt SFU implementation or have questions about WebRTC, come chat on Discord or open an issue on GitHub.