Thanks for the award, kixelated. xD
webrtc has a lot of annoying setup. but after it connects it offers low latency. how do you feel MoQ compares after the connection setup is completed? any advantages/ any issues?
As for the media pipeline, there's no latency on the transmission side and the receiver can choose the latency. You literally have to build your own jitter buffer and choose when to render individual frames.
How close are we to having QUIC actually usable in browsers (meaning both browsers and infrastructure supports it, and it 'just works')
How does QUIC get around the NAT problem? WebRTC requires STUN/TURN to get through full cone NAT, particularly the latter is problematic, since it requires a bunch of infra to run.
Cloud services are pretty TCP/HTTP centric which can be annoying. Any provider that gives you UDP support can be used with QUIC, but you're in charge of certificates and load balancing.
QUIC is client->server so NATs are not a problem; 1 RTT to establish a connection. Iroh is an attempt at P2P QUIC using similar techniques to WebRTC but I don't think browser support will be a thing.
I immediately jumped ship to WebTransport when Chrome added support. But I suppose there's no other option if you need P2P support in the browser.
As for the NAT problem, that's mainly an issue for peer-to-peer scenarios. If you have a publicly addressable server at one end, you don't need all of the complications of a full ICE stack, even for WebRTC. For cases where you do need TURN (e.g. for WebRTC with clients that may be on networks where UDP is completely blocked), you can use hosted services, see https://iceperf.com/ for some options.
And as for MoQ - the main thing it requires from browsers is a WebTransport implementation. Chrome and Firefox already have support and Safari has started shipping an early version behind a feature flag. To make everything "just work" we'll need to finish some "streaming format" standards, but the good news is that you don't need to wait for that to be standardized if you control the original publisher and the end subscriber - you can make up your own and the fan out infrastructure in the middle (like the MoQ relay network we've deployed) doesn't care at all what you do at that layer.
Unfortunately the NAT problem is more common than you think :( Lot's of corporate networks use full cone NAT (I know ours does), and so does AWS (if you don't have a public IP, but go through igw), so some sort of NAT punchthrough seems to be necessary for WebRTC.
I wonder if WebTransport has its own solution to the problem.
But I guess you can always rely on turn - by the way, does MoQ have some sort of ICE negotiation mechanism, or do we need to build that on top?
It is likely that I am missing this due to not being super familiar with these technologies, but how does this prevent desync between audio and video if there are lost packets on, for the example, the audio track, but the video track isn't blocked and keeps on playing?
Essentially though, there are typically some small jitter buffers at the receiver and the player knows how draw from those buffers, syncing audio and video. Someone who works more on the player side could probably go into a lot more interesting detail about approaches to doing that, especially at low latencies. I know it can also get complicated with hardware details of how long it takes an audio sample vs. a video frame to actually be reproduced once the application sinks it into the playback queue.
`relay.moq.dev` currently uses GeoDNS to route to the closest edge. I'd like to use anycast like Cloudflare (and QUIC's preferred_address), but cloud offerings for anycast + UDP are limited.
The relays nodes currently form a mesh network and gossip origins between themselves. I used to work at Twitch on the CDN team so I'd like to eventually add tiers, but it's overkill with near zero users.
The moq-relay and terraform code is all open source if you're super curious.
But I can at least say that we use anycast to route to a network-proximal colo.
Does your team have any concrete plans to reduce the TCP vs. QUIC diff with respect to goodput [1]? The linked paper claims seeing up to a 9.8% video bitrate reduction from HTTP/2 (TCP) to HTTP/3 (QUIC). Obviously, MoQ is based on a slightly different stack, so the results don't exactly generalize. I can imagine the problems are similar, though.
(I find this stuff fascinating, as I spent the last few months investigating the AF_XDP datapath for MsQuic as part of my master's thesis. I basically came to the conclusion that GSO/GRO is a better alternative and that QUIC desperately needs more hardware offloads :p)
Works really nicely on Chrome, though. Looking forward to the Safari support as I find myself using Orion more and more.
(meaning the link before the edit, https://moq.dev/watch/)
It uses a feature we haven't yet implemented, but we're thinking about how we might implement it at our scale, SUBSCRIBE_ANNOUNCES[1].
[1]: https://www.ietf.org/archive/id/draft-ietf-moq-transport-12....
Submitters: "Please submit the original source. If a post reports on something found on another site, submit the latter." - https://news.ycombinator.com/newsguidelines.html
Edit: since people seem to agree that this was the wrong move, I'm going to undo it.
Edit 2: undone now!
I used to work in live video platform. I have found MoQ interesting enough to work on it again.
And I guess if webview engines like Microsoft Edge WebView2 supports it then developers can use it immediately (wrapping it).
But how about from the other side, I guess OBS and YouTube must start supporting it to actually be useful?
And yeah, being able to publish from something like OBS is something I worked on before joining Cloudflare, but it depends a lot on what you do at the "streaming format" layer which is where all the media-aware details live. That layer is still evolving and developing with WARP being the leading spec so far. As that gels more, it'll make sense to bake things into OBS, etc. Already today though you can use Norsk (https://norsk.video/) to publish video using a rudimentary fMP4-based format similar to early versions of the WARP draft.
As for YouTube, Google has some folks who have been very active contributors to MoQT, but I'm not certain exactly how/where/when they plan to deploy it in products like YouTube.
Technically, WebCodecs is not required since you can use MSE to render, but it's more work and higher latency.
Working on a WebSocket fallback and OPUS encoder in WASM to support Safari.
I've noticed that both Chrome and Firefox tend to have less consistent HTTP/3 usage when using system DNS instead of the DoH resolver because a lot of times the browser is unable to fetch HTTPS DNS records consistently (or at all) via the system resolver.
Since HTTP/3 support on the server has to be advertised by either an HTTPS DNS record or a cached Alt-Svc header from a previous successful HTTP/2 or HTTP/1.1 connection, and the browsers tend to prefer recycling already open connections rather than opening new ones (even if they would be "upgraded" in that case), it's often much trickier to get HTTP/3 to be used in that case. (Alt-Svc headers also sometimes don't cache consistently, especially in Firefox in my experience.)
Also, to make matters even worse, the browsers, especially Chrome, seem to automatically disable HTTP/3 support if connections fail often enough. This happened to me when I was using my university's Wi-Fi a lot, which seems to block a large (but inconsistent) amount of UDP traffic. If Chrome enters this state, it stops using HTTP/3 entirely, and provides no reasoning in the developer tools as to why (normally, if you enable the "Protocol" column in the developer tools Network tab, you can hover over the listed protocol to get a tooltip explaining how Chrome determined the selected protocol was the best option available; this tooltip doesn't appear in this "force disabled" state). Annoyingly, Chrome also doesn't (or can't) isolate this state to just one network, and instead I suddenly stopped being able to use HTTP/3 at home, either. The only actual solution/override to this is to go into about:flags (yes, I know it's chrome://flags now, I don't care) and make sure that the option for QUIC support is manually enabled. Even if it's already indicated as "enabled by default", this doesn't actually reflect the browser's true state. Firefox also similarly gives up on HTTP/3, but its mechanism seems to be much less "sticky" than Chrome's, and I haven't had any consistent issues with it.
To debug further: I'd first try checking to see if EncryptedClientHello is working for you or not; you can check https://tls-ech.dev to test that. ECH requires HTTPS DNS record support, so if that shows as working, you can ensure that your configuration is able to parse HTTPS records (that site also only uses the HTTPS record for the ECH key and uses HTTP/1.1 for the actual site, so it's fairly isolated from other problems). Next, you can try Fastly's HTTP/3 checker at https://http3.is which has the benefit of only using Alt-Svc headers to negotiate; this means that the first load will always use HTTP/2, but you should be able to refresh the page and get a successful HTTP/3 connection. Cloudflare's test page at https://cloudflare-quic.com uses both HTTPS DNS records and an Alt-Svc header, so if you are able to get an HTTP/3 connection to it first try, then you know that you're parsing HTTPS records properly.
Let me know how those tests perform for you; it's possible there is an issue in Firefox but it isn't occurring consistently for everyone due to one of the many issues I just listed.
(If anyone from Cloudflare happens to be reading this, you should know that you have some kind of misconfiguration blocking https://cloudflare-quic.com/favicon.ico and there's also a slight page load delay on that page because you're pulling one of the images out of the Wayback Machine via https://web.archive.org/web/20230424015350im_/https://www.cl... when you should use an "id_" link for images instead so the Internet Archive servers don't have to try and rewrite anything, which is the cause of most of the delays you typically see from the Wayback Machine. (I actually used that feature along with Cloudflare Workers to temporarily resurrect an entire site during a failed server move a couple of years back, it worked splendidly as soon as I learned about the id_ trick.) Alternatively, you could also just switch that asset back to https://www.cloudflare.com/img/nav/globe-lang-select-dark.sv... since it's still live on your main site anyway, so there's no need to pull it from the Wayback Machine.)
I've spent a lot of time experimenting with HTTP/3 and it's weird quirks over the past couple of years. It's a great protocol, it just has a lot of bizarre and weirdly specific implementation and deployment issues.
Webtransport on Firefox currently has issues See: https://bugzilla.mozilla.org/show_bug.cgi?id=1969090
Watching the Big Buck Bunny demo at https://moq.dev/watch/?name=bbb on my mobile phone leaves a lot of horizontal black lines. (Strangely, it is OK on my PC despite using the same Wi-Fi network.) Is it due to buffer size? Can I increase it client-side, or should it be done server-side?
Also, thanks for not missing South Kora in your "global" CDN map!
And you're right that MoQ primarily benefits developers, not end users. It makes it a lot easier to scale and implement features; indirect benefits.
I read this as first QUIC CDN, and thought that can't be true. Dug a little deeper and learned that Media over QUIC is it's own thing. Looks pretty cool.
madsushi•5h ago
kixelated•3h ago
madsushi•2h ago
kixelated•2h ago
The Big Buck Bunny example on the website is actually streamed using CMAF -> MoQ code I wrote.
englishm•3h ago
WARP's development (at the IETF) up until now has been largely spearheaded by Will Law, but it's an IETF spec so anyone can participate in the working group and help shape what the final standard looks like. WARP is a streaming format designed mainly for live streaming use cases, and builds on a lot of experience with other standards like DASH. If that doesn't fit your use case, you can also develop your own streaming format, and if it's something you think others could benefit from, too, you could bring it to the IETF to standardize.