Teardown of the SpaceX Starlink User Terminal https://news.ycombinator.com/item?id=25277171 (December 2, 2020 — 158 points, 138 comments)
If one is doing 1Gbps of traffic which is 100 byte UDP packets, that's a million packets per second you're gonna need to process.
A 1Ghz CPU only then gets 1000 cycles to process each one...
Very doable, but certainly not easy unless your engineers like hand coding assembly and having to think about every lookup table trick in the book...
The way it usually works is that the initial packets are handled in software but once the endpoints are established it flows through hardware. Sometimes certain patterns are always handled in software. The software could be a patched kernel or a XDP style kernel bypass.
Source: worked peripherally on an Intel Puma cable modem router/gateway that used DPDK or something like it. So I'm not 100% sure, but it is an educated guess.
100 byte?? Starlink has regular 1500 byte MTU.
Specifically for cases of forwarding DPDK-style approach can be faster because it will incur fewer buffer copies.
Starlink only does 25-200Mbps and average packets are like 7-8x larger so at most you're doing ~36000 PPS which is pretty manageable even on 1Ghz
Forty-one? So who does not have root access to "your" user terminal?
On a more serious note, is this any different from ISPs having a remote management system for ISP provided routers? In terms of privacy, if SpaceX didn't have access to the user terminal, they could still just capture your traffic on the sattelite or the ground stations
The risk of access to the router is more that they can access your network and touch unprotected and vulnerable things rather than active monitoring.
No, wiretaps on modern networks do not rely on backdoors, or even big labeled front doors like SSH, on individual subscriber devices. Instead it is built into the lower level routing. When an ISP gets a warrant (or whatever relevant document your country uses) they configure their routers to tag all of your traffic and mirror it to a server to be recorded. It’s entirely invisible to the subscriber, and highly automated.
The topic at hand is local network access. Some examples of things happening on your local network that you might not want Elon/ISP to see:
- Many people have public shares on their NAS for things like media or family photos
- Security cameras
- Printers
- If you're casting, the title of everything you watch is broadcasted on the network
- Even if you're not casting, if you use an Android TV then the title of everything you watch is also broadcasted to the network by default
- The list of all your devices
- If you torrent things, then the hash of all your torrents is likely broadcasted to your local network (through Local Peer Discovery)
Maybe, but in more and more European countries, ISPs are required to accommodate you hooking up your own router/modem. E.g., I am on fiber and if I want to I can hook up my own router directly to fiber with an SFP+ module (I currently use the ISP-provided media converter, but my own router). Lots of tech users use their own Ubiquiti/OPNsense/OpenWrt routers, so no remote management.
I wonder if this requirement applies to Starlink as well, since they are an ISP.
DSL tech is far simpler and it's always a combo unit so I could see a case where you would be allowed to bring your own DSL modem.
But it just doesn't work like that for DOCSIS or GPON where the cable modems or ONTs these days do much more than just media conversion - SIP, PPPoE, IGMP, etc. even if they don't do Wi-Fi (so ISPs don't call them "routers" - except SingTel, which uses "ONR" to distinguish these units because they are in fact routers for IPTV and SIP).
For all of those modems/ONTs, the firmware updates and the configuration for telephony/SIP and PPPoE are controlled by the ISP and also tested to work with their OLT or CMTS so it's just not possible for the ISP to guarantee support for any random modem or ONT.
And to support the advanced configuration required these days for VoIP, IPTV, etc. on the "modem" or "ONT", ISPs basically have a backdoor called TR-069 which is really not too dissimilar to what Starlink has access to with their SSH keys.
Even if you get "true" dumb modems or ONTs which do not do any routing whatsoever, the device on the other side still has full control over your dumb device via the DOCSIS provisioning process or GPON's OMCI. Starlink seems to be using SSH instead of building a whole protocol - because satellite tech is proprietary and doesn't need to work on other hardware.
So, I find that it's highly unlikely that the ISP is officially required to support a user supplied modem, although I haven't consulted the EU laws on this.
At most, I think using your own router would require the EU ISPs to provide bridge mode support, but that's not special to EU. However, the TR-069 backdoor is still active even with bridge mode.
It can be fairly easy to stop TR-069 with a "dumb" ONT (usually SFP) but ISPs can and will notice that. Whether they allow it is up to them.
Ziggo (called UPC in other EU countries) uses DOCSIS. The instructions on how to use your own DOCSIS modem are at the following link (in Dutch): https://www.ziggo.nl/klantenservice/apparaten/wifi-modems/ei...
Edit: it really is using your own modem. It's not about putting it in bridge mode.
Plus depending on model (like Arris modems), I can do things like set the password of the day seed (away from the factory default) to further lock it down and gain management access remotely.
DOCSIS is slowly dying here anyway and bleeding customers because the cable providers are not competitive when it comes to internet. If they didn't have better linear TV packages the bleed would even be larger.
For many modems on the customer market this also can mean that the ISP can push their own version of the firmware for a modem if you buy identical - such as pushing SURFboard updates.
Not really, when you want to increase the bandwith, e.g. with vectoring[1], you need to have all neighbor modems to participate, which prevent free modem choice for the users.
>But it just doesn't work like that for DOCSIS or GPON where the cable modems or "ONT" router combo units these days do much more than just media conversion - SIP, PPPoE, IGMP, etc.
In Belgium, the ONT is just media conversion these days, SIP is done on the provider box, so you can have your own GPON SFP.
>so it's just not possible for the ISP to guarantee support for any random modem or ONT.
The ISP doesn't have to guarantee support to let you use your own hardware. It just have to give you the specs to use it and let you plug the ISP box if you can't configure vlan of dhcpv6 client.
Tr069 allows the isp to remotely configure their equipment which most people are happy with, but if you want to use your own then that’s fine, and obviously unless you enable it the isp won’t configure your router or any other equipment.
Starlink acts far more than a media converter.
At least in Finland the norm is that you can use your own DOCSIS modem from any manufacturer, you just tell the ISP your modem's MAC address.
Not for GPON, though.
At no point was my firmware swapped out, or anything of that nature.
https://pon.wiki/guides/masquerade-as-the-att-inc-bgw320-500...
Discord: https://discord.gg/8311
In the past (and again on some fiber networks), there was https://en.wikipedia.org/wiki/Network_interface_device and the separation occurred there.
Note that it might not be the same at L1 and L2, e.g. your ISP may own the physical lines up to the side of your building, and the data link layer up to your modem.
Article 5 of the EU Net Neutrality Regulation states that “end-users should be free to choose between various types of terminal equipment […]. Providers of internet access services should not impose restrictions on the use of terminal equipment connecting to the network […].” [1]
However, currently only 5 member states fully comply with this (Germany, the Netherlands, Belgium, Finland and Lithuania). [2]
E.g. NL ISPs provide all necessary parameters for PPPoE and SIP (telephony). [3]
[1] https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:...
[2] https://fsfe.org/activities/routers/routers.en.html
[3] https://www.kpn.com/w3/file?uuid=cd5f3398-4bad-4cdc-ac18-4f6... (DSL) https://www.kpn.com/w3/file?uuid=563993a1-e48a-485a-90a8-738... (FttH AON/PON) https://www.kpn.com/w3/file?uuid=b9774a1d-f1cb-4c17-8972-251... (SIP)
Interestingly satellite earth station is explicitly mentioned, so maybe we can use our own Starlink dish, but not out own modem... (law is weird)
‘terminal equipment’ means:
(a) equipment directly or indirectly connected to the interface of a public telecommunications network to send, process or receive information; in either case (direct or indirect), the connection may be made by wire, optical fibre or electromagnetically; a connection is indirect if equipment is placed between the terminal and the interface of the network;
(b) satellite earth station equipment;
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv%...
The Netherlands regulator interprets this by saying that “the network connection point must be passive” and “all (radio) devices that are located at the premises of the end user and that are connected to the network connection point are end devices”. [1]
[1] https://www.acm.nl/system/files/documents/beleidsregel-handh... (Dutch)
> a connection is indirect if equipment is placed between the terminal and the interface of the network
If I check the official version in my native language, that (translated back to English by me) explicitly says:
> in the case of indirect connection there is an additional equipment/device between the terminal equipment and the interface of the network
If we assume that the the interface of the network is the optical or copper cable and the terminal equipment is your router, then (at least to my understanding) this additional equipment could be the modem of the service provider. The EU law doesn't speak about active or passive status of these equipments.
Maybe the Dutch law is stricter?
If you google translate the official French version to English via Google Translate, you got:
> a connection is indirect if a device is interposed between the terminal equipment and the interface of the public network
From German:
> in the case of an indirect connection, a device is connected between the terminal equipment and the interface of the public network
From Dutch:
> a connection is indirect when a device is placed between the terminal equipment and the network interface
No. At least in The Netherlands the regulations are very clear. Any device that directly or indirectly connected to fiber, copper, or 'electromagnetic waves':
https://wetten.overheid.nl/BWBR0038908/2016-12-28/#Artikel1
Also, if you read the rest of the regulation, they make it very clear that the ISP should accommodate the user making it possible to use a passive end from the ISP (so, just the fiber/copper, with no active devices from the ISP required):
https://wetten.overheid.nl/BWBR0045477/2022-01-27
ONTs these days do much more than just media conversion - SIP, PPPoE, IGMP
The ONT (at least in the definition used in our country) only does media conversion. Usually a dedicated device is used as the ONT (Genexis/Nokia ONT) that just does the fiber <-> ethernet media conversion. You can use the ISP's or bring your own. Some people use an SFP+ module as the ONT.
PPPoE, SIP, etc. are usually handled by a combined router/modem.
For all of those modems/ONTs, the firmware updates and the configuration for telephony/SIP and PPPoE are controlled by the ISP and also tested to work with their OLT or CMTS so it's just not possible for the ISP to guarantee support for any random modem or ONT.
It is not a problem in practice. E.g. my provider allows you to bring your modem/router, as long as it supports PPPoE and VLANs. For example, I currently use the provider's media converter and use a Unifi Gateway Max as my modem/router (it does PPPoE). Before that I had fiber directly hooked up to my own Fritz!Box with an SFP+ module. I'm on some Dutch tech forums and people use a lot of different equipment:
- ONT: provider or their own media converter, or an SFP+ module (typically Zaram or AVM).
- Router/modem: Unifi (e.g. Cloud Gateway with fiber <-> ethernet converter or a Dream Machine with an SFP+ module), OPNsense (handles PPPoE as well), Fritz!Box, a plain old Linux distribution, OpenWrt (seems more rare).
ONTs which do not do any routing whatsoever, the device on the other side still has full control over your dumb device via the DOCSIS provisioning process or GPON's OMCI
I think ONTs are less of a problem, because it's on the other side of a security boundary. The modem/gateway is where you don't want an ISP backdoor.
So, I find that it's highly unlikely that the ISP is officially required to support a user supplied modem, although I haven't consulted the EU laws on this.
It is, as long as the ONT and modem correspond to the specs. E.g. my provider requires PPPoE and VLANs [1]. As long a router/modem supports it, they have to allow it. Of course, they don't have to debug issues inside, say a Unifi gateway for you. But if such a device fulfills the requirements, they have to allow it on their network.
Again, people here do all kinds of stuff. Like recently I saw someone who uses a Banana PI R4 with a Zaram SFP+ module as their ONT+modem/router. And the ISP has to allow it, because the user is allowed to replace any active component.
When I experimented a bit with SFP+ modules, etc., I had the ISP on the phone some times and they were very helpful and accommodating and said that my setup was pretty normal compared to what they saw some other tech people doing.
[1] https://assets.ctfassets.net/zuadwp3l2xby/2Yp0HtLJPKBUX5mqr3...
I don't think you quite understand how this works.
The ISP controls whatever the other end of that fiber is plugged into. It doesn't matter if the medium is fiber, or copper, or a piece of string. The ISP always has control of the other side of the customer interface. It doesn't matter if the box physically resides in your home or not.
In the case of Starlink, it's all contained within one box.
In the case of DOCSIS (cable), you may physically own the modem, but the ISP controls the firmware it netboots and has full remote admin to the device.
So, let's talk about fiber. So:
The ISP controls whatever the other end of that fiber is plugged into. It doesn't matter if the medium is fiber, or copper, or a piece of string. The ISP always has control of the other side of the customer interface. It doesn't matter if the box physically resides in your home or not.
Sure, the ISP owns the other end, but what's your point? By using my own router (and my own SFP+ module, which is less important), the ISP does not have a device (backdoor) on my network and cannot control my router. Sure, they could capture traffic on their end, but at that point it's pretty much all encrypted anyway. If I don't trust my provider knowing to what individual hosts I connect, I could set my router to tunnel all traffic to another host/VPN/whatever.
At any rate, using your own router + maybe modem removes the worst backdoor when it comes to providers.
If a normal ISP wants to operate in country a, they need infrastructure in country a. This means they either follow country a's laws or that infrastructure gets seized.
Starlink could just as well be operating entirely from the US, and there's very little that foreign governments could do to stop them if they break some foreign laws. They could make payments and shipping complicated, which is probably why Starlink would rather comply if the requests are somewhat reasonable, but Musk has indicated multiple times that he's willing to stand up to unreasonable restrictions if the need is dire enough.
And you can always go after people - and I mean both Starlink executives and customers.
This is facilitated by two things.
First, we at Starlink are on the cusp of forcing light into 'particle only' mode, or PAM. With PAM, as the light is a particle, not a wave, it undergoes perfect time dilation.
Thus, transmit time between the moon and earth is instantaneous!
With this idea moderately solidified, we're preparing an aggressive launch schedule, to get that hardware on site!
Starlink receivers will be able to transmit via PAM too... once we've flushed out the tech.
All it will require is a series of future software updates, so buy your Starlink now!
eg they could outright ban the sale of StarLink products, ya know, being in charge of the laws and all
The problem then with Starlink is nobody is manufacturing compatible third party Starlink terminals, at least yet.
I ended up extracting 802.1x certificates with this[2] or a similar tool and interfacing directly with the ONT using OPNSense [3]. I was so angry I filed an FTC complaint because I had to do this bypass to do my job (the latency was so bad).
---
[1] -- Linking to Reddit due to DSLReports going Down. https://www.reddit.com/r/ATTFiber/comments/1dwwh61/comment/l...
[2] -- https://github.com/0x888e/certs
[3] -- https://forum.opnsense.org/index.php?PHPSESSID=t6vvukft2ahga...
I know this because I tested this.
However, part of being compatible is that the modem bust be configurable and controllable by the ISP. The ISP will typically flash their own firmware to your modem without asking or telling you.
The same applies to cell networks. The actual modem in your phone is running a binary blob provided by the network operator.
This is done for very good reason, without control of all modems in the netork, the operator can't provide reliable service. A broken modem in the network screaming out non-compliant signals can screw things up for everyone.
It's a necessary evil for the network operator to have privileged control over the hardware on their network. You solve this by treating your modem as a direct, untrusted, unprotected hole into the public internet. You also must assume that the ISP can inspect traffic on their network. To solve all of this, you put a firewall between it and your local network.
I really don't think it's at all a problem for ISPs to have this level of control over customer modems. It is a problem when the ISP bundles the modem and router, but that's a different conversation. Just always use your own router.
Dozens/hundreds/thousands of web servers servers can easily share one private key in a certificate, public keys offer even more options on sane designs. Directly authenticating 41 servers using ssh-keys is just poor, slap dash engineering.
It is not, amd I can't see how my earlier comment can be read as recommending that. This is a solved problem for private keys (using load balancers, for example) , so public keys are lower-hanging fruit than that.
Edit: upon rereading, I cam see how the word "share" would be ambiguous in the context of if a private key. I meant "jointly make use of", rather than "distribute copies throughout the fleet". I have exited my root comment to make my meaning clearer.
So the terminal accepts "sshauthority1"
Then the 41 remote sites contact sshauthority1 to get a 1 hour (10 minutes, 10 days, whatever) long certificate for "site18"
If a remote site is compromised sshauthority1 no longer issues certificates, and within an hour (10 minutes, 10 days, etc) the remote site can no longer reach the terminals.
Revoking a key from that many terminals (many of which will be offline) if one of the 41 keys is exposed is not trivial.
Now if sshauthority1 is compromised then you've got the same issue with rotation (although can CRL it), but it's easier to secure one or two authorities than 41 keys.
I wasn't suggesting it, and frankly can't see how that could be a solution in this instance. I was making a comparison against current practices on a harder problem to solve , i.e. safely scaling a single private key in an SSL certificate across many servers is solved today without a 1:1 server to certification ratio
HTTPS uses X.509. OpenSSH has no interest in supporting X.509 or, AFAIK, for changing their version to support anything but "self-signed" keys.
There's more than 1 way to skin this cat, and no, I'm not asking for the a specific solution you suggested.
SpaceX can implement any internal auth-scheme they choose to connect to a handful (not 41) of SSH intermediate instances, which then connect to the terminals
I imagine starlink has more than 1-2 sysadmins. I think a hundred pubkeys would be reasonable.
Having some way to remotely push updates, and having some kinda of (preferably with your consent!) remote access might be reasonable, but I would expect that to be via some kind of intermediate gateway/app/something and not direct from a sysadmin’s individual account.
I imagine the most ideal situation would be 1) minimising the number keys, 2) use hardware backed authentication or certificate based authentication, 3) lock up the private keys somewhere safe.
The idea of 41 points of failure that can SSH into any starlink terminal is not appealing.
Not to defend this but curious: Unless the terminal is attached to a local network that also has internet access, the satellite network would need to be traversed in order to connect using those keys, right? What kind of NAT/etc does Starlink use for satellite routers?
We have a separate piece of software to remotely access devices in prod to help diagnose engineering issues, where we can pull up a REPL, but that's under access control and gated by devops etc.
[0]https://www.npr.org/2025/04/15/nx-s1-5355896/doge-nlrb-elon-...
Anyone has links to resources about how to emulate a firmware that connects to external devices (GPS here), any ready solutions?
> Android Emulator is downstream from the QEMU emulator. It adds support for booting Android devices, emulates typical Android hardware (OpenGL, GPS, GSM, Sensors) and a GUI interface. The android emulator extends qemu in various ways.
I was looking for some overall guide of taking a firmware that interacts with the external environment and how to emulate it
(a) emulate platform (e.g. x86/Arm SoC) running "firmware" = OS/drivers/userspace
(b) emulate devices (e.g. MCU, sensors, gateware, firmware) communicating with (a)
(c) simulate behavior of OS+userspace software running on (a)
(d) other
QEMU (OSS)
Ant Micro Renode (OSS)
Arm FVP (commercial)
Intel SIMICS (commercial)The fact that they could just dump the filesystem tells me there's no protection employed at SpaceX aside from the boot loader mentioned in the article.
Rather, please consider spending your resources wisely, on something that benefits everyone and makes your product better. For power users, a theoretical ability to modify your product (possibly, in a ways you've never even thought about) can be a valuable benefit. So, unless you're certain to be seriously harmed by this in some way, please consider not wasting your (and your users') time on something of questionable value.
Just saying how it looks like from a technical end-user perspective. I'm just really tired and even somewhat depressed of having to hack my devices (lights, cat feeders, now a rowing machine) to make them work properly.
Usually there's a UART. Apparently Starlink terminals don't have a UART, so the guy took off the eMMC memory chip (which is basically a soldered microSD card) instead.
walterbell•1mo ago