This 'no-root' architecture is exactly what the Sovereign AI space needs right now.
I build decentralized local inference clusters (splitting LLM layers across machines). The biggest pain point is setting up secure tunnels between residential nodes without dealing with WireGuard kernel modules or root access on borrowed hardware.
Two technical questions:
How does Muti handle persistent connections for high-throughput streams (like token streaming)?
Do you have plans for a 'Service Discovery' layer? (e.g. telling Node A that Node B is hosting 'Ollama-Port-11434').
I'd love to test this as the transport layer for my distributed inference stack or discuss potential customization specialist models
andris9•5mo ago
For persistent, high-throughput traffic, Muti Metroo maintains long-lived connections and multiplexes multiple logical streams over a single peer link, each with independent flow control. This works well for token streaming, where low latency matters more than raw bandwidth. In residential networks, QUIC is usually the best choice, with HTTP/2 and WebSocket also available.
Service discovery is handled via the port-forwarding model. A node can advertise a named endpoint (e.g. an Ollama instance), and another node can bind a local listener to that key. The mesh routes traffic end-to-end encrypted, so from the client’s perspective it behaves like a local port even though the service is remote.
For distributed inference, the main constraints are latency and hop count - extra hops add delay, which is fine for background work but relevant for interactive use. Everything runs in userspace, and outbound connections plus QUIC make it usable behind typical residential NATs.
shadowx1soverg•5mo ago
andris9•5mo ago
Service discovery is handled via the port-forwarding model. A node can advertise a named endpoint (e.g. an Ollama instance), and another node can bind a local listener to that key. The mesh routes traffic end-to-end encrypted, so from the client’s perspective it behaves like a local port even though the service is remote.
For distributed inference, the main constraints are latency and hop count - extra hops add delay, which is fine for background work but relevant for interactive use. Everything runs in userspace, and outbound connections plus QUIC make it usable behind typical residential NATs.