I got frustrated with the slowness of rsync and made an alternative that works faster by using multiple parallel connections, direct encrypted TCP when available, and other optimizations. I also added cool features like the ability to maintain a persistent ssh connection to the server for fast one-offs, the ability to download to your laptop while working in an ssh shell on a server, and the ability to do direct remote-remote transfers without forwarding your ssh agent (by using restricted ssh keys on the receiver that will only execute a specific request signed with the key on your laptop). It is also designed to be more robustly/flexibly scriptable than alternatives.
Comments
mika6996•1h ago
Why is this better than rsync?
greaber•1h ago
There are a bunch of reasons, some of which are explained in the description and in the docs. For me, the most important benefit is speed of copying. Note that syq doesn't currently implement rsync's delta merge algorithm, which allows it to avoid copying data that is already in the destination file but at a shifted offset. I might implement this (or an enhanced version of it) in the future. If your workloads have a lot of cases like this then syq might not be better than rsync for you, but I found that for me this rarely came up.
amelius•39m ago
There's something vastly superior over rsync still: "btrfs send" and "btrfs receive" (but require you to have btrfs on source and destination).
greaber•2m ago
Thanks for the suggestion! I'm not very familiar with btrfs send and btrfs receive. Browsing the documentation, it looks cool, but maybe you could say more about the specific things that make it so much better for your use case? Could any of them be incorporated into a more general tool like syq? Also, it doesn't look like btrfs send helps transfers go fast by parallelizing them or has anything like syq's remote-remote transfer feature, which made me wonder if it would actually make sense to make syq able to work as an authenticated fast transport for a stream generated by btrfs send/receive.
nimih•30m ago
Browsing the repo, I can see a couple of advantages:
- Single contributor. Software bugs are generally caused by developers writing code. By reducing the number of contributors, syq has cleverly reduced the surface area for defects to sneak in.
- Distribution via shell script. rsync is bundled in most linux distributions, which means you have to deal with annoying software updates from time to time. Syq, OTOH, tells you to pipe curl into bash to execute a shell script, which means one-and-done installation and maintenance.
- UI clarity. If you search StackOverflow for rsync, you'll see thousands of questions asking how to accomplish various tasks with rsync, both straightforward and arcane. By contrast, `syq` isn't even a tag on SO. The obvious conclusion here is that rsync's interface is so byzantine, and its documentation so poor, that users must resort to asking strangers for help, a problem obviously not shared by syq.
greaber•16m ago
- No need to be snarky. It's a new project, while rsync is time tested, and that is certainly an advantage for rsync and all old software.
- You can install by piping curl to bash, using brew, or by compiling yourself. One thing about distribution I would argue syq gets right is that you are never relying on whatever version of syq happens to be installed on a server. The client syq always talks to a server syq tagged with the exact same version, and when your local copy of syq installs a remote copy, it verifies that the remote binary is signed by me.
- I never said or implied that rsync has a byzantine interface or poor documentation. I think rsync is well documented and the interface is overall fine. Syq even has an rsync compatibility mode, so you don't have to learn any new syntax if you don't want. Still, I tried to provide good docs for syq too, and honestly even if there were a gap in the docs, all you have to do these days is ask AI to look at the source code and tell you what to do. I think this also mitigates the trust issue with a new project from a single contributor.
ProphetOfParado•7m ago
How do you hope to convince sys-admins to install this? I work with clusters where I do not have superuser access, so this is a no go for me.
cynicalsecurity•11m ago
I'm eager to replace battle-tested rsync with a new half-baked tool that no one uses for a tiny speed improvement. I'm also going to enable automatic updates of that tool too.
xz18r•3m ago
You live up to your username!
greaber•45s ago
I'm seeing 5x speedups on my real workloads, and you can see some synthetic benchmarks here or run your own https://greaber.github.io/syq-bench/
mika6996•1h ago
greaber•1h ago
amelius•39m ago
greaber•2m ago
nimih•30m ago
- Single contributor. Software bugs are generally caused by developers writing code. By reducing the number of contributors, syq has cleverly reduced the surface area for defects to sneak in.
- Distribution via shell script. rsync is bundled in most linux distributions, which means you have to deal with annoying software updates from time to time. Syq, OTOH, tells you to pipe curl into bash to execute a shell script, which means one-and-done installation and maintenance.
- UI clarity. If you search StackOverflow for rsync, you'll see thousands of questions asking how to accomplish various tasks with rsync, both straightforward and arcane. By contrast, `syq` isn't even a tag on SO. The obvious conclusion here is that rsync's interface is so byzantine, and its documentation so poor, that users must resort to asking strangers for help, a problem obviously not shared by syq.
greaber•16m ago
- You can install by piping curl to bash, using brew, or by compiling yourself. One thing about distribution I would argue syq gets right is that you are never relying on whatever version of syq happens to be installed on a server. The client syq always talks to a server syq tagged with the exact same version, and when your local copy of syq installs a remote copy, it verifies that the remote binary is signed by me.
- I never said or implied that rsync has a byzantine interface or poor documentation. I think rsync is well documented and the interface is overall fine. Syq even has an rsync compatibility mode, so you don't have to learn any new syntax if you don't want. Still, I tried to provide good docs for syq too, and honestly even if there were a gap in the docs, all you have to do these days is ask AI to look at the source code and tell you what to do. I think this also mitigates the trust issue with a new project from a single contributor.
ProphetOfParado•7m ago