I've had good luck using `zfs allow` to grant non-root backup users the ability to only add snapshots to their datasets to avoid the "attacker compromises prod and then jumps to the backup server and deleted the backups". It is an extra step to clean up old snapshots, but worth the risk-reduction.
You can also split administration up so that, e.g., my friend sending me snapshots can't even log in as root on his backup server.
soupbowl•6mo ago
A good tip thanks, I did not know about 'zfs allow', I'll be playing with that this weekend.
sgc•6mo ago
`zfs allow` is rather complex. What specific set of permissions are you allowing your users?
benlivengood•6mo ago
For the receiving side the user needs `receive` permission for the dataset to receive new snapshots, but it's convenient to grant `create,mount` as well so the user can send new child datasets which may have mount options in the source dataset, relying on inheritance for the allow permissions to apply to the child datasets.
On the sending side `send` is enough, but for tools like syncoid and znapzend `hold,release` are useful as well since typically they hold the latest snapshot on the source which the destination also has so that it can't be deleted on the source before it's used to send an incremental stream up to a newer snapshot only available on the source.
trod1234•6mo ago
Personally, I'm not seeing how this can possibly have any risk-reduction from a professional standpoint.
You rely upon the permissions model not being broken, but once you have a local login, even with limited perms...a large attack surface is suddenly opened, and the nature of attack surface is the odds immediately go up that there is some piece of code running locally that will allow local priv escalation.
Its relatively simple a lot of times to either escalate local privileges, or trick a green admin to escalate privileges for the attacker (i.e. bind-mount namespaces/ebpf).
If you aren't doing a one-way offline backup, it carries the same risks as replication and all the ransomware related risks through rolling/resource exhaustion.
benlivengood•6mo ago
You can also split administration up so that, e.g., my friend sending me snapshots can't even log in as root on his backup server.
soupbowl•6mo ago
sgc•6mo ago
benlivengood•6mo ago
On the sending side `send` is enough, but for tools like syncoid and znapzend `hold,release` are useful as well since typically they hold the latest snapshot on the source which the destination also has so that it can't be deleted on the source before it's used to send an incremental stream up to a newer snapshot only available on the source.
trod1234•6mo ago
You rely upon the permissions model not being broken, but once you have a local login, even with limited perms...a large attack surface is suddenly opened, and the nature of attack surface is the odds immediately go up that there is some piece of code running locally that will allow local priv escalation.
Its relatively simple a lot of times to either escalate local privileges, or trick a green admin to escalate privileges for the attacker (i.e. bind-mount namespaces/ebpf).
If you aren't doing a one-way offline backup, it carries the same risks as replication and all the ransomware related risks through rolling/resource exhaustion.