That's a very long writeup to say "the computer did what I asked it to do".

Use the slop to make the slop.

You gave your agent access to your system with your permissions. Then you asked it to do something that you can do with your permissions, and it did. You don't want it to do what it can do, but instead of doing the correct thing (don't give it your permissions) you want to add a million and one rules, that will never be complete, restricting its access.

The correct thing, and this isn't hard, is to give it a different set of permissions than yours. Do not let it get into your .ssh directory at all, problem solved. People keep posting these stories as if there's something useful in them, and there is:

The moral of the story is don't be a fool. Give your agents different, restricted, permissions, than your own permissions.

This is the same approach when working with people. Your agent should be, from an authentication and authorization perspective, treated like any other employee: Given an appropriate set of permissions to their needs, not root level access to all your systems.