Of course you still need one binary per CPU architecture. But when you rely on a dynamic link, you need to build from the same architecture as the target system. At that point cross-compiling stops being reliable.