Iterating topology is way more powerful than iterating weights. As far as I am aware, FPGAs can only be reprogrammed a fixed # of times, so you won't get very far into the open sea before you run out of provisions. It doesn't matter how fast you can run the network if you can't find any useful instances of it.
The fastest thing we have right now for searching the space of SNNs is the x86/ARM CPU. You could try to build something bespoke, but it would probably start to look like the same thing after a while. Decades of OoO, prefetching and branch prediction optimizations go a very long way in making this stuff run fast. Proper, deterministic SNNs have a requirement for global serialization of spiking events, which typically suggests use of a priority queue. These kinds of data structures and operational principles are not very compatible with mass scale GPU compute we have on hand. A tight L1 latency domain is critical for rapidly evaluating many candidates per unit time.
Of all the computational substrates available to us, spiking neural networks are probably the least friendly when it comes to practical implementation, but they also seem to offer the most interesting dynamics due to the sparsity and high dimensionality. I've seen tiny RNNs provide seemingly impossible performance in small-scale neuroevolution experiments, even with wild constraints like all connection weights being fixed to a global constant.
That's not true unless you're talking about mask programmed FPGAs where the configuration is burned into the metal layers to avoid the silicon area overhead of configuration memory and even in this case the finite number is exactly one, because the FPGA comes preprogrammed out of the fab.
Almost every conventional FPGA stores its configuration in SRAM. This means you have the opposite problem. You need an extra SPI chip to store your FPGA configuration and program the FPGA every time you start it up.
The big problem with SNNs is that there is no easy way to train them. You train them like ANNs with back propagation, which means SNNs are just an exotic inference target and not a full platform for both training and inference.
He had hacked 60%+ of the world's IoT at one point. Largest botnet we ever saw. Everyone's devices had developed weird delayed connectivity issues, variable pingtimes, random packet losses with re-transmits.
He kept saying he was trying to bring about emergent AGI, blathering on about spiking and variable network delay between IoT nodes, clusters, and blah blah blah.
"There's 10B IoT devices already!". He was frantic. Wild eyed. "I have to finish this".
Well I arrested his ass, and this won't work as a defense Segrey!
-- Random NSA agent
kingstnap•6mo ago
Your energy costs are a function of activity factor. Which is how many 0 to 1 transitions you have.
If you wanted to be efficient, the correct thing to do is have most voltages remain unchanged.
What makes more sense to me is something like mixture of experts routing but you only update the activated experts. Stock fish does something similar with partial updating NN for board positions.
imtringued•6mo ago
For those who don't know why this matters. Transistors and all electrical devices including wires are tiny capacitors. For a transistor to switch from one state to another it needs to charge or discharge as quickly as possible. This charging/discharging process costs energy and the more you do it, the more energy is used.
A fully trained SNN does not change its synapses, which means that the voltages inside the routing hardware, that most likely dominate the energy costs by far, do not change. Meanwhile classic ANNs have to perform the routing via GEMV over and over again.