I'm not saying this is possible or if I've done it. I'm aware that the general consensus is that transformers are the ruling paradigma and they're both data hungry and smartest. As per google: "Training a 100 million parameter Transformer model on a low-end or slow CPU is highly impractical and will take anywhere from a few weeks to nearly a year, depending on the size of your training dataset." - assume your model can be trained on slow CPU in about a day/couple of hours. While it lacks encyclopedic knowledge it may be armed with heavily curated scientific knowledge. So: what do you then ?