I'm not that familiar with this subject in general (and not at all for japan) but I have read some about grain agriculture in east asia. And one thing resounds from the record here: people do not like millet and will do their best not to eat it. In northern china (the region I'm most familiar with) people who could afford to eat rice or wheat did so as much as possible, the wealthiest landowners generally did not eat millet at all.

I don't know how this fits into the history here, but if they got rice & millet at the same time and could farm enough rice, it fits with what I've read about other places where both grains were available.

FWIW millet eats fine to my modern palate but then I've only had the probably better tasting modern varieties, who knows what that shit was like a few thousand years ago. I also have access to a wide variety of grains and I might feel differently if I had to pick one to eat every day of my life. Similar thing with oats, which have occupied a similar role in the mediterranean for a long ass time: animal fodder if you could afford wheat, your food if you couldn't.

I've read that in the Americas, some cultures abandoned farming, and went back to foraging. One reason was that having large stores of transportable food made you a sitting duck for your neighbors wanting to come and help themselves.