Making it an on-demand cache instead is a neat next step. Whether it helps or hurts depends on the actual input: If the input image uses every pixel value anyway, the additional overhead of checking whether the table entry is computed is just unnecessary extra with no value.

But if a typical image only uses a few pixel values, then the amortized cost of just calculating the few needed table entries may very well be significantly below the cost of the current algorithm.

If images are somewhere in between, or their characteristics not well known, then simply trying out both approaches with typical input data is a good approach!

Unless you’re perfectly happy with 0.2 seconds, for example because the runtime of some other parts take so long that dwarfs those 0.2s, then why bother.