If mathematics is understood as axioms + formal rules of inference + symbol manipulation, then no physical interaction is required for learning. A mind could, in principle: receive axioms, apply logical inference, derive theorems (Hilbert / formalist view).

The catch is, however, where do the axioms and concepts come from. Even the most abstract mathematics relies on conceptual primitives like number, equality, ordering, continuity, infinity, symmetry and these concepts are not innate in full mathematical form, but are bootstrapped from counting physical objects, spatial navigation, motion and change, containment and boundary, symmetry and repetition.

Cognitive science strongly suggests that human mathematical intuition is deeply grounded in embodied experience.