High dimensionality signifies that your model has learned a nice, complex feature space of language.
The dimensionality peak in LLMs (and to a weaker extent, speech-audio models) marks a phase of higher-order linguistic abstraction, which we showed with probing.
