I think this is actually a fair point, but that in itself contains the answer: his actions collapsed any coalition of supporters he originally had, from every direction. He failed to please /anyone./
Anyone can give a one-line answer as to why each PM from Thatcher to Sunak got kicked out of office -- sometimes just one word. I don't think that's the case with Starmer. Clearly he didn't shine as PM, hence he's out. But the diversity of views as to what he did wrong is what's unusual.