Fantasy never really does scale correctly, even when it tries to be big.
There aren't nearly enough castles. There should be a castle like every 3 miles. Literally every knight has a castle, sometimes multiple. You cannot be a knight without owning a castle. There are thousands of knights.
If you can’t see and hear it think, how can you be sure it isn’t faking it?