Here’s part of my towns Castle. It got shafted a bit in the war, but it’s still standing. I’ve only ever been in the actual grounds once by blagging myself in for free as part of a film crew many years ago.
Besides, I prefer the views from around it where the castle is included too. I don’t see the point looking out from it… there’s plenty of other bits of the cliff to stand on, and you don’t get charged for that.