On each side of the castle there are little gates, but unfortunately for the attackers those were defended as well.