I have never replaced my stuffing box hose, so I can't comment on whether or not the shaft needs to be removed.
I can say, however, that rudder removal is NOT necessary. The 323 has a full skeg-hung rudder, so rudder removal wouldn't help as the skeg would still be in the way.
The 323 has an offset angled shaft so that the shaft (with the prop removed) will just clear the rudder & skeg. (see photo)