Up front, I will state I am not an expert about this, but have read articles (and of course believe everything published on the internet). Fortunately have not had this problem myself, even when I bought Selene, after she had sat on the hard for 6 months or more.
The port side should not be bowing in. This is not a good thing.
Worried? Well, I would be worried about the yard, and their nonchalance reaction. The chances are it will indeed pop out - but it should not happen, and there is likely to be some damage which can vary from the insignificant to the worrying. It may be worth getting a second opinion from a surveyor - it may cost you money, but if there was a problem would make your life a lot easier. Regardless I would photograph it. Inspect carefully. Keep a written record.
Chances are, small deflection: minimal damage. Exterior, look for fine cracking on the gelcoat - will it leave a mark as it 'pops out'?. Interior, look carefully. Depending on where it was positioned it could have damaged interior layups, e.g. anything tabbed or attached to he hull. Was it near the battery boxes? Are they still securely tabbed in place? Anything else near the bulge which could have moved/broken? Any sign of the layup cracking?
Fiberglass is amazingly strong and resilient stuff, and the Pearsons are well built. But it pays to be careful. No need to panic, but good reason to be concerned.
IMHO. Good luck, and sorry for the situation.