I also have one of those tiny 2-people West Marine inflatables. It always makes me nervous - they have a lot of windage for the size, and as you discovered rowing against a strong wind (or current) is very difficult and disheartening! I am looking out on Craigslist for a slightly larger tender with a motor. Glad you got back at all with that wind!
Where we are, the good news is that you can sail all year - the bad news is that while the summer winds are strong (typically 20-25 knots every day), and in winter very light (currently around 5 knots). For some reason Selene does not seem to be very competitive in 5 knot winds...I wonder why not? :-) But at least you can lie in the sun in your shorts over winter....