I can definitely relate to your frustrations -- it drives me up the wall sometimes. I think that the issue is definitely more pronounced in the central arrondissements. The 6th is probably ground zero for that type of behaviour as so many tourists pass through it. I've also had that experience in the 5th, but funnily enough it pretty much never happens in the 17th, where I work. I guess it's because tourists don't tend to go there. But if I were to cross the road and head into Montmartre, suddenly the behaviour crops up again...
I don't have a very strong accent. In fact, sometimes French people think I'm French until I tell them otherwise. So it's annoying in other circumstances where people have already presumed that I don't speak French before they've even spoken to me, as it reveals their prejudices and makes a mockery of the republican idea that anyone can be French regardless of their origins (because clearly a lot of people think that while anyone can be French, some French people are more French than others). I mean, I'm not complaining that people don't consider me French, as, well, I'm not French. But if they don't consider me French on account of my physical appearance, does that mean that they think likewise of others who look like me? I'm inclined to believe that that is indeed is the case, which is quite a depressing thought, as it would suggest that France isn't as colour-blind as its constitution would make it out to be.
When a French person speaks English to me, I insist on replying in French. Eventually, they would revert to French as my French is almost always better than their English, but there are some who insist on holding out. I feel like the only real solution is to simply state that I want to speak French because I'm in France. After all, if French people complain about foreigners not speaking French, then when foreigners actually make the effort to learn French and speak it fluently, then they shouldn't be denied the right to speak it.
I certainly don't think you're being 'hyper sensitive' about the issue. I suppose you could ask your husband the following question, "If you went to London, would you want Londoners to speak to you in French?" If he answers "no", then perhaps you could ask him why not. Then he might be able to relate to your point of view :)