A Couple Of Times After Sex I Have Ended Up

A couple of times after sex i have ended up with one side of my vagina swollen and sore. It isnt rough sex. We use condoms. I have some allergies which include high allergy to birch. I was wondering if it could be an allergy to latex even though it doesnt hapoen everytime