Hey, So My Boyfriend And I Have Been Dating For

Hey, so my boyfriend and I have been dating for almost 2 years. We've had sex multiple times. But every once in a while when I'm still horny I get soft. I don't know what my problem is. I am attracted to him so I know it's not that because just thinking or looking at him sometimes can get me hard. And now it's been happening more often and he feels like he's the reason but I Know he's not. I tried telling him that he's not and that sometimes I just need time but it's harder when you're a gay couple and he has to get ready and clean to have sex. I don't know what to do or what my problem is. Am I just overthinking about not getting soft that I get soft or is there something really wrong with me?