What Causes Penis To Go Limp During Sex?

My husband says he masturbates too much and that is why is all of a sudden going limp when we have sex. I went off of the pill a few months ago and we use condoms now and I thought it was them, but we have tried to have sex without the condoms and it is still happening. I am an attractive, slim female for 36 years old but am starting to get worried something is wrong with me. He will be 38 in April of this year.