Does Spotting In Between Menstrual Cycles Affect Fertility?

Hi, I've had spotting around 5 dpo on and off until my period for several months now. Before that I had spotting also at different times of the month. I have been to several doctors and they all seem unconcerned, especially since they've done an ultrasound and found nothing abnormal. Now I'm ttc and I'm worried this will affect fertility. Do you have any thoughts/suggestions/etc.?