Suggest Treatment For A 4 Inch Fibroid

i have a 4 inch fibroid and am trying to get pregnant. It doesn't effect my womb lining in any way. My fertility specialist says to remove the fibroid risks damage to the womb and possibly to the ovaries from hormones I would have to take before the surgery (my ovarian reserve is extremely low). She advises not to remove the fibroid. I have a bicornuate uterus also and am concerned about increased risk of miscarriage and birth defects (due to lack of space or reduced blood flow) from both the bicornuate uterus and the fibroid. Does the fibroid increase these risks?