I Work On A Gourmet Mushroom Farm. I Handle Everything

I work on a gourmet mushroom farm. I handle everything from packing the bags of spawn in boxes being shipped to harvesting the mushrooms and also dealing with conaminated bags. Is it safe for my to be around the conaminated bags and spores of the mushrooms? Bags that are conaminated are normally conaminated with green mold or black mold. Can or will this affect my unborn child?