Could Taking Lithium During Early Pregnancy Cause Birth Defects?

I was taking lithium during the first 6 weeks of my pregnancy, not knowing I was pregnant. I immediately stopped once I found out and consulting my phychiatrist as well. My question is should I be worried about any birth defects my baby may have, and how likely is my baby to have birth defects?