Is It Safe To Breast Feed After Taking Dose Of Phentermine?

Hi I took phentermine at 8:30 this morning. My 2 month old pretty much stopped breastfeeding but was really fussy tonight around 8:00. I let him kind of sick a little to calm him down for less than a minute and he didn't latch on. I do think any milk came out if it did it was only a drop. Should I be worried?