What Causes Yellow Crust At The Center Of The Nipples?

I ve always had flat/inverted nipples. For a long time now I ve had yellow crust in the center of my nipple. It comes off if I pick at it (or sometimes it gets caught in my clothing) and when it comes off it hurts. After while is grows back. Should I be concerned?