What Causes Pain After Getting A Metal Brace For Trapesium Fracture?

I broke my trapesium bone 3 weeks ago, it was put in a removeable velcro 7 METAL BRACE, BUT STILL HURTS JUST AS MUCH. sorry for the caps, hard to type. should I go back to the Dr. or does this bone take a long time to heal? it moves in the brace, and every time it does, it really hurts still.