Can Plaster Used For Fracture Repair Be Removed Without Doctor's Approval?

Hi, 3 weeks back I had an twisted ankle of right leg and after doing xRay got to know that there are two hairline fractures, Doctor applied Plaster, Now I am able to walk, with almost no/very little pain. can I remove the Plaster or should I have to go to that same doctor for further consultation?