Does Swelling In Fractured Leg Delay The Healing Process?

Hi
I broke my ankle three weeks ago, had an orif surgery and was recently put in a fiberglass cast and still non weight bearing. I went to school today and crutched around for longer than regular. I did hit my cast hard because I tripped and the pain was very minimal but there was quite a bit of swelling in my cast. It wasn't extremely tight but it was less roomy than usual. The swelling went down after I elevated my ankle. But I'm just worried if it delays healing . If bones can shift or the bones are delayed in healing
Is this okay? If not do I stop going to school and using my crutches?