Can Occasional High Blood Sugar Spikes After A Missed Dose, Cause Lasting Damage?

I m 70 years old and have been on Metformin and Glipizide for about 5 years. I usually don t take readings on a daily basis as I ve learned over time what to expect with what I eat and do as far as exercise. We just had a family gathering and I did consume a few things that are not what I should be eating and remembered , with the Grandkids all playing with me, I forgot to take the evening dose of meds. When I took a reading 40 minutes after the meal it read 320....wow. After taking the meds and drinking a couple glasses of water I was able to get it down to a reasonable level of 178. Tonight, my reading was 170 as I sit here watching the football game. My question is; what kind of damage can a once in a while high sugar level cause, and should I not have that piece of Apple Pie with Vanilla ice cream ?