Sometimes My Eyeglasses Cause Pain Surrounding My Ear, Especially The

Sometimes my eyeglasses cause pain surrounding my ear, especially the part behind my ear. My outer ear feels hot and I am very uncomfortable. When I touch my ear its worse. All I can do is take my glasses off for a few hours and eventually it goes away. My glasses are not too tight and this problem is very sporadic, not frequent.