I Have A Lipoma On My Left Back Near Shoulder

I have a lipoma on my left back near shoulder blade, its there 4yrs its now 6cm it moves a bit and is semi soft isnt painful, but since doc has been pushing and feeling it it hurts around it..im very worried as gp is sending me to see a specialist because its not well difined and big..im so worried its cancer. Plz help put my mind at easy.