I'm A Man With A Quarter Sized Lump In My

I'm a man with a quarter sized lump in my left breast and it hurts alittle.My GP says it's probably a fat deposit and don't worry..I think he's nuts...I'd like to know if I should see a doctor and what type?I think I should get it removed....But then again I'm a roofer....What do you think...Thanx