I Have A Total Cholesterol Level Of 6.1. I M

I have a total cholesterol level of 6.1. I m 54 years old, five foot seven and a half inches and under 10 stone. My father died of a heart attack aged 74. He did not have high cholesterol but he was overweight and had been a heavy smoker for much of his life. I enjoy a small coffee late on most days. should I cut this out or is it relatively safe given the milk content?