Is It Safe To Take Synthroid For The Following TSH Levels?

I am not sure if my family physician is treating my thyroid correctly. In 2014 i was prescribed .1MG synthroid while my TSH was averaging about .03 and my T4 was 1.5 - 2.0. By the fall of 2014 I was feeling terrible and by chance called another doctor to refill my prescription and she said to get off of that dose immediately . I asked him about it, and he said that the T4 is more important and that is what he goes by. I got him to lower my dose down to 50mcg which currently brought me to TSH 6.31 with T4 at 1.1 .I thought he would raise my Synthroid level a little since it seems high, but says there is no need to change the dosage. Since he seems to be going by the T4 I m wondering if this is correct. Can you please straighten me out. I m wondering whether to see someone else. Thank you