Hello and welcome to HCM,
Food safety guidelines specify that perishable foods such as milk, eggs, meat,etc should not be left unrefrigerated any longer than possible with a maximum of 2 hours unrefrigerated being the cut-off. Milk, especially, is an excellent medium for the growth of bacteria and all of the time the milk was not in the safe zone (under 41 degrees) and in the danger zone for
bacterial growth (41-140 degrees) you are taking a chance at getting food
poisoning. Despite putting the milk back in the refrigerator, the refrigerator will not "kill" any bacteria that have already grown there.
So, the answer to your question is, if the milk was left unrefrigerated at any point in time longer than two hours at above 41 degrees it should be thrown out to avoid the chance of food poisoning.
Have a good day! Kathryn J. Shattler, MS,RDN