Is It Safe To Leave The Children With A Drug Addict?

I have twin grandbabies and the father is taking this drug and keeping it very quiet. Is this something that I need to be concerned about? Do I need to have someone contact the judge because the mother keeps the babies most of the time but then the father keeps the babies from Thursday to Monday night every other week, and it scares me. He has a serious arrest record and has been convicted for felony assault etc. what do you think?