Me An My Be Live With His Mom. Basically Renting

Me an my be live with his mom. Basically renting a room in her den. I have two cats (one long hair and one short) an they have a cat and two dogs. Currently my cats hide out in our room and there litter box is there. I change it frequently an do what I can about the hair. He is about to have a son via a past relationship and we will be getting full custody an bringing home a new born baby at the end of November. Both of us are looking at first time parenting without the option to give our baby a nursery where it can be cat free. I've tried researching about this an have found pet hair won't be a problem with regular cleaning but what about the litter box being in the same room as the baby? Is that going to put the child at risk in any way, shape or form?