Is It Safe To Have Unprotected Sex While Using Birth Control Pills?

Hi. I have been married for about three weeks and have been using the pill long before that. I know it is safest to use a condom to prevent pregnancy but we sometimes go without it. I want to know if it is safe to have sex during the use of the inactive tablets without a condom.