Does Coming Off Clobazam Cause Epilepsy To Reoccur?

I have had epilepsy for 5 months now, I was perscribed Epilim chrono, 1000mg and also Clobazam to help me sleep. However, I am finding that the Clobazam are giving me side effects such as forgetfulness, I feel tired, sluggish and its hard to wake up in the morning, I'm also experiencing a decreased sex drive and generally not myself. I was wondering if I just stopped taking the Clobazam would this effect my Epilepsy? Or have any other effects on me?