Hi Jayan , 
Male factor is the commonest cause for 
infertility . 
Varicocele alone is not a cause for infertility. Only when all other possible causes have been ruled out, varicocele should be blamed.
I assume that repeated 
semen analysis showed a poor count & quality. 
Before going for higher management try to remove the cause for the poor sperm quality. Only 
healthy sperm can give a healthy baby. 
Kindly confirm if
1.your blood sugars/ Serum Thyroid & Prolactin are normal  
2. Reduce body weight , if your on to 
overweight 3. Avoid smoking( even 
passive smoking )  / alcohol/ any other substance abuse
4. Have cold water bath 2-3 time / day if possible
5. Try methods to reduce mental stress( I know it's easy to say , but tough to practice)- by 
meditation , Yoga
6. Kindly repeat semen analysis from an infertility centre- with abstinence not less than 3 days & not more than 6 days.
If they report any infection / WBC/ Pus cells above normal , you will need a course of antibiotics for 2 weeks.
Expecting your wife's tubal factor & ovulation factors have been ruled out.
Since you are married for 7 yrs, after this basic evaluation & treatment you can try IUI with good hormonal induction & support. 
In IUI they wash the semen & separate the healthy sperms & keep it close to the waiting egg. Pregnancy chances are 4 times.
Even after 3 IUI if you don't get a positive result you can go for IVF, where only the healthy sperm is selected .
Don't worry , The lowest count for which I could get a pregnancy was 5 million.
All the best 
Dr.Balakrishnan