Aromatase activity has commonly been associated with male infertility characterized by testicular dysfunction with low serum testosterone and/or testosterone to estradiol ratio. In this subset of patients, and particularly in those with hypogonadism, elevated levels of circulating estradiol may establish a negative feedback on the hypothalamic-pituitary-testicular axis by suppressing follicle-stimulating hormone (FSH) and luteinizing hormone (LH) production and impaired spermatogenesis. Hormonal manipulation via different agents such as selective estrogen modulators or aromatase inhibitors to increase endogenous testosterone production and improve spermatogenesis in the setting of infertility is an off-label option for treatment. We carried out a systematic review and meta-analysis of the literature of the past 30 years in order to evaluate the benefits of the use of aromatase inhibitors in the medical management of infertile/hypoandrogenic males. Overall, eight original articles were included and critically evaluated. Either steroidal (Testolactone) or nonsteroidal (Anastrozole and Letrozole) aromatase inhibitors were found to statistically improve all the evaluated hormonal and seminal outcomes with a safe tolerability profile. While the evidence is promising, future prospective randomized placebo-controlled multicenter trials are necessary to better define the efficacy of these medications.