Aim: Studies have demonstrated neuropsychological deficits across a variety of cognitive domains in depression. These deficits are observable both in major depressive disorder (MDD) and in bipolar disorder (BD) and are present in each phase of the illness, including euthymia. Adverse childhood experiences (ACE) have been associated with an increased risk of developing psychiatric disorders and cognitive deficits. The aim of this study was to assess neuropsychological performances in a sample of MDD and BD patients during a depressive episode compared to healthy controls (HC) and, to investigate if ACE affect the cognitive profiles in the three groups. Methods: Seventy-six BD patients, 57 MDD patients, and 57 HC underwent neuropsychological assessment for cognitive performances through the Brief Assessment of Cognition in Schizophrenia and Wisconsin Card Sorting Test. Results: Both BD and MDD patients obtained significantly lower domain scores across the entire battery compared to HC. Splitting the sample according to exposure to ACE (high and low), the differences observed in the whole sample persisted only in the subsample of those patients exposed to high ACE. Conclusion: This study confirms that cognitive impairment is present both in MDD and BD, albeit in different degrees of severity, and highlights the importance of early stress as a moderator factor when investigating cognitive functions in mood disorders. © 2017 The Authors.