While the discussion on the foundations of social understanding mainly revolves around the notions of empathy, affective mentalizing, and cognitive mentalizing, their degree of overlap versus specificity is still unclear. We took a meta-analytic approach to unveil the neural bases of cognitive mentalizing, affective mentalizing, and empathy, both in healthy individuals and pathological conditions characterized by social deficits such as schizophrenia and autism. We observed partially overlapping networks for cognitive and affective mentalizing in the medial prefrontal, posterior cingulate, and lateral temporal cortex, while empathy mainly engaged fronto-insular, somatosensory, and anterior cingulate cortex. Adjacent process-specific regions in the posterior lateral temporal, ventrolateral, and dorsomedial prefrontal cortex might underpin a transition from abstract representations of cognitive mental states detached from sensory facets to emotionally-charged representations of affective mental states. Altered mentalizing-related activity involved distinct sectors of the posterior lateral temporal cortex in schizophrenia and autism, while only the latter group displayed abnormal empathy related activity in the amygdala. These data might inform the design of rehabilitative treatments for social cognitive deficits.