Analysis of the bat viruses most closely related to SARS-CoV-2 indicated that the virus probably required limited adaptation to spread in humans. Nonetheless, since its introduction in human populations, SARS-CoV-2 must have been subject to the selective pressure imposed by the human immune system. We exploited the availability of a large number of high-quality SARS-CoV-2 genomes, as well as of validated epitope predictions, to show that B cell epitopes in the spike glycoprotein (S) and in the nucleocapsid protein (N) have higher diversity than nonepitope positions. Similar results were obtained for other human coronaviruses and for sarbecoviruses sampled in bats. Conversely, in the SARS-CoV-2 population, epitopes for CD4+ and CD8+ T cells were not more variable than nonepitope positions. A significant reduction in epitope variability was instead observed for some of the most immunogenic proteins (S, N, ORF8 and ORF3a). Analysis over longer evolutionary time frames indicated that this effect is not due to differential constraints. These data indicate that SARS-CoV-2 evolves to elude the host humoral immune response, whereas recognition by T cells is not actively avoided by the virus. However, we also found a trend of lower diversity of T cell epitopes for common cold coronaviruses, indicating that epitope conservation per se is not directly linked to disease severity. We suggest that conservation serves to maintain epitopes that elicit tolerizing T cell responses or induce T cells with regulatory activity.