Vision and touch play a critical role in spatial development, facilitating the acquisition of allocentric and egocentric frames of reference, respectively. Previous works have shown that children's ability to adopt an allocentric frame of reference might be impaired by the absence of visual experience during growth. In the current work, we investigated whether visual deprivation also impairs the ability to shift from egocentric to allocentric frames of reference in a switching-perspective task performed in the visual and haptic domains. Children with and without visual impairments from 6 to 13 years of age were asked to visually (only sighted children) or haptically (blindfolded sighted children and blind children) explore and reproduce a spatial configuration of coins by assuming either an egocentric perspective or an allocentric perspective. Results indicated that temporary visual deprivation impaired the ability of blindfolded sighted children to switch from egocentric to allocentric perspective more in the haptic domain than in the visual domain. Moreover, results on visually impaired children indicated that blindness did not impair allocentric spatial coding in the haptic domain but rather affected the ability to rely on haptic egocentric cues in the switching-perspective task. Finally, our findings suggested that the total absence of vision might impair the development of an egocentric perspective in case of body midline-crossing targets.