Numerous studies agree that time is represented in spatial terms in the brain. Here we investigate how a deficit in orienting attention in space influences the ability to mentally travel in time, that is to recall the past and anticipate the future. Right brain-damaged patients, with (RBD-N+) and without neglect (RBD-N-), and healthy controls (HC) were subjected to a Mental Time Travel (MTT) task. Participants were asked to project themselves in time to past, present or future (i.e., self-projection) and, for each self-projection, to judge whether events were located relatively in the past or the future (i.e., self-reference). The MTT-task was performed before and after a manipulation, through prismatic adaptation (PA), inducing a leftward shift of spatial attention. Before PA, RBD-N+ were slower for future than for past events, whereas RBD-N- and HC responded similarly to past and future events. A leftward shift of spatial attention by PA reduced the difference in past/future processing in RBD-N+ and fastened RBD-N- and HC's response to past events. Assuming that time concepts, such as past/future, are coded with a left-to-right order on a mental time line (MTL), a recursive search of future-events can explain neglect patients' performance. Improvement of the spatial deficit following PA reduces the recursive search of future events on the rightmost part of the MTL, facilitating exploration of past events on the leftmost part of the MTL, finally favoring the correct location of past and future events. In addition, the study of the anatomical correlates of the temporal deficit in mental time travel through voxel-based lesion-symptom mapping showed a correlation with a lesion located in the insula and in the thalamus. These findings provide new insights about the inter-relations of space and time, and can pave the way to a procedure to rehabilitate a deficit in these cognitive domains.