People conceptualize space using different spatial reference frames, based either in the body or the environment. Many studies attribute this cognitive diversity to spatial language, but their effects are confounded by differences across cultures and experimental tasks. Here we tested this hypothesis in people and tasks that are directly comparable. Indigenous BaYaka adults in the Congo basin reconstructed simple object arrays from memory and later described the same arrays aloud. Reference frames diverged across modalities in three primary ways. First, they used body-based frames seven times more often in their spatial memory than in their spatial descriptions of the same stimuli. Second, linguistic and non-linguistic responses were uncorrelated across participants, despite substantial variation. Third, each of two factors – spatial aspect and axis – influenced spatial language and memory differently, even in the same individuals. The results show that this fundamental feature of spatial thinking does not reflect patterns of spatial language.