Graphs abound in our culture, but the brain mechanisms of graphicacy are unknown. Here, using scatterplots, we tested two hypotheses about the brain areas underlying graphicacy. First, at the perceptual level, we hypothesized that the visual processing of scatterplots recycles cortical regions devoted to the perception of the principal axis of objects. Second, at the semantic level, we speculated that the math-responsive network active during mathematical truth judgments should also be involved in graph perception. Using fMRI, we found that graph trend judgement recruits a right lateral occipital area involved in detecting objects’ orientation, as well as a right anterior intraparietal region also recruited during mathematical tasks. Both behavior and brain activity were driven by the t-value, which indexes the graph’s statistical correlation. We suggest that, like literacy and numeracy, graphicacy relies on the recycling of brain areas previously attuned to a similar problem, here the perception of object orientation.