The ability to combine linguistic guidance from others with direct experience is central to human development, enabling safe and rapid learning in new environments. How do people integrate these two sources of knowledge, and how might AI systems? We present a computational framework that models social learning as joint probabilistic inference of structured causal world models given both sensorimotor and linguistic data. Using behavioral and simulation studies of learning across 10 video games, we show how linguistic guidance shapes exploration by reducing risky interactions and speeding up key discoveries, in both humans and models. Most notably, we demonstrate successful cross-embodiment knowledge transfer: both human- and model-generated advice speeds up both human and model learning, revealing how structured, language-compatible representations might enable human-machine collaborative learning.