Can large language models really see through someone else’s eyes—acting, asking, and searching in socially efficient ways—or are they bound to a single, data-driven point of view? This is the question behind “Who Sees What? Structured Thought-Action Sequences for Epistemic Reasoning in LLMs,” presented by Luca Annese (BiConnect Lab, Università degli Studi di Milano-Bicocca) at ICSR 2025 – International Conference on Social Robotics.
The work is a collaboration with Sabrina Patania, Silvia Serino, Tom Foulsham, Silvia Rossi, Azzurra Ruggeri, and Dimitri Ognibene.
Read the paper: https://lnkd.in/dGhHG6UK
Why perspective-taking matters for AI (and people)
In everyday interaction, we constantly reason about what others know, see, and believe. We choose questions that reduce uncertainty efficiently; we act while keeping in mind what others can and cannot observe; and we weigh the costs of information-gathering. For LLM-based agents, doing the same requires more than generic chain-of-thought—it calls for epistemic reasoning: tracking who knows what, under partial observability, and planning actions accordingly.
Our approach: structured thought-action sequences
The team introduces a structured pipeline that generates diverse, principled examples of how an agent might think and act when other minds are involved. Rather than offering a single “optimal” script, the pipeline produces multiple thought-action sequences in natural language:
G-type: optimal path from start point to end goal;
E-type: epistemical path until the first informative action;
L-type: local-optimal path, where at each time the best action is considered against other possibilities.
These examples show the agent different ways to tackle different perspective challenges.
These structured sequences serve two purposes:
- Training/conditioning: They provide LLMs with richer demonstrations of epistemic reasoning than a lone “best plan.”
- Evaluation: By contrasting reasonable alternatives, they reveal when an agent mistakes its own knowledge for others’ or overlooks action costs (e.g., asking vs. exploring).
Key takeaways
- Contrast beats single best plans. To improve perspective-taking, LLMs benefit from contrastive examples—side-by-side alternatives that make trade-offs explicit, instead of only the final, “best” course of action.
- Partial observability is still hard. When different agents see different slices of the world, models struggle to keep beliefs aligned over time.
- Action costs matter. Reasoning about how to learn (ask, move, look, wait) requires acknowledging that actions have heterogeneous costs—and that the cheapest step isn’t always the most informative.
- Richer environments are needed. Progress depends on settings where agents must explicitly model others’ beliefs, not just infer them implicitly from text.
What this means for socially aware AI
If we want robots and assistants that collaborate naturally—asking better questions, deferring when appropriate, and exploring efficiently—we must teach them to reason about other minds. The study shows that carefully designed, structured demonstrations help LLMs internalize these skills more reliably than generic prompting.
“Improving ‘perspective-taking’ needs more than the single ‘best’ plan: contrastive examples matter.”
What’s next
The BiConnect team is extending this line of work in three directions:
- Belief-explicit benchmarks: Tasks where agents must represent and update others’ beliefs, not just describe them.
- Cost-aware planning: Integrating heterogeneous action costs directly into the model’s decision-making.
- Open-world scenarios: Building richer, partially observable environments to stress-test socially aware exploration and dialogue.
Acknowledgments
This research was conducted by Luca Annese, Sabrina Patania, Silvia Serino, Tom Foulsham, Silvia Rossi, Azzurra Ruggeri, and Dimitri Ognibene within the BiConnect Lab at Università degli Studi di Milano-Bicocca, and presented at ICSR 2025.