Overview
Robots are increasingly expected to collaborate with people in dynamic, ambiguous environments—from assisting with household tasks to supporting decision-making in workplaces. For such collaboration, it is not enough for robots to execute commands: they must understand perspectives, intentions, and shared goals. In this line of research, we investigate how Large Language Models (LLMs), combined with active perception and structured reasoning, can support perspective-taking, causal understanding, and adaptive dialogue in embodied settings.
What we study
- Perspective-taking in collaborative tasks We study how LLMs can model what another agent sees, knows, or intends. Using extended versions of the classic Director Task, we evaluate whether LLM-based agents can resolve ambiguity in partially observable environments by integrating reasoning (ReAct-style thought sequences) with active exploration. This allows us to test not only linguistic abilities but also embodied coordination skills, bridging toward theory of mind.
- Developmental insights for AI Inspired by Selman’s developmental stages of perspective-taking, we simulate child-like reasoning in LLMs, from egocentric viewpoints to more advanced reciprocal and societal perspectives. This developmental lens allows us to probe how perspective-taking emerges in LLMs and to benchmark their collaborative behaviour against human developmental trajectories.
- Structured reasoning and planning We integrate symbolic planners with LLMs to generate structured examples of “thought–action” sequences. These help LLMs learn not only what to do but why, grounding decisions in perspective-aware reasoning. This hybrid approach addresses current limitations in disambiguating hidden or occluded states and supports more consistent collaborative strategies.
- Social continual learning Beyond isolated tasks, we develop frameworks where agents continually refine their causal knowledge through interaction. In the SCOOP project, we explore how LLMs can ask meaningful “why” and “what if” questions to an oracle or a human partner, building reusable causal models across tasks. This supports robust decision-making in dynamic, partially observable environments.
Why it matters
Embedding perspective-taking and causal reasoning into LLM-powered robots has two transformative outcomes:
- For AI and robotics: It leads to socially-aware, adaptive robots that can collaborate effectively, understand ambiguity, and generalise across tasks.
- For cognitive science: It provides new computational models of perspective-taking and collaborative learning, grounded in developmental psychology and tested in embodied interaction.
Methods at a glance
- Framework ReAct (Reason Act) and multimodal perception;
- Tasks Modified Director Tasks, ecological household scenarios, collaborative planning;
- Architectures Hybrid LLM and planner systems, LLMs with causal graph modules, oracle-aided continual learning;
- Evaluation Task success, efficiency, clarification strategies, developmental stage alignment, robustness in dynamic environments.
Project Members
Dr. Luca Annese
luca.annese1@unimib.it
Dr. Sabrina Patania, PhD
sabrina.patania@unimib.it
Dr. Anna Lambiase
a.lambiase3@campus.unimib.it
Dr. Anita Pellegrini
a.pellegrini19@campus.unimib.it
Prof. Silvia Serino
silvia.serino@unimib.it
Prof. Tom Foulsham
foulsham@essex.ac.uk
Prof. Dimitri Ognibene
dimitri.ognibene@unimib.it
References
- Annese, L., Patania, S., Serino, S., Foulsham, T., Rossi, S., Ruggeri, A., & Ognibene, D. (2025). Who Sees What? Structured Thought-Action Sequences for Epistemic Reasoning in LLMs. Proceedings of ICSR 2025 .
- Patania, S., Annese, L., Lambiase, A., Pellegrini, A., Foulsham, T., Ruggeri, A., Rossi, S., Serino, S., & Ognibene, D. (2025). Growing Perspectives: Modelling Embodied Perspective Taking and Inner Narrative Development Using LLMs. Proceedings of ICDL 2025 .
- Patania, S., Annese, L., Pellegrini, A., Serino, S., Lambiase, A., Pallonetto, L., Rossi, S., Colombani, S., Foulsham, T., Ruggeri, A., & Ognibene, D. (2025). PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision. Proceedings of IAS-19 .
- Ognibene, D., Patania, S., Annese, L., Koyuturk, C., Ruggeri, A., Colombani, S., Garzotto, F., Vizzari, G. (2025). SCOOP: A Framework for Proactive Collaboration and Social Continual Learning through Natural Language Interaction and Causal Reasoning. Poster at RLDM 2025 .