{"id":1001,"date":"2025-09-15T17:46:53","date_gmt":"2025-09-15T17:46:53","guid":{"rendered":"https:\/\/biconnect.psicologia.unimib.it\/?p=1001"},"modified":"2025-09-16T12:54:28","modified_gmt":"2025-09-16T12:54:28","slug":"who-sees-what-teaching-llms-to-take-anothers-perspective","status":"publish","type":"post","link":"https:\/\/biconnect.psicologia.unimib.it\/index.php\/who-sees-what-teaching-llms-to-take-anothers-perspective\/","title":{"rendered":"Who Sees What? Teaching LLMs to Take Another\u2019s Perspective"},"content":{"rendered":"\n

Can large language models really see through someone else\u2019s eyes<\/em>\u2014acting, asking, and searching in socially efficient ways\u2014or are they bound to a single, data-driven point of view? This is the question behind \u201cWho Sees What? Structured Thought-Action Sequences for Epistemic Reasoning in LLMs,\u201d<\/strong> presented by Luca Annese<\/strong> (BiConnect Lab, Universit\u00e0 degli Studi di Milano-Bicocca) at ICSR 2025 \u2013 International Conference on Social Robotics<\/strong>.
The work is a collaboration with Sabrina Patania, Silvia Serino, Tom Foulsham, Silvia Rossi, Azzurra Ruggeri,<\/strong> and Dimitri Ognibene<\/strong>.<\/p>\n\n\n\n

Read the pape<\/strong>r: https:\/\/lnkd.in\/dGhHG6UK<\/a><\/p>\n\n\n\n

\n\n\n\n

Why perspective-taking matters for AI (and people)<\/h2>\n\n\n\n

In everyday interaction, we constantly reason about what others know, see, and believe<\/strong>. We choose questions that reduce uncertainty efficiently; we act while keeping in mind what others can and cannot observe; and we weigh the costs<\/strong> of information-gathering. For LLM-based agents, doing the same requires more than generic chain-of-thought\u2014it calls for epistemic reasoning<\/strong>: tracking who knows what, under partial observability<\/strong>, and planning actions accordingly.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\n\n\n\n

Our approach: structured thought-action sequences<\/h2>\n\n\n\n
\n
\n

The team introduces a structured pipeline<\/strong> that generates diverse, principled examples<\/strong> of how an agent might think and act when other minds are involved. Rather than offering a single \u201coptimal\u201d script, the pipeline produces multiple thought-action sequences<\/strong> in natural language:<\/p>\n\n\n\n

G-type<\/strong>: optimal path from start point to end goal;<\/p>\n\n\n\n

E-type<\/strong>: epistemical path until the first informative action;<\/p>\n\n\n\n

L-type<\/strong>: local-optimal path, where at each time the best action is considered against other possibilities.<\/p>\n\n\n\n

These examples show the agent different ways to tackle different perspective challenges.<\/p>\n\n\n\n

These structured sequences serve two purposes:<\/p>\n\n\n\n

    \n
  1. Training\/conditioning:<\/strong> They provide LLMs with richer demonstrations of epistemic reasoning than a lone \u201cbest plan.\u201d<\/li>\n\n\n\n
  2. Evaluation:<\/strong> By contrasting reasonable alternatives, they reveal when an agent mistakes its own knowledge for others\u2019<\/strong> or overlooks action costs<\/strong> (e.g., asking vs. exploring).<\/li>\n<\/ol>\n<\/div>\n\n\n\n
    \n
    \"\"<\/figure>\n<\/div>\n<\/div>\n\n\n\n
    \n\n\n\n

    Key takeaways<\/h2>\n\n\n\n
      \n
    • Contrast beats single best plans.<\/strong> To improve perspective-taking, LLMs benefit from contrastive examples<\/strong>\u2014side-by-side alternatives that make trade-offs explicit, instead of only the final, \u201cbest\u201d course of action.<\/li>\n\n\n\n
    • Partial observability is still hard.<\/strong> When different agents see different slices of the world, models struggle to keep beliefs aligned over time.<\/li>\n\n\n\n
    • Action costs matter.<\/strong> Reasoning about how<\/em> to learn (ask, move, look, wait) requires acknowledging that actions have heterogeneous costs<\/strong>\u2014and that the cheapest step isn\u2019t always the most informative.<\/li>\n\n\n\n
    • Richer environments are needed.<\/strong> Progress depends on settings where agents must explicitly model others\u2019 beliefs<\/strong>, not just infer them implicitly from text.<\/li>\n<\/ul>\n\n\n\n
      \n\n\n\n

      What this means for socially aware AI<\/h2>\n\n\n\n

      If we want robots and assistants that collaborate naturally\u2014asking better questions, deferring when appropriate, and exploring efficiently<\/strong>\u2014we must teach them to reason about other minds<\/strong>. The study shows that carefully designed, structured demonstrations<\/strong> help LLMs internalize these skills more reliably than generic prompting.<\/p>\n\n\n\n

      \n

      \u201cImproving \u2018perspective-taking\u2019 needs more than the single \u2018best\u2019 plan: contrastive examples matter<\/strong>.\u201d<\/p>\n<\/blockquote>\n\n\n\n

      \n\n\n\n

      What\u2019s next<\/h2>\n\n\n\n

      The BiConnect team is extending this line of work in three directions:<\/p>\n\n\n\n

        \n
      1. Belief-explicit benchmarks:<\/strong> Tasks where agents must represent and update<\/strong> others\u2019 beliefs, not just describe them.<\/li>\n\n\n\n
      2. Cost-aware planning:<\/strong> Integrating heterogeneous action costs<\/strong> directly into the model\u2019s decision-making.<\/li>\n\n\n\n
      3. Open-world scenarios:<\/strong> Building richer, partially observable environments<\/strong> to stress-test socially aware exploration and dialogue.<\/li>\n<\/ol>\n\n\n\n
        \n\n\n\n

        Acknowledgments<\/h2>\n\n\n\n

        This research was conducted by Luca Annese, Sabrina Patania, Silvia Serino, Tom Foulsham, Silvia Rossi, Azzurra Ruggeri,<\/strong> and Dimitri Ognibene<\/strong> within the BiConnect Lab<\/strong> at Universit\u00e0 degli Studi di Milano-Bicocca<\/strong>, and presented at ICSR 2025<\/strong>.<\/p>\n\n\n\n

        \n\n\n\n

        Slides<\/h2>\n\n\n\n