llfbench

LLF-Bench (Learning from Language Feedback Benchmark) is a new evaluation framework designed to assess the ability of AI agents to learn interactively from natural language feedback. Unlike traditional benchmarks that rely on numeric rewards or simple planning, LLF-Bench enables agents to solve a diverse set of sequential decision-making tasks, such as user recommendations, poem writing, navigation, and robot control, using natural language instructions and feedback. The key innovation of LLF-Bench is its focus on teaching agents to learn from verbal feedback, mimicking human learning processes. It incorporates various randomization techniques like paraphrasing and environment randomization to ensure agents are exposed to novel and diverse learning scenarios, helping to measure their ability to adapt and learn from different forms of feedback.

The benchmark provides a unified OpenAI Gym interface for all tasks, allowing for easy configuration of feedback types, such as suggestions, explanations, or performance evaluations. This flexibility makes LLF-Bench an ideal platform for studying how agents respond to different kinds of feedback and for advancing research in interactive learning from language. It aims to bridge the gap between natural language understanding and interactive learning, offering a comprehensive tool to push the development of AI systems that can learn from real-world, human-like communication.