lilgym

lilGym is a benchmark designed for language-conditioned reinforcement learning in visual environments. It consists of 2,661 human-written, highly compositional natural language statements, each grounded in an interactive visual setting. Example of such a sentence is: “There is a black block as the base of a tower with at least two blocks”. Such statements are paired with executable Python programs representing their meanings, enabling precise reward computation for every possible state. The dataset forms thousands of distinct Markov Decision Processes (MDPs) with varying difficulty levels, providing a complex environment for testing and evaluating reinforcement learning models that integrate both language and visual reasoning.

The lilGym framework is specifically designed to facilitate experiments in language-guided visual reasoning tasks, offering a challenging open problem for researchers. Despite achieving some non-trivial performance with existing models, the benchmark reveals gaps in the current approaches, highlighting the difficulty of learning language-conditioned tasks in dynamic, visual environments.