This long-term project builds upon our 13-year history of pioneering work in continual and lifelong learning. We’re developing the next generation of continual learning algorithms that acquire and reuse modular skills to solve and zero-shot long-horizon tasks in open-world settings.
Deep representations are likely to be more transferrable for continual learning if they represent reusable cohesive modules within the deep network (Mendez & Eaton, 2023),
and so we investigated mechanisms for compositional lifelong learning for both object recognition (Mendez & Eaton, 2021) and RL (Mendez et al., 2022). We showed that lifelong learning
using compositional representations dramatically outperforms non-compositional representations, and enables
zero-shot generalization to new tasks that are combinations of known modules (Mendez et al., 2022; Hussing et al., 2024). We are currently expanding these compositional representations to develop modular skills that can be dynamically combined, toward the goal of solving long-horizon RL problems. We’re also leveraging ideas for out-of-distribution detection to enable task-agnostic lifelong learning in non-stationary and open-world settings (Gummadi et al., 2022; Gummadi et al., 2024).
Our general-purpose framework for compositional continual or lifelong learning.
Offline reinforcement learning (RL) is a promising direction that allows RL agents
to pre-train on large datasets, avoiding the recurrence of expensive data collection.
To advance the field, it is crucial to generate large-scale datasets. Compositional
RL is particularly appealing for generating such large datasets, since 1) it permits
creating many tasks from few components, 2) the task structure may enable trained
agents to solve new tasks by combining relevant learned components, and 3) the
compositional dimensions provide a notion of task relatedness. This paper provides
four offline RL datasets for simulated robotic manipulation created using the 256
tasks from CompoSuite (Mendez et al., 2022a). Each dataset is collected from an
agent with a different degree of performance, and consists of 256 million transitions.
We provide training and evaluation settings for assessing an agent’s ability to learn
compositional task policies. Our benchmarking experiments show that current offline
RL methods can learn the training tasks to some extent and that compositional
methods outperform non-compositional methods. Yet current methods are unable
to extract the compositional structure to generalize to unseen tasks, highlighting a
need for future research in offline compositional RL.
@inproceedings{Hussing2024RoboticManipulation,author={Hussing, Marcel and Mendez-Mendez, Jorge and Singrodia, Anisha and Kent, Cassandra and Eaton, Eric},year={2024},title={Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning},booktitle={1st Reinforcement Learning Conference (RLC)},}
Despite outstanding semantic scene segmentation in closed-worlds,
deep neural networks segment novel instances poorly, which is required for
autonomous agents acting in an open world. To improve out-of-distribution
(OOD) detection for segmentation, we introduce a metacognitive approach in
the form of a lightweight module that leverages entropy measures,
segmentation predictions, and spatial context to characterize the
segmentation model’s uncertainty and detect pixel-wise OOD data in real-time.
Additionally, our approach incorporates a novel method of generating
synthetic OOD data in context with in-distribution data, which we use to
fine-tune existing segmentation models with maximum entropy training. This
further improves the metacognitive module’s performance without requiring
access to OOD data while enabling compatibility with established pre-trained
models. Our resulting approach can reliably detect OOD instances in a scene,
as shown by state-of-the-art performance on OOD detection for semantic
segmentation benchmarks.
@inproceedings{Gummadi2024Metacognitive,author={Gummadi, Meghna and Kent, Cassandra and Schmeckpeper, Karl and Eaton, Eric},year={2024},title={A Metacognitive Approach to Out-of-Distribution Detection for Segmentation},booktitle={International Conference on Robotics and Automation (ICRA)},}
A major goal of artificial intelligence (AI) is to create an agent capable of acquiring a
general understanding of the world. Such an agent would require the ability to continually
accumulate and build upon its knowledge as it encounters new experiences. Lifelong or
continual learning addresses this setting, whereby an agent faces a continual stream of
problems and must strive to capture the knowledge necessary for solving each new task it
encounters. If the agent is capable of accumulating knowledge in some form of compositional
representation, it could then selectively reuse and combine relevant pieces of knowledge to
construct novel solutions. Despite the intuitive appeal of this simple idea, the literatures
on lifelong learning and compositional learning have proceeded largely separately. In an
effort to promote developments that bridge between the two fields, this article surveys their
respective research landscapes and discusses existing and future connections between them.
@article{Mendez2023CompositionalSurvey,author={Mendez, Jorge and Eaton, Eric},year={2023},title={How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition},journal={Transactions on Machine Learning Research},month=jun,}
Humans commonly solve complex problems by decomposing them into easier
subproblems and then combining the subproblem solutions. This type of compositional
reasoning permits reuse of the subproblem solutions when tackling future
tasks that share part of the underlying compositional structure. In a continual or
lifelong reinforcement learning (RL) setting, this ability to decompose knowledge
into reusable components would enable agents to quickly learn new RL tasks by
leveraging accumulated compositional structures. We explore a particular form of
composition based on neural modules and present a set of RL problems that intuitively
admit compositional solutions. Empirically, we demonstrate that neural
composition indeed captures the underlying structure of this space of problems.
We further propose a compositional lifelong RL method that leverages accumulated
neural components to accelerate the learning of future tasks while retaining
performance on previous tasks via off-line RL over replayed experiences.
@inproceedings{Mendez2022ModularLifelongRL,author={Mendez, Jorge and van Seijen, Harm and Eaton, Eric},year={2022},title={Modular lifelong reinforcement learning via neural composition},booktitle={Proceedings of the International Conference on Learning Representations (ICLR)},}
We present CompoSuite, an open-source simulated robotic manipulation benchmark for compositional
multi-task reinforcement learning (RL). Each CompoSuite task requires a particular robot arm to
manipulate one individual object to achieve a task objective while avoiding an obstacle. This
compositional definition of the tasks endows CompoSuite with two remarkable properties. First,
varying the robot/object/objective/obstacle elements leads to hundreds of RL tasks, each of which
requires a meaningfully different behavior. Second, RL approaches can be evaluated specifically for
their ability to learn the compositional structure of the tasks. This latter capability to functionally
decompose problems would enable intelligent agents to identify and exploit commonalities between
learning tasks to handle large varieties of highly diverse problems. We benchmark existing single-task,
multi-task, and compositional learning algorithms on various training settings, and assess their
capability to compositionally generalize to unseen tasks. Our evaluation exposes the shortcomings of
existing RL approaches with respect to compositionality and opens new avenues for investigation.
@inproceedings{Mendez2022CompoSuite,author={Mendez, Jorge and Hussing, Marcel and Gummadi, Meghna and Eaton, Eric},year={2022},title={CompoSuite: A compositional reinforcement learning benchmark},booktitle={Conference on Lifelong Learning Agents (CoLLAs)},}
While deep neural networks (DNNs) have achieved impressive classification performance in
closed-world learning scenarios, they typically fail to generalize to unseen categories in dynamic open-world
environments, in which the number of concepts is unbounded. In contrast, human and animal learners
have the ability to incrementally update their knowledge by recognizing and adapting to novel
observations. In particular, humans characterize concepts via exclusive (unique) sets of essential
features, which are used for both recognizing known classes and identifying novelty. Inspired by
natural learners, we introduce a Sparse High-level-Exclusive, Low-level-Shared feature representation
(SHELS) that simultaneously encourages learning exclusive sets of high-level features and essential,
shared low-level features. The exclusivity of the high-level features enables the DNN to automatically
detect out-of-distribution (OOD) data, while the efficient use of capacity via sparse low-level features
permits accommodating new knowledge. The resulting approach uses OOD detection to perform
class-incremental continual learning without known class boundaries. We show that using SHELS for
novelty detection results in statistically significant improvements over state-of-the-art OOD detection
approaches over a variety of benchmark datasets. Further, we demonstrate that the SHELS model
mitigates catastrophic forgetting in a class-incremental learning setting, enabling a combined novelty
detection and accommodation framework that supports learning in open-world settings.
@inproceedings{Gummadi2022SHELS,author={Gummadi, Meghna and Kent, Cassandra and Mendez, Jorge and Eaton, Eric},year={2022},title={SHELS: Exclusive feature sets for novelty detection and continual learning without class boundaries},booktitle={Conference on Lifelong Learning Agents (CoLLAs)},}
A hallmark of human intelligence is the ability to construct
self-contained chunks of knowledge and adequately reuse them
in novel combinations for solving different yet structurally
related problems. Learning such compositional structures has
been a significant challenge for artificial systems, due to
the combinatorial nature of the underlying search problem.
To date, research into compositional learning has largely
proceeded separately from work on lifelong or continual
learning. We integrate these two lines of work to present a
general-purpose framework for lifelong learning of
compositional structures that can be used for solving a
stream of related tasks. Our framework separates the learning
process into two broad stages: learning how to best combine
existing components in order to assimilate a novel problem,
and learning how to adapt the set of existing components to
accommodate the new problem. This separation explicitly
handles the trade-off between the stability required to
remember how to solve earlier tasks and the flexibility
required to solve new tasks, as we show empirically in an
extensive evaluation.
@inproceedings{Mendez2021Lifelong,title={Lifelong learning of compositional structures},author={Mendez, Jorge and Eaton, Eric},booktitle={International Conference on Learning Representations},year={2021},}
