How Efficient Are Today’s Continual Learning Algorithms?
监督持续学习涉及从不断增长的标签数据流中更新深度神经网络(DNN)。尽管大多数工作都关注克服灾难性遗忘问题，但持续学习的一个重要动机是能够高效更新网络，而不是在训练数据随时间增长时从头开始重新训练。尽管最近的持续学习方法在很大程度上解决了灾难性遗忘问题，但几乎没有人关注这些算法的效率。在这里，我们研究最近用于增量班级学习的方法，并证明许多方法在计算、记忆和存储方面都是极其高效的。有些方法甚至需要更多的计算量，比从头开始训练还要糟糕！我们 argue that 为了使持续学习具有实际适用性，研究社区不能忽视这些算法所使用的资源。持续学习不仅仅是减轻灾难性遗忘问题。
Supervised Continual learning involves updating a deep neural network (DNN) from an ever-growing stream of labeled data. While most work has focused on overcoming catastrophic forgetting, one of the major motivations behind continual learning is being able to efficiently update a network with new information, rather than retraining from scratch on the training dataset as it grows over time. Despite recent continual learning methods largely solving the catastrophic forgetting problem, there has been little attention paid to the efficiency of these algorithms. Here, we study recent methods for incremental class learning and illustrate that many are highly inefficient in terms of compute, memory, and storage. Some methods even require more compute than training from scratch! We argue that for continual learning to have real-world applicability, the research community cannot ignore the resources used by these algorithms. There is more to continual learning than mitigating catastrophic forgetting.
https://arxiv.org/abs/2303.18171
https://arxiv.org/pdf/2303.18171