Meta-Learning Parameterized First-Order Optimizers using Differentiable Convex Optimization
在机器学习和控制中，传统的优化方法在很大程度上依赖于第一阶更新规则。为特定任务选择正确的方法和超参数通常需要进行试错或从业者直觉的选择，推动了元学习领域的发展。我们提出了一个元学习框架，该框架的内循环优化步骤涉及解决一个可微的凸优化问题(DCO)。我们展示了这种方法的理论吸引力，通过展示它可以实现一类线性最小二乘问题的一步优化，只要元学习器足够接触到类似任务。 DCO更新规则的各种实例在一系列代表性实验设置上与传统的优化器进行了比较。
Conventional optimization methods in machine learning and controls rely heavily on first-order update rules. Selecting the right method and hyperparameters for a particular task often involves trial-and-error or practitioner intuition, motivating the field of meta-learning. We generalize a broad family of preexisting update rules by proposing a meta-learning framework in which the inner loop optimization step involves solving a differentiable convex optimization (DCO). We illustrate the theoretical appeal of this approach by showing that it enables one-step optimization of a family of linear least squares problems, given that the meta-learner has sufficient exposure to similar tasks. Various instantiations of the DCO update rule are compared to conventional optimizers on a range of illustrative experimental settings.
https://arxiv.org/abs/2303.16952
https://arxiv.org/pdf/2303.16952