MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities
检测异常（OOO）样本对于在关键应用领域（如自动驾驶和机器人辅助手术）部署机器学习模型非常重要。现有的研究主要集中在图像数据的单模态场景。然而，现实世界应用是多模态的，因此从多个模态的信息来增强 OOO 检测的有效性至关重要。为了建立更真实的多模态 OOO 检测的基础，我们引入了世界上第一个 benchmark，MultiOOD，它具有多样化的数据集大小和不同的模态组合。我们首先评估现有的单模态 OOO 检测算法在 MultiOOD，观察到仅仅增加额外的模态就能带来很大的改进。这强调了利用多个模态进行 OOO 检测的重要性。基于在 ID 和 OOO 数据之间的模态预测差异以及其与 OOO 性能的强烈相关性，我们提出了 Agree-to-Disagree（A2D）算法，以便在训练过程中鼓励这种差异。此外，我们引入了一种名为 NP-Mix 的全新异常合成方法，它通过利用最近邻类的信息来探索更广泛的特征空间，补充 A2D，从而增强 OOO 检测性能。在 MultiOOD 的大量实验中，训练使用 A2D 和 NP-Mix 大大改进了现有的 OOO 检测算法。我们的源代码和 MultiOOD 基准可以在这个链接处找到：https://url.com/
Detecting out-of-distribution (OOD) samples is important for deploying machine learning models in safety-critical applications such as autonomous driving and robot-assisted surgery. Existing research has mainly focused on unimodal scenarios on image data. However, real-world applications are inherently multimodal, which makes it essential to leverage information from multiple modalities to enhance the efficacy of OOD detection. To establish a foundation for more realistic Multimodal OOD Detection, we introduce the first-of-its-kind benchmark, MultiOOD, characterized by diverse dataset sizes and varying modality combinations. We first evaluate existing unimodal OOD detection algorithms on MultiOOD, observing that the mere inclusion of additional modalities yields substantial improvements. This underscores the importance of utilizing multiple modalities for OOD detection. Based on the observation of Modality Prediction Discrepancy between in-distribution (ID) and OOD data, and its strong correlation with OOD performance, we propose the Agree-to-Disagree (A2D) algorithm to encourage such discrepancy during training. Moreover, we introduce a novel outlier synthesis method, NP-Mix, which explores broader feature spaces by leveraging the information from nearest neighbor classes and complements A2D to strengthen OOD detection performance. Extensive experiments on MultiOOD demonstrate that training with A2D and NP-Mix improves existing OOD detection algorithms by a large margin. Our source code and MultiOOD benchmark are available at this https URL.
https://arxiv.org/abs/2405.17419
https://arxiv.org/pdf/2405.17419.pdf