DualContrast: Unsupervised Disentangling of Content and Transformations with Implicit Parameterization
无监督地分离内容和变换最近引起了很多研究, 因为它们在解决下游无监督任务(如聚类、对齐和形状分析)方面的有效性。 这个问题在分析面向形状的现实生活中科学图像数据集方面尤为重要, 这些数据集的下游任务具有重要性。 现有的工作通过明确地参数化变换因素来解决这个问题, 明显减少了它们的表达性。 此外, 在无法直接参数化变换的情况下, 它们不适用。 类似于这样的明确方法之外的一种选择是自监督方法与数据增强, 它隐含地解开变换和内容。 我们证明了带有数据增强的自监督方法在现实场景中的内容和变换的低分离度。 因此, 我们开发了一种新的自监督方法, DualContrast, 特别针对形状集中的图像数据集进行内容和解变换的无需监督方法。 我们的广泛实验展示了DualContrast在现有自监督方法和明确参数化方法上的优越性。 我们利用DualContrast将蛋白质身份和蛋白质构象从细胞3D蛋白质图像中解开。 此外,我们还使用DualContrast解开MNIST中的变换、Linemod Object数据集中的视角变换以及Starmen数据集中的人体运动变形。
Unsupervised disentanglement of content and transformation has recently drawn much research, given their efficacy in solving downstream unsupervised tasks like clustering, alignment, and shape analysis. This problem is particularly important for analyzing shape-focused real-world scientific image datasets, given their significant relevance to downstream tasks. The existing works address the problem by explicitly parameterizing the transformation factors, significantly reducing their expressiveness. Moreover, they are not applicable in cases where transformations can not be readily parametrized. An alternative to such explicit approaches is self-supervised methods with data augmentation, which implicitly disentangles transformations and content. We demonstrate that the existing self-supervised methods with data augmentation result in the poor disentanglement of content and transformations in real-world scenarios. Therefore, we developed a novel self-supervised method, DualContrast, specifically for unsupervised disentanglement of content and transformations in shape-focused image datasets. Our extensive experiments showcase the superiority of DualContrast over existing self-supervised and explicit parameterization approaches. We leveraged DualContrast to disentangle protein identities and protein conformations in cellular 3D protein images. Moreover, we also disentangled transformations in MNIST, viewpoint in the Linemod Object dataset, and human movement deformation in the Starmen dataset as transformations using DualContrast.
https://arxiv.org/abs/2405.16796
https://arxiv.org/pdf/2405.16796.pdf