Contrastive Multiview Coding
Abstract
Humans view the world through many sensory channels, e.g., the longwavelength light channel, viewed by the left eye, or the highfrequency vibrations channel, viewed by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a “dog” can be seen, heard, and felt). We hypothesize that a powerful representation is one that models viewinvariant factors. Based on this hypothesis, we investigate a contrastive coding scheme, in which a representation is learned that aims to maximize mutual information between different views but is otherwise compact. Our approach scales to any number of views, and is viewagnostic. The resulting learned representations perform above the state of the art for downstream tasks such as object classification, compared to formulations based on predictive learning or single view reconstruction, and improve as more views are added. Code and reference implementations are released on our project page: http://github.com/HobbitLong/CMC/.
1 Introduction
A foundational idea in coding theory is to learn compressed representations that nonetheless can be used to reconstruct the raw data. This idea shows up in contemporary representation learning in the form of autoencoders [50] and generative models [30, 20], which try to represent a data point or distribution as losslessly as possible. Yet lossless representation might not be what we really want, and indeed is trivial to achieve – the raw data itself is a lossless representation. What we might instead prefer is to keep the “good” information (signal) and throw away the rest (noise). How can we identify what information is signal and what is noise?
To an autoencoder, or a maximum likelihood generative model, a bit is a bit (since the goal is to maximize the log likelihood of the data). No one bit is better than any other. Our conjecture in this paper is that some bits are in fact better than others. Some bits code important properties like semantics, physics, and geometry, while others code attributes that we might consider less important, like incidental lighting conditions or thermal noise in a camera’s sensor.
We hypothesize that the good bits are the ones that are shared between multiple views of the world, for example between multiple sensory modalities like vision, sound, and touch. Under this perspective “presence of dog” is good information, since dogs can be seen, heard, and felt, but “camera pose” is bad information, since a camera’s pose has little or no effect on the acoustic and tactile properties of the imaged scene. There is significant evidence in the cognitive science and neuroscience literature that such crossview representations are encoded across different regions of the brain e.g. [12, 25].
Our goal is therefore to learn representations that capture information shared between multiple sensory views but that are otherwise compact (i.e. throw away the bad information). To do so, we employ contrastive learning, where we learn a feature embedding such that views of the same scene map to nearby points while views of different scenes map to far apart points. In particular, we adapt the recently proposed method of Contrastive Predictive Coding (CPC) [46], except we simplify it – removing the recurrent network – and generalize it – show how to apply it to arbitrary collections of views, rather than just to temporal predictions. In reference to CPC, we term our method Contrastive Multiview Coding (CMC). The contrastive objective in our formulation, as in CPC, is based on Noise Contrastive Estimation (NCE) [21]. This objective can be understood as attempting to maximize the mutual information between the representations of each view.
We intentionally leave “good bits” only loosely defined and treat its definition as an empirical question. Ultimately, the proof is in the pudding: we consider a representation to be good if it makes subsequent problem solving easy, on tasks of human interest. For example, a useful representation of images might be a feature space in which it is easy to learn to recognize objects. We therefore evaluate our method by testing if the learned representations transfer well to standard semantic recognition tasks. On several benchmark tasks, our method achieves state of the art results, compared to other methods for unsupervised representation learning. We additionally find that the quality of the representation improves as a function of the number of views used for training. Finally, we compare the contrastive formulation of multiview learning to the recently popular approach of crossview prediction, and find that in headtohead comparisons, the contrastive approach learns stronger representations.
The core ideas that we build on: contrastive learning, mutual information maximization, and deep representation learning, are not new and have been explored in the literature on representation and multiview learning [34, 59, 2]. Our main contribution is to set up a framework to extend these ideas to any number of views, and we show the resulting significant benefits to the learned representation, in terms of tasks such as object recognition. Fig. 4 gives a pictorial overview of our framework for the different learning tasks we consider in this paper, to learn representations across datasets with different sets of views.
Our main contributions are:

We apply contrastive learning to the multiview setting, where we learn representations that attempt to maximizes mutual information between different views of the same scene (e.g., between different image channels, or different modalities).

Our approach yields representations that outperform the stateoftheart in selfsupervised learning in headtohead comparisons.

We compare the contrastive objective to crossview prediction, finding an advantage to the contrastive approach.

We extend the framework to learn from more than two views, and show that as the number of views increases, the quality of the learned representation improves.
2 Background
Unsupervised representation learning is about learning transformations of the data that make subsequent problem solving easier [5]. This field has a long history, starting with classical methods with well established algorithms, such as principal components analysis (PCA [29]) and independent components analysis (ICA [26]). These methods tend to learn representations that focus on lowlevel variations in the data, which are not very useful from the perspective of downstream tasks such as object recognition.
Representations better suited to such tasks have been learnt using deep neural networks, starting with seminal techniques such as Boltzmann machines [54, 50], autoencoders [23], variational autoencoders [30], generative adversarial networks [20] and autoregressive models [45]. Numerous other works exist, for a review see [5]. A powerful family of models for unsupervised representations are collected under the umbrella of “selfsupervised” learning [62, 61, 28, 57, 48]. In these models, an input to the model is transformed into an output , which is supposed to be close to another signal , which itself is related to in some meaningful way. Examples of such pairs are: luminance and chrominance color channels of an image [62], patches from a single image [46], modalities such as vision and sound [47] or the frames of a video [57]. Clearly, such examples are numerous in the world, and provides us with nearly infinite amounts of training data: this is one of the appeals of this paradigm.
Closely related to selfsupervised learning is the idea of multiview learning, which is a general term involving many different approaches such as cotraining [6], multikernel learning [11] and metric learning [4]; for comprehensive surveys please see [59, 34]. Nearly all existing works have dealt with one or two views such as video or image/sound. However, in many situations, many more views are available to provide training signals for any representation.
The objective functions used to train deep learning based representations in many of the above methods are either reconstructionbased loss functions such as Euclidean losses in different norms e.g. [27], adversarial loss functions [20] that learn the loss in addition to the representation, or contrastive losses e.g. [21, 24, 46, 2] that take advantage of the cooccurence of multiple views. Another recently introduced novel objective function is instance discrimination [58]. In this work, we compare the two most commonly used objectives: predictive and contrastive.
The prior works most similar to our own (and inspirational to us) are Contrastive Predictive Coding (CPC) [46] and
Deep InfoMax [22]. These two methods, like ours, learn representations by contrasting between congruent and incongruent representations of a scene, and are motivated as forms of infomax learning. CPC learns from two views – the past and future – and is applicable to sequential data. Deep Infomax [24] considers the two views to be the input to a neural network and its output. These two methods share the same mathematical objective, but differ in the definition of the views. Our technical method is also highly related, but differs in the following ways: we extend the objective to the case of more than two views; and we use a loss function which more closely follows the original method of noise contrastive estimation [21] (See details in Section 3.5). Although CPC, Deep InfoMax, and the present paper are all very similar at the mathematical level, they each explore a different set of view definitions, architectures, and application settings, and each contributes its own unique empirical investigation of this paradigm of representation learning.
Concurrent Work. Concurrently with our work two new papers have appeared on arXiv in the last few weeks: Deep Infomax++[3] and CPC++[22]. The former extends Deep InfoMax to the “multiview” setting where the two views are different data augmentations of a single data point. The latter expands on CPC [46] by using bidirectional prediction and larger scale networks. Both papers achieve impressive performance gains primarily by using larger architectures. In the present paper we have compared only to the original CPC [46] and Deep InfoMax [24] papers, finding that CMC outperforms these methods when all are tested with the same architecture and dataset (in particular, with AlexNet [32], and STL10 [10]; see Table 1). Given the new results in [22] and [3], we expect that CMC, and perhaps most prior representation learning methods, would perform much more strongly if we scale the architecture and data. A preliminary exploration of this is given in Section 4.1.2, where we achieve 60.1% top1 accuracy on ImageNet with CMC using Resnet101, compared to 61.0% with CPC++ using Resnet170 and 60.2% with Deep Infomax++ using a heavilycustomized Resnet. To our best knowledge, CMC, CPC++, and Deep Infomax++ form the first batch of unsupervised/selfsupervised learning algorithms which surpass the supervised AlexNet for ImageNet classification.
3 Contrastive Multiview Coding
Our goal is to learn representations that capture information shared between multiple sensory views without human supervision. We start by reviewing previous predictive learning (or reconstructionbased learning) methods, and then elaborate on contrastive learning within two views. We show connections to mutual information maximization and extend it to scenarios including more than two views.
We consider a collection of views of the data, denoted as . For each view , we denote as a random variable representing samples following .
3.1 Predictive Learning
Let and represent two views of a dataset. For instance, might be the luminance of a particular image and the chrominance. We define the predictive learning setup as a deep nonlinear transformation from to through latent variables , as shown in Fig. 3. Formally, and , where and represent the encoder and decoder respectively and is the prediction of given . The parameters of the encoder and decoder models are then trained using an objective function that tries to bring “close to” . Simple examples of such an objective include the or loss functions. Note that these objectives assume independence between each pixel or element of and , i.e., , thereby reducing their ability to model correlations or complex structure. The predictive approach has been extensively used in representation learning, for example, colorization [61, 62] and predicting sound from vision [47].
3.2 Contrasting Two Views
The idea behind contrastive learning is learning the parameters of a model by discriminating or comparing between samples from different distributions. Given a dataset of and that consists of a collection of samples , we consider contrasting congruent and incongruent pairs. Formally, we refer to samples from the joint distribution as positives, i.e., or , and samples from the product of marginals as negatives, i.e., or .
We learn a score function that is higher for positive samples but lower for negative samples . Similar to recent setups for contrastive learning [46, 21, 40], we train this function to correctly select a single positive sample out of a set which contains other negatives:
(1) 
To construct , we simply fix one view and enumerate the other view, leading to the objective we minimize:
(2) 
where is the number of possible negative samples for a given samples . In Sec. A.1, we show that the optimal score function for this loss is proportional to the density ratio between joint distribution and product of marginals :
(3) 
In practice, can be extremely large, and so directly minimizing Eq. 2 is infeasible. In Section 3.5, we show an approximation based on Noise Contrastive Estimation [21] that allows for tractable computation.
We implement the score function as a neural network, but other continuous and differentiable parametric functions can be used instead. To extract compact latent representations of and , we employ two encoders and with parameters and respectively. The latent representions are extracted as , . On top of these features, the score is computed as the exponential of a bivariate function of and , e.g., a bilinear function parameterized by :
(4) 
Loss in Eq. 2 treats view as anchor and enumerates over . Symmetrically, we can get by anchoring at . We add them up as our twoview loss:
(5) 
After the contrastive learning phase, we use the representations and/or , depending on our paradigm. This process is visualized in Fig. 2. To evaluate the representations, we add a linear classifier and train the classifier parameters or finetune the entire representation with labeled data for a specific task. See Section 4 for details on the experiments.
3.3 More than Two Views
We present more general formulations of Eq. 2 that can handle any number of views. We call them the “core view” and “full graph” paradigms, which offer different tradeoffs between efficiency and effectiveness. These formulations are visualized in Fig. 4.
Suppose we have a collection of views . The “core view” formulation sets apart one view that we want to optimize over, say , and builds pairwise representations between and each other view , by optimizing the sum of a set of pairwise objectives:
(6) 
A second, even more general formulation is the “full graph” where we consider all pairs , and build relationships in all. By involving all pairs, the objective function that we optimize is:
(7) 
Both these formulations have the effect that information is prioritized in proportion to the number of views that share that information. This can be seen in the information diagrams visualized in Fig. 5. The number in each partition of the diagram indicates how many of the pairwise objectives, , that partition contributes to. Under both the core view and full graph objectives, a factor, like “presence of dog”, that is common to all views will be preferred over a factor that affects fewer views, such as “depth sensor noise”.
The computational cost of the full graph formulation is combinatorial in the number of views. However, it is clear from Fig. 5 that this enables the full graph formulation to capture more information between different views, which may prove useful for downstream tasks. For example, the mutual information between and or and is completely ignored in the core view paradigm (as shown by a count in the information diagram).
3.4 Connecting to Mutual Information
The contrastive learning paradigm is connected to mutual information maximization between the variable and ; mutual information is defined as:
(8) 
Intuitively, the contrastive loss discriminates between samples from joint distribution and samples from product of marginals, thus maximizing the discrepancy between the distribution of their latent representations. The formal proof is given by [46] and it shows that:
(9) 
where, as above, is the number of negative pairs in sample set . Hence minimizing the objective maximizes the lower bound on the mutual information , which on the other side is bounded above by according to data processing inequality. The dependency on also suggests that using more negative samples can lead to an improved representation; we show that this is indeed the case in Section 4. Finally, we note that recent work [37] shows that the bound in Eq. 9 can be very weak; and finding better estimators of mutual information is an important open problem.
Given three views , the mutual information is upper bounded by the minimum of the pairwise mutual information [36]. This leads us to the general idea of extending the framework to multiple views by considering pairwise mutual information maximization to learn a representation. Given views, we consider a total of pairs; and learn for each pair of views . As above, we call this the “full graph” paradigm. For a large number of views, this paradigm can be expensive to learn; instead we can just learn as in Eq. 6, where is picked as a special view, and call this the “core view” mode. This corresponds to maximizing mutual information between and all other views, which can serve as a reasonable proxy to the “full graph” paradigm.
3.5 Approximating Full Softmax
Better representations using in Eq. 2 are learnt by using many negative samples. However, computing the full softmax loss is prohibitively expensive for large . We alleviate computational overhead in two ways: (a) resorting to NoiseContrastive Estimation [21] to approximate the full softmax, such a trick was also used in [40] for example; (b) contrasting subpatches rather than full images to increase the number of negatives inside each batch, similar to [38, 58, 24].
3.5.1 Leveraging NoiseContrastive Estimation
Given an anchor from , the probablity that an atom from is the best match of , using the score is given by:
(10) 
where the normalization factor is expensive to compute for large . Here we use , where modulates the distribution.
NoiseContrastive Estimation [21] (NCE) is an effective way to estimate unnormalized statistical models. NCE fits a density model to data distributed as (unknown) distribution , by using a binary classifier to distinguish it from noise samples distributed as . To learn , we use a binary classifier, which treats as the data sample when given . The noise distribution we choose here is a uniform distribution over all atoms from , i.e., . If we sample noise samples to pair with each data sample, the posterior probability that a given atom comes from the data distribution is:
(11) 
and we estimate this probability by replacing with our model distribution . Minimizing the negative logposterior probability of correct labels over data and noise samples yields our objective, which is the NCEbased approximation of Eq. 2:
(12)  
Memory bank. Following [58], we maintain a memory bank to store latent features for each training sample. Therefore, we can efficiently retrieve noise samples from the memory bank to pair with each positive sample without recomputing their features. The memory bank is dynamically updated with features computed on the fly.
An alternative to the NCE based approximation above, is to simply do way softmax classification with noise samples retrieved from the memory bank. We note that CPC [46] and Deep InfoMax [24] use this way softmax classification as their ultimate contrastive loss rather than the NCEbased contrastive loss in Eq. 12 (but note that CPC refers to the approximation as also “based on NCE”). Empirically we have found that the way softmax classification approach performed worse than our NCEbased approximation, given the same number of noise samples.
3.5.2 Contrasting Subpatches
Instead of contrasting features from the last layer, patchbased method [24] contrasts feature from the last layer with features from previous layers, hence increasing the number of negative pairs. For instance, we use features from the last layer of to contrast with feature points from feature maps produced by the first several conv layers of . This is equivalent to contrast between global patch from one view with local patches from the other view. In this fashion, we directly perform way loss, the same as [46, 24] for a fair comparison in Sec. 4.1.1.
Such patchbased contrastive loss is computed within each minibatch and does not require a memory bank. Therefore, deploying it in parallel training schemes is easy and flexible. However, patchbased contrastive loss usually yields suboptimal results compared to NCEbased contrastive loss, according to our experiments.
4 Experiments
We extensively evaluate our Contrastive Multiview Coding (CMC) framework on a number of datasets and tasks. We start by evaluating on two established image representation learning benchmarks: STL10 and Imagenet. We further validate our framework on video representation learning tasks, where we use image and optical flow modalities, as the two views that are jointly learned. The last set of experiments extends our CMC framework to more than two views and provides empirical evidence of it’s effectiveness.
4.1 CMC on Images
Given a dataset of RGB images, we convert them to the Lab image color space, and split each image into L and ab channels, as originally proposed in SplitBrain autoencoders [62]. During contrastive learning, L and ab from the same image are treated as the positive pair, and ab channels from other randomly selected images are treated as a negative pair (for a given L). Each split represents a view of the orginal image and is passed through a seprate encoder. This corresponds to the “full graph” model of Eq. 2 with L and ab channels as the two views. As in SplitBrain, we design these two encoders by evenly splitting a given deep network, such as AlexNet [32], into subnetworks across the channel dimension. By concatenating representations layerwise from these two encoders, we achieve the final representation of an input image. As proposed by previous literature [46, 24, 2], the quality of such a representation is evaluated by freezing the weights of encoder and training linear or nonlinear classifiers on top of each layer.
4.1.1 Stl10
STL10 [10] is an image recognition dataset designed for developing unsupervised or selfsupervised learning algorithms. It consists of unlabeled training RGB image samples and labeled samples for each of the classes.
Setup. We adopt the same data augmentation strategy and network architecture as those in DIM [24]. A variant of AlexNet takes as input images, which are randomly cropped and horizontally flipped from the original size images. For a fair comparison with DIM, we also train our model in a patchbased contrastive fashion during unsupervised pretraining. With the weights of the pretrained encoder frozen, a twolayer fully connected network with 200 hidden units is trained on top of different layers for 100 epochs to perform 10way classification. We also investigated the strided crop strategy of CPC [46]. Fixed sized overlapping patches of size with an overlap of pixels are cropped and fed into the network separately. This ensures that features of one patch contain minimal information from neighbouring patches; and increases the available number of negative pairs for the contrastive loss. Additionally, we include NCEbased contrastive training and linear classifier evaluation.
Comparison. We compare CMC with the state of the art unsupervised methods in Table 1. Three columns are shown: the conv5 and fc7 columns use respectively these layers of AlexNet as the encoder (again remembering that we split across channels for L and ab views). For these two columns we can compare against the all methods except CPC, since CPC does not report these numbers in their paper [24]. In the Strided Crop setup, we only compare against the approaches that use contrastive learning, DIM and CPC, since this method was only used by those works. We note that in Table 1 for all the methods except SplitBrain, we report numbers are shown in the original paper. For SplitBrain, we reimplemented their model faithfully and report numbers based on our reimplementation (we verified the accuracy of our SplitBrain code by the fact that we get very similar results with our reimpementation as in the original paper [62] for ImageNet experiments, see below).
The family of contrastive learning methods, such as DIM, CPC, and CMC, achieve higher classification accuracy than other methods such as SplitBrain that use predictive learning; or BiGAN that use adversarial learning. CMC significantly outperforms DIM and CPC in all cases. We hypothesize that this outperformance results from the modeling of crossview mutual information, where viewspecific noisy details are discarded. Another headtohead comparison happens between CMC and SplitBrain, both of which modeling images as seprated L and ab streams; we achieve a nearly absolute improvement for conv5 and improvement for fc5. Finally, we notice that the predictive learning methods suffer from a big drop in performance when the encoding layer is switched from conv5 to fc7. On the other hand, the contrastive learning approaches are much more stable across layers, suggesting that the mutual information maximization paradigm learns more semantically meaningful representations shared by the different views. From a practical perspective, this is a significant advantage as the selection of specific layers should ideally not change downstream performance by too much.
Method  classifier  conv5  fc7  Strided Crop 

AE  MLP  62.19  55.78   
NAT [7]  64.32  61.43    
BiGAN [15]  71.53  67.18    
SplitBrain [62]  72.35  63.15    
DIM [24]  MLP  72.57  70.00  76.97 
CPC [46]      77.81  
CMC(Patch)  Linear  76.65  79.25  82.58 
CMC(Patch)  MLP  80.14  80.11  83.43 
CMC(NCE)  Linear  80.69  84.73   
CMC(NCE)  MLP  83.03  85.06   
Supervised  68.70 
4.1.2 ImageNet
ImageNet [13] consists of 1000 image classes and is frequently considered as a testbed for unsupervised representation learning algorithms.
Setup. To compare with other methods, we adopt standard AlexNet and split it into two encoders. Because of splitting, each layer only connects to half of the neurons in the previous layer, and therefore the number of parameters in our model halves. We remove local response layer and add batch normalization to each layer. Two variants of CMC are considered: patchbased and memorybased. For the patchbased CMC model, we set the batch size as 100 and adopt patch features from pool2 layer. For the memorybased CMC model, we adopt ideas from [58] for computing and storing a memory. We retrieve negative pairs from the memory bank to contrast each positive pair. The training details are present in Sec. B.2.
ImageNet classification task. Following [61], we evaluate task generalization of the learned representation by training 1000way linear classifiers on top of different layers. Table 2 shows the results of comparing the two variants of CMC against other models, both predictive and contrastive. The NCEbased CMC variant is the best among all these methods; futhermore the CMC methods tend to perform better at higher convolutional layers, similar to the other contrastive model InstDis [58]. The NCEbased CMC model consistently performs better than the patchbased model due to the use of NCE as well as many more contrasting images.
ImageNet Classification Accuracy  
Method  conv1  conv2  conv3  conv4  conv5 
ImageNetLabels  19.3  36.3  44.2  48.3  50.5 
Random  11.6  17.1  16.9  16.3  14.1 
DataInit [31]  17.5  23.0  24.5  23.2  20.6 
Context [14]  16.2  23.3  30.2  31.7  29.6 
Colorization [61]  13.1  24.8  31.0  32.6  31.8 
Jigsaw [43]  19.2  30.1  34.7  33.9  28.3 
BiGAN [15]  17.7  24.5  31.0  29.9  28.0 
SplitBrain [62]  17.7  29.3  35.4  35.2  32.8 
Counting [44]  18.0  30.6  34.3  32.5  25.7 
InstDis [58]  16.8  26.5  31.8  34.1  35.6 
RotNet [18]  18.8  31.7  38.7  38.2  36.5 
DeepCluster [9]  12.9  29.2  38.2  39.8  36.1 
CMC(Patch)  17.8  30.8  34.2  37.5  38.1 
CMC(NCE)  18.4  33.5  38.1  40.4  42.6 
Effect of the number of negative samples. We investigate the relationship between the number of negative pairs in NCEbased loss and the downstream classification accuracy on a randomly chosen subset of classes of Imagenet (the same set of classes is used for any number of negative pairs). We train a 100way linear classifier using CMC pretrained features with varying number of negative pairs, starting from pairs upto (in multiples of ). Fig. 6 shows that the accuracy of the resulting classifier steadily increases but saturates at around with samples.
Accuracy (%)  ResNet50  ResNet101 

Top1  58.1  60.1 
Top5  81.4  82.8 
CMC with ResNets. We verify the scalability of CMC with larger networks such as ResNets. The results are shown in Table 3, where ResNet50 and ResNet101 achieve and top1 accuracies, respectively. To our best knowledge, our CMC, together with concurrent CPC++ [22] and Deep InfoMax++ [3], are the first batch of unsupervised/selfsupervised learning methods that surpass supervised AlexNet on ImageNet classification task. Note that CPC on ResNet101 achieved , therefore for the same architecture, CMC results in a absolute improvment over CPC, and nearly a relative improvement.
4.2 CMC on Videos
Method  # of Views  UCF101  HMDB51 

Random    48.2  19.5 
ImageNet    67.7  28.0 
VGAN* [56]  2  52.1   
LTMotion* [35]  2  53.0   
TempCoh [41]  1  45.4  15.9 
Shuffle and Learn [39]  1  50.2  18.1 
Geometry [17]  2  55.1  23.3 
OPN [33]  1  56.3  22.1 
ST Order [8]  1  58.6  25.0 
Cross and Learn [51]  2  58.7  27.2 
CMC (V)  2  55.3   
CMC (D)  2  57.1   
CMC (V+D)  3  59.1  26.7 
We apply CMC on videos by drawing insight from the twostreams hypothesis [52, 19], which posits that human visual cortex consists of two distinct processing streams: the ventral stream, which performs object recognition, and the dorsal stream, which processes motion. In our formulation, given an image that is a frame centered at time , the ventral stream associates it with a neighbouring frame , while the dorsal stream connects it to optical flow centered at . Therefore, we extract , and from two modalities as three views of a video; for optical flow we use the TVL1 algorithm [60]. Two separate contrastive learning objectives are built within the ventral stream and within the dorsal stream . For the ventral stream, the negative sample for is chosen as a random frame from another randomly chosen video; for the dorsal stream, the negative sample for is chosen as the flow corresponding to a random frame in another randomly chosen video.
Pretraining. We train CMC on UCF101 [55] and use two CaffeNets [32] for extracting features from images and optical flows, respectively. In our implementation, represents 10 continuous flow frames centered at . We use batch size of 128 and contrast each positive pair with 127 negative pairs. CMC is trained with Adam for 300 epochs, with an initial learning rate of 0.001 which is decayed by a factor of 5 after 200 and 250 epochs.
Action recognition. We apply the learn representation to the task of action recognition. The spatial network from [53] is a wellestablished paradigm for evaluating pretrained RGB network on action recognition task. We follow the same spirit and evaluate the transferability of our RGB CaffeNet on UCF101 and HMDB51 datasets. We initialize the action recognition CaffeNet up to conv5 using the weights from the pretrained RGB CaffeNet. The averaged accuracy over three splits is present in Table 4. Unifying both ventral and dorsal streams during pretraining produces higher accuracy for downstream recognition than using only single stream. Increasing the number of views of the data from to (using both streams instead of one) provides a boost for UCF101. Furthermore, on UCF101, we outperform all other methods; and on HMDB51, CMC is secondbest in performance.
4.3 Extending CMC to More Views
We further extend our CMC learning framework to multiview scenarios. We experiment on the NYUDepthV2 [42] dataset which consists of 1449 labeled images. We focus more on understanding the behavior and effectiveness of CMC rather than competing with the current stateoftheart. The views we consider are: luminance (L channel), chrominance (ab channel), depth, surface normal [16], and semantic labels.
Setup. To extract features from each view, we use a neural network with 5 convolutional layers, and 1 fully connected layer. As the size of the dataset is relatively small, we adopt the patchbased contrastive objective to increase the number of negative pairs. Patches with a size of are randomly cropped from the original images for contrastive learning (from images of size ). For downstream tasks, we discard the fully connected layers and evaluate using the convolutional layers as a representation. This is because the fully convolutional network can adapt to tasks such as semantic segmentation where input size changes.
4.3.1 Measuring representation quality as the number of views increases
To measure the quality of the learned representation, we consider the task of predicting semantic labels from the representation of . We follow the core view paradigm and use are the core view, thus learning a set of representations on by contrasting different views with . A UNet style architecture [49] is utilized to perform the segmentation task. Contrastive training is performed on the above architecture that is equivalent of the UNet’s encoder. After contrastive training is completed, we initialize the encoder weights of the UNet from the encoder (which are equivalent architectures) and keep them frozen. Only the decoder is trained during this finetuning stage.
Since we use the patchbased contrastive loss, in the 1 view setting case, CMC coincides with DIM [24]. The 24 view cases contrast L with ab, and then sequentially add depth and surface normals, but in all cases, the patch based loss is used because the amount of data is small. The semantic labeling results are measured by mean IoU over all classes and pixel accuracy are shown in Fig. 7. We see that the performance steadily improves as new views are added. We have tested different orders of adding the views, and they all follow a similar pattern.
Pixel Accuracy (%)  mIoU (%)  

Random  45.5  21.4 
CMC (coreview)  57.1  34.1 
CMC (fullgraph)  57.0  34.4 
Supervised  57.8  35.9 
We also compare CMC with two baselines. First, we randomly initialize and freeze the encoder, and we call this the Random baseline; it serves as a lower bound on the quality since the representation is just a random projection. Rather than freezing the randomly initialized encoder, we could train it jointly with the decoder. This endtoend Supervised baseline serves as an upper bound. The results are presented in Table 5, which shows our CMC produces high quality feature maps even though it’s unaware of the downstream task.
4.3.2 Is CMC improving all views?
A desirable unsupervised representation learning algorithm operating on multiple views or modalities should improve the quality of representations for all views. We therefore investigate our CMC framwork beyond L channel. To treat all views fairly, we train these encoders following the full graph paradigm, where each view is contrasted with all other views.
Metric (%)  L  ab  Depth  Normal  

Random  mIoU  21.4  15.6  30.1  29.5 
pix. acc.  45.5  37.7  51.1  50.5  
CMC  mIoU  34.4  26.1  39.2  37.8 
pix. acc.  57.0  49.6  59.4  57.8  
Supervised  mIoU  35.9  29.6  41.0  41.5 
pix. acc.  57.8  52.6  59.1  59.6 
We evaluate the representation of each view by predicting the semantic labels from only the representation of , where is L, ab, depth or surface normals. This uses the fullgraph paradigm. As in the previous section, we compare CMC with Random and Supervised baselines. As shown in Table 6, the performance of the representations learned by CMC using fullgraph significantly outperforms that of randomly projected representations, and approaches the performance of the fully supervised representations. Furthermore, the fullgraph representation provides a good representation learnt for all views, showing the importance of capturing different types of mutual information across views.
4.4 Predictive Learning vs. Contrastive Learning?
While experiments in section 4.1 show that contrastive learning outperforms predictive learning [62] in the context of Lab color space, it’s unclear whether such an advantage is due to the natural inductive bias of the task itself. To further understand this, we go beyond chrominance (ab), and try to answer this question when geometry or semantic labels are present.
We consider three view pairs on the NYUDepth dataset: (1) L and depth, (2) L and surface normals, and (3) L and segmentation map. For each of them, we train two identical encoders for L, one using contrastive learning and the other with predictive learning. We then evaluate the representation quality by training a linear classifier on top of these encoders on the STL10 dataset.
Accuracy on STL10 (%)  
Views  Predictive  Contrastive 
L, Depth  55.5  58.3 
L, Normal  58.4  60.1 
L, Seg. Map  57.7  59.2 
Random  25.2  
Supervised  65.1 
The comparison results are shown in Table 7, which shows that contrastive learning consistently outperforms predictive learning in this scenario where both the task and the dataset are unknown. We also include “random” and “supervised” baselines similar to that in previous sections. Though in the unsupervised stage we only use 1.3K images from a dataset much different from the target dataset STL10, the object recognition accuracy is close to the supervised method, which uses an endtoend deep network directly trained on STL10.
Given two views and of the data, the predictive learning approach approximately models . Furthermore, losses used typically for predictive learning, such as pixelwise reconstruction losses usually impose an independence assumption on the modeling: . On the other hand, the contrastive learning approach by construction does not assume conditional independence across dimensions of . We conjecture that this is one reason for the superior performance of constrastive learning approaches over predictive learning.
5 Conclusion
We have presented a contrastive learning framework which enables the learning of unsupervised representations from multiple views of a dataset. The principle of maximization of mutual information enables the learning of powerful representations. A number of empirical results show that our framework performs well compared to predictive learning and scales with the number of views.
Acknowledgements
Thanks to Devon Hjelm for providing implementation details of Deep InfoMax, Zhirong Wu and Richard Zhang for helpful discussion and comments. This material is based upon work supported by Google Cloud.
References
 [1] Information Diagram  Wikipedia. https://en.wikipedia.org/wiki/Information_diagram.
 [2] S. Arora, H. Khandeparkar, M. Khodak, O. Plevrakis, and N. Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
 [3] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910, 2019.
 [4] A. Bellet, A. Habrard, and M. Sebban. Similarity learning for provably accurate sparse linear classification. arXiv preprint arXiv:1206.6476, 2012.
 [5] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
 [6] A. Blum and T. Mitchell. Combining labeled and unlabeled data with cotraining. In Proceedings of the eleventh annual conference on Computational learning theory, pages 92–100. ACM, 1998.
 [7] P. Bojanowski and A. Joulin. Unsupervised learning by predicting noise. In Proceedings of the 34th International Conference on Machine LearningVolume 70, pages 517–526. JMLR. org, 2017.
 [8] U. Buchler, B. Brattoli, and B. Ommer. Improving spatiotemporal selfsupervision by deep reinforcement learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 770–786, 2018.
 [9] M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149, 2018.
 [10] A. Coates, A. Ng, and H. Lee. An analysis of singlelayer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215–223, 2011.
 [11] C. Cortes, M. Mohri, and A. Rostamizadeh. Learning nonlinear combinations of kernels. In Advances in neural information processing systems, pages 396–404, 2009.
 [12] H. E. Den Ouden, P. Kok, and F. P. De Lange. How prediction errors shape perception, attention, and motivation. Frontiers in psychology, 3:548, 2012.
 [13] J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. FeiFei. Imagenet: A largescale hierarchical image database. 2009.
 [14] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 1422–1430, 2015.
 [15] J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial feature learning. In International Conference on Learning Representations, 2017.
 [16] D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with a common multiscale convolutional architecture. In Proceedings of the IEEE international conference on computer vision, 2015.
 [17] C. Gan, B. Gong, K. Liu, H. Su, and L. J. Guibas. Geometry guided convolutional neural networks for selfsupervised video representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5589–5597, 2018.
 [18] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728, 2018.
 [19] M. A. Goodale and A. D. Milner. Separate visual pathways for perception and action. Trends in neurosciences, 1992.
 [20] I. Goodfellow, J. PougetAbadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014.
 [21] M. Gutmann and A. Hyvärinen. Noisecontrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297–304, 2010.
 [22] O. J. Hénaff, A. Razavi, C. Doersch, S. Eslami, and A. v. d. Oord. Dataefficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.09272, 2019.
 [23] G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504–507, 2006.
 [24] R. D. Hjelm, A. Fedorov, S. LavoieMarchildon, K. Grewal, A. Trischler, and Y. Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
 [25] J. Hohwy. The predictive mind. Oxford University Press, 2013.
 [26] A. Hyvärinen, J. Karhunen, and E. Oja. Independent component analysis, volume 46. John Wiley & Sons, 2004.
 [27] P. Isola, J.Y. Zhu, T. Zhou, and A. A. Efros. Imagetoimage translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
 [28] P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Learning visual groups from cooccurrences in space and time. arXiv preprint arXiv:1511.06811, 2015.
 [29] I. Jolliffe. Principal component analysis. Springer, 2011.
 [30] D. P. Kingma and M. Welling. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
 [31] P. Krähenbühl, C. Doersch, J. Donahue, and T. Darrell. Datadependent initializations of convolutional neural networks. arXiv preprint arXiv:1511.06856, 2015.
 [32] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
 [33] H.Y. Lee, J.B. Huang, M. Singh, and M.H. Yang. Unsupervised representation learning by sorting sequences. In Proceedings of the IEEE International Conference on Computer Vision, pages 667–676, 2017.
 [34] Y. Li, M. Yang, and Z. M. Zhang. A survey of multiview representation learning. IEEE Transactions on Knowledge and Data Engineering, 2018.
 [35] Z. Luo, B. Peng, D.A. Huang, A. Alahi, and L. FeiFei. Unsupervised learning of longterm motion dynamics for videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2203–2212, 2017.
 [36] I. Lütkebohle. BWorld Robot Control Software. https://en.wikipedia.org/wiki/Multivariate_mutual_information, 2008. [Online; accessed 19July2008].
 [37] D. McAllester and K. Statos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv:1811.04251, 2018.
 [38] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013.
 [39] I. Misra, C. L. Zitnick, and M. Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision, pages 527–544. Springer, 2016.
 [40] A. Mnih and K. Kavukcuoglu. Learning word embeddings efficiently with noisecontrastive estimation. In Advances in neural information processing systems, pages 2265–2273, 2013.
 [41] H. Mobahi, R. Collobert, and J. Weston. Deep learning from temporal coherence in video. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 737–744. ACM, 2009.
 [42] P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
 [43] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69–84. Springer, 2016.
 [44] M. Noroozi, H. Pirsiavash, and P. Favaro. Representation learning by learning to count. In Proceedings of the IEEE International Conference on Computer Vision, pages 5898–5906, 2017.
 [45] A. v. d. Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016.
 [46] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
 [47] A. Owens, P. Isola, J. McDermott, A. Torralba, E. H. Adelson, and W. T. Freeman. Visually indicated sounds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2405–2413, 2016.
 [48] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
 [49] O. Ronneberger, P. Fischer, and T. Brox. Unet: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computerassisted intervention, pages 234–241. Springer, 2015.
 [50] R. Salakhutdinov and G. Hinton. Deep boltzmann machines. In Artificial intelligence and statistics, pages 448–455, 2009.
 [51] N. Sayed, B. Brattoli, and B. Ommer. Cross and learn: Crossmodal selfsupervision. arXiv preprint arXiv:1811.03879, 2018.
 [52] G. E. Schneider. Two visual systems. Science, 1969.
 [53] K. Simonyan and A. Zisserman. Twostream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pages 568–576, 2014.
 [54] P. Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986.
 [55] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
 [56] C. Vondrick, H. Pirsiavash, and A. Torralba. Generating videos with scene dynamics. In Advances In Neural Information Processing Systems, pages 613–621, 2016.
 [57] X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794–2802, 2015.
 [58] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via nonparametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
 [59] C. Xu, D. Tao, and C. Xu. A survey on multiview learning. arXiv preprint arXiv:1304.5634, 2013.
 [60] C. Zach, T. Pock, and H. Bischof. A duality based approach for realtime tvl 1 optical flow. In Joint pattern recognition symposium, pages 214–223. Springer, 2007.
 [61] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In European conference on computer vision, pages 649–666. Springer, 2016.
 [62] R. Zhang, P. Isola, and A. A. Efros. Splitbrain autoencoders: Unsupervised learning by crosschannel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1058–1067, 2017.
Appendix
Appendix A Proofs
We prove that: (a) the optimal score function is proportional to density ratio between the joint distribution and product of marginals , as shown in Eq. 3; (b) Minimizing the contrastive loss maxmizes a lower bound on the mutual information between two views, as shown in Eq. 9
We will use the most general formula of contrastive loss shown in Eq. 1 for our derivation. But we note that replacing with is straightforward. The overall proof follows a similar derivation introduced in [46].
a.1 Score function as density ratio estimator
We first show that the optimal score function that minimizes Eq. 1 is proportional to the density ratio between joint distribution and product of marginals, shown as Eq. 3. For notation convenience, we denote as data distribution and as noise distribution . The loss in Eq. 1 is indeed a crossentropy loss of classifying the correct positive pair out from the given set . Without loss of generality, we assume the first pair in is positive or congruent and all others are negative or incongruent. The optimal probability for the loss, , should depict the fact that comes from the data distribution while all other pairs come from the noise distribution . Therefore,
where we plug in the definition of and , and divide for both the numerator and denominator. By comparing above equation with the loss function in Eq. 1, we can see that the optimal score function is proportional to the density ratio . The above derivation is agnostic to which layer the score function starts from, e.g., can be defined on either the raw input or the latent representation . As we care more about the property of the latent representation, for the following derivation we will use , which is proportional to .
a.2 Maximizing lower bound on MI
Now we substitute the score function in Eq. 1 with the above density ratio, and the optimal loss objective becomes:
Therefore, for any two views and , we have . As the increases, the approximation step becomes more accurate. Given any , minimizing maximizes the lower bound on the mutual information . We should note that increasing to infinity does not always lead to a higher lower bound. While increases with a larger , the optimization problem becomes harder and also increases.
Appendix B Implementation Details
b.1 Stl10
For a fair comparison with DIM [24] and CPC [46], we adopt the same architecture as that used in DIM and split it into two encoders, each shown as in Table 8. For the implementation of the score function, we adopt similar “encoderanddotproduct” strategy, which is tantamount to a bilinear model.
Half of AlexNet[32] for STL10  

Layer  X  C  K  S  P 
data  64  *  –  –  – 
conv1  64  48  3  1  1 
pool1  31  48  3  2  0 
conv2  31  96  3  1  1 
pool2  15  96  3  2  0 
conv3  15  192  3  1  1 
conv4  15  192  3  1  1 
conv5  15  96  3  1  1 
pool5  7  96  3  2  0 
fc6  1  2048  7  1  0 
fc7  1  2048  1  1  0 
fc8  1  64  1  1  0 
In the patchbased contrastive learning stage, we use Adam optimizer with an initial learning rate of , , . We train for a total of epochs with learning rate decayed by after and epochs. In the nonlinear classifier evaluation stage, we use the same optimizer setting. For the NCEbased contrastive learning stage, we train for 200 epochs with the learning rate initialized as and further decayed by 10 for every epochs after the first epochs. The temperature is set as . In general, works reasonably well.
b.2 ImageNet
For patchbased contrastive loss, we use the same optimizer setting as in Sec. B.1 except that the learning rate is initialized as 0.01.
For NCEbasd contrastive loss in both full ImageNet and ImageNet100 experiments present in Sec. 4.1.2, the encoder architecture used for either L or ab channels is shown in Table 9. In the unsupervised learning stage, we use SGD to train the network for a total of epochs. The temperature is set as by following previous work [58]. The learning rate is initialized as with a decay of 10 for every epochs after the first epochs. Weight decay is set as and momentum is kept as . For the linear classification stage, we train for epochs. The learning rate is initialized as and decayed by every epochs after the first epochs. We set weight decay as and momentum as .
Half of AlexNet[32] for ImageNet  

Layer  X  C  K  S  P 
data  224  *  –  –  – 
conv1  55  48  11  4  2 
pool1  27  48  3  2  0 
conv2  27  128  5  1  2 
pool2  13  128  3  2  0 
conv3  13  192  3  1  1 
conv4  13  192  3  1  1 
conv5  13  128  3  1  1 
pool5  6  128  3  2  0 
fc6  1  2048  6  1  0 
fc7  1  2048  1  1  0 
fc8  1  128  1  1  0 
b.3 UCF101 and HMDB51
b.4 NYU DepthV2
While experimenting with different views on NYU DepthV2 dataset, we encode the features from patches with a size of . The detailed architecture is shown in Table 10. In the unsupervised training stage, we use Adam optimizer with an initial learning rate of , , . We train for a total of epochs with learning rate decayed by after , , and epochs. For the downstream semantic segmentation task, we use the same optimizer setting but train for fewer epochs. We only train epochs for CMC pretrained models, and train epochs for the Random and Supervised baselines until convergence. For the classification task evaluated on STL10, we use the same optimizer setting as in Sec. B.1.
Encoder Architecture on NYU  

Layer  X  C  K  S  P 
data  128  *  –  –  – 
conv1  64  64  8  2  3 
pool1  32  64  2  2  0 
conv2  16  128  4  2  1 
conv3  8  256  4  2  1 
conv4  8  256  3  1  1 
conv5  4  512  4  2  1 
fc6  1  512  4  1  0 
fc7  1  256  1  1  0 