IFT6266 Final project: conditional image generation

In the first proposed model familly, the formal problem we have to solve is to find a fixed mapping \(f: x^{(b)} \to x^{(c)}\). This task can be tackled with a parametric approach, namely, finding the best parameter set \(\theta\) in \(f(x^{(b)};\theta) = \hat{x}^{(c)}\).

We will define \(f(x^{(b);\theta})\) to be a densely connected, multi-layer neural network (note that this means that the input image \(x\) has to be flattened into a vector). The basic building block of this model is an affine transformation which is then non-linearly transformed through a rectified linear unit:

\begin{equation} Y = max(0, XW + b) \end{equation}

where \(X\) is the design matrix containing one example per row and \(W\) and \(b\) are the parameters of the transformation. These layers can be chained sequentially:

\begin{equation} Y^{(i+1)} = max(0, Y^{(i)}W^{(i)} + b^{(i)}) \end{equation}

Where \(Y^{0} \equiv X\). At the output layer, a element-wise sigmoid activation will be used:

\begin{equation} \sigma(x) = \frac{1}{1 + e^{-x}} \end{equation}

which constrain the generated pixel values to be between 0 and 1. They can then be scaled by 255 to yield a valid image.

The weights can then be iteratively updated by making small steps in the direction opposite to the gradient of a loss function. This loss function will either be the squared error:

\begin{equation} SE = (x^{(c)} - \hat{x}^{(c)})^{2} \end{equation}

If we view each pixel's output as a probability of being activated in a particular channel, then a natural loss function is the sum of the pixel-wise binary crossentropy:

\begin{equation} XE = - \sum_{i,j,k} (x_{i,j,k}^{(c)}log(\hat{x}^{(c)}_{i,j,k}) + (1 - x_{i,j,k}^{(c)})log(1 - \hat{x}^{(c)}_{i,j,k}) ) \end{equation}

To prevent overfitting, a term penalizing large weights is added to the loss

\begin{equation} L_{W} = \alpha W^{T}W \end{equation}

During training, the distribution from a layer's output change as the weights of this layer are updated. This problem, refered to as the internal covariate shift , complicates the learning task of the downstream layers. This can be mitigated by using batch normalization ², a technique in which the input of a layer is normalized as a function of the mean and variance of the inputs in a mini-batch:

\begin{equation} y = \gamma \frac{x - \mu}{\sqrt{\sigma^{2} + \delta}} + \beta \end{equation}

Where γ and β are learned parameters and δ is a small constant added for numerical stability.

For the gradient descent, two different momentum schemes are considered. First, the "vanilla" momentum:

\begin{eqnarray} g = \nabla_{\theta} L(x^{(c)}, \hat{x}^{(c)}, \theta) \\ v = \alpha v - \epsilon g \\ \theta = \theta + v \end{eqnarray}

Where α and ε are fixed constants. The ADAM algorithm is also considered:

\begin{eqnarray} g = \nabla_{\theta} L(x^{(c)}, \hat{x}^{(c)}, \theta) \\ r = \rho_{1}r + (1 - \rho_{1})g \\ s = \rho_{2}s + (1 - \rho_{2})g\odot g \\ \tilde{r} = \frac{r}{1 - \rho_{1}^{t}} \\ \tilde{s} = \frac{s}{1 - \rho_{2}^{t}} \\ \theta = \theta - \epsilon \frac{\tilde{r}}{\sqrt{\tilde{s} + \delta}} \end{eqnarray}

In the second familly of models considered, the goal is to model the probability distribution \(P(x^{(c)}|x^{(b)})\). This is achieved via a generator network, \(G(x^{(b)}| z)\) where the vector \(z \sim U(0, 1)\).

One effective way of training the generator is by using an auxiliary discriminator network ³, which learns a mapping \(D(x^{(c)}|x^{(b)}) = p\), where \(p\) is the probability that \(x^{(c)}\) comes from the data distribution. The two networks are then trained in tandem using the same techniques described above. The discriminator loss is the binary cross-entropy where the positive examples are drawn from the data distribution and the negative from the generator:

\begin{equation} LD = - log(D(x^{(c)}|x^{(b)}) - log(1 - D(G(x^{(b)}, z)|x^{(b)})) \end{equation}

The generator is trained to fool the discriminator:

\begin{equation} LG = - D(G(x^{(b)}, z)|x^{(b)}) \end{equation}

The actual architecture of \(G\) and \(D\) follow the outline of the DCGAN paper ⁴, which make the following recommendation:

• Use convolutional instead of affine layers in G and D
• Use strided convolutions to upsample and downsample
• Use batch normalization in G and D
• Use ReLU hidden activations and a tanh output activation in G
• Use a leaky ReLU (a ReLU with a non-zero slope in the negative part) in D

In convolutional layers, the affine mapping is replaced by convolutions with kernels which are slided across the image to produce feature maps. By increasing the stride (the "step size" of the sliding of the kernels), the dimensionality of the feature maps can be reduced.

To incorporate the latent code \(z\) in the input to the generator, the length vector is projected by an affine map and reshaped to a 64x64 matrix, allowing it to be treated as a regular channel.

Also, instead of feeding only the border to the generator, a complete image is used where the center is the output of a model from the first familly described above:

\begin{eqnarray} \tilde{G} \equiv \tilde{G}(x^{(b)}, f(x^{(b)}) = \hat{x}^{(c)}_{(0)}, z) \end{eqnarray}

The network class is a sequential collection of layers defining a model. The most interesting bits are probably the __cache_generator function, which allows a number of mini-batches of data to be cached in shared variables to eliminate the memory bottleneck, especially when running on the GPU, and (especially) the __make_training_function method, which actually implements SGD with momentum and the ADAM algorithm.

There is a function, train_GAN, which take as input two compiled Network objects and train them with the GAN framework. The data and the latent code is passed by defining python generators from which the function sample batches.

Since I was starting from scratch, I needed a couple of simple toy problems to validate my layers.

All of the dense layers were tested on the simple problem of fitting a noisy sine function, with satisfactory results:

The convolutional layers were tested with an implementation of LeNet5 on the MNIST dataset:

The recurrent layers were tested on a classification task between two 2D gaussian clusters, where the inputs are a variable number of sampling from a given cluster.

The GAN training function was tested on a generation task where the targeted distribution is a simple 2D normal distribution.

The gold standard is manual inspection of the generated images, however I also looked into alternative metrics to quantify the quality of the generated image, which as we all know is an interesting problem in itself. We can see an image as a collection of pixels, which themselves can be viewed as instances of 256 different classes. In this view, it is possible to measure a probability distribution over pixel values for a given image. It is then possible to measure the Kullback-Liebler divergence (KL) between this distribution and another one. KL measures (in a sense) how much 2 probability distribution differs. We can choose to compare the distribution from the generated image with the true image, or the distribution from the border. As training progresses, this divergence goes down:

This measure jumps around also, which suggest that smoothing might be necessary, but it actually seems to be a good indication of the fit quality. Surprisingly this works as well when comparing with the true center patch or with the border! Of course, minimizing this divergence only ensures that the pixels are sampled from the right distribution but it cannot quantify the structure in the images.

I have three similar models which have what I deem to be acceptable results. The patches are quite blurry, but they contain the right structure (at least partially). The models are different configuration of a 3 hidden layer MLP with 1k hidden units per layer, with batchnorm at each layer and a sigmoid activation at the last layer. For the code, see good_models/model₀{1,3,4}.py.

The first network is trained using SGD with momentum on a mean square error loss, using a very small learning rate (1e-7).

The second network is like the first one but using the ADAM algorithm with the default parameters.

The third one is like the second one, but the loss is a sum of the pixel-wise binary crossentropy. This effectively views each output as the probability that the given pixel in the given channel is fully activated.

These are relatively simple models, and so I'm pleasantly surprised that they work so well :)