Generative Adversarial Networks (GANs) for Style Transfer: A Complete DCGAN Implementation Guide

This tutorial explains how to implement artistic style transfer using Deep Convolutional GANs (DCGANs) with complete Keras and PyTorch code from Microsoft's AI-For-Beginners repository.

The AI-For-Beginners curriculum provides a hands-on lesson demonstrating Generative Adversarial Networks (GANs) for style transfer in lessons/4-ComputerVision/10-GANs/. This implementation trains a generator network to transform content images—such as MNIST digits—into artistic styles sampled from the WikiArt dataset, using the adversarial training paradigm defined in StyleTransfer_Keras.ipynb and GANPyTorch.ipynb.

DCGAN Architecture for Style Transfer

A style-transfer GAN follows the classic generator-discriminator adversarial loop. The generator learns to map content images to stylized outputs that mimic a target artistic domain, while the discriminator learns to distinguish between real artwork (from WikiArt) and generated images. The pipeline flows as: content image → generator → stylized output, with the discriminator providing adversarial feedback to improve generation quality.

Generator Network Design

In lessons/4-ComputerVision/10-GANs/StyleTransfer_Keras.ipynb, the generator uses transposed convolutions to upsample a latent vector into a full-resolution image. The architecture begins with a dense layer projecting a 100-dimensional noise vector into a 7×7×256 tensor, followed by Conv2DTranspose layers with batch normalization and ReLU activations.

def make_generator_model():
    model = tf.keras.Sequential(name="generator")
    model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((7, 7, 256)))          # 7×7×256

    model.add(layers.Conv2DTranspose(128, (5,5), strides=(1,1), padding='same',
                                     use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(64, (5,5), strides=(2,2), padding='same',
                                     use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(3, (5,5), strides=(2,2), padding='same',
                                     use_bias=False, activation='tanh'))
    return model

The final tanh activation constrains pixel values to the range [-1, 1], matching the normalized input distribution of the training images.

Discriminator Network Design

The discriminator in make_discriminator_model() functions as a binary classifier using Conv2D layers to detect whether an input image is authentic artwork or generator output. It employs LeakyReLU activations and dropout layers for regularization.

def make_discriminator_model():
    model = tf.keras.Sequential(name="discriminator")
    model.add(layers.Conv2D(64, (5,5), strides=(2,2), padding='same',
                            input_shape=[28,28,3]))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(128, (5,5), strides=(2,2), padding='same'))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1, activation='sigmoid'))
    return model

Training Implementation and Loss Functions

The training loop implements the minimax game between networks using binary cross-entropy loss. According to the repository source code, the generator receives an adversarial loss encouraging it to fool the discriminator, while the discriminator learns to classify real versus generated images accurately.

Keras/TensorFlow Training Loop

The train_step() function in StyleTransfer_Keras.ipynb uses tf.GradientTape to compute simultaneous gradients for both networks. The discriminator loss combines real and fake classification errors, while the generator loss targets classification as real.

cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=False)

def discriminator_loss(real_output, fake_output):
    real_loss = cross_entropy(tf.ones_like(real_output), real_output)
    fake_loss = cross_entropy(tf.zeros_like(fake_output), fake_output)
    return real_loss + fake_loss

def generator_loss(fake_output):
    return cross_entropy(tf.ones_like(fake_output), fake_output)

# Optimizers

gen_opt = tf.keras.optimizers.Adam(1e-4)
disc_opt = tf.keras.optimizers.Adam(1e-4)

@tf.function
def train_step(real_images):
    noise = tf.random.normal([BATCH_SIZE, NOISE_DIM])
    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:
        generated = generator(noise, training=True)

        real_output = discriminator(real_images, training=True)
        fake_output = discriminator(generated, training=True)

        gen_loss = generator_loss(fake_output)
        disc_loss = discriminator_loss(real_output, fake_output)

    grads_gen = gen_tape.gradient(gen_loss, generator.trainable_variables)
    grads_disc = disc_tape.gradient(disc_loss, discriminator.trainable_variables)

    gen_opt.apply_gradients(zip(grads_gen, generator.trainable_variables))
    disc_opt.apply_gradients(zip(grads_disc, discriminator.trainable_variables))

Both optimizers use a learning rate of 1e-4 to ensure stable convergence during adversarial training.

PyTorch Implementation

The lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb notebook provides an equivalent implementation using torch.nn modules. The Generator class mirrors the Keras architecture with ConvTranspose2d, batch normalization, and ReLU activations.

class Generator(nn.Module):
    def __init__(self, nz=100, ngf=64, nc=3):
        super(Generator, self).__init__()
        self.main = nn.Sequential(
            # input Z, goes into a convolution

            nn.ConvTranspose2d(nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),

            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),

            nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),

            nn.ConvTranspose2d(ngf * 2, nc, 4, 2, 1, bias=False),
            nn.Tanh()
        )
    def forward(self, input):
        return self.main(input)

Running Style Transfer Inference

After adversarial training converges, the generator alone functions as the style-transfer model. The inference pipeline loads a content image, scales pixel values to [-1, 1], feeds the tensor through the trained generator, and converts the output back to standard image format.

# Assume `generator` is already trained

def stylise(image_path, output_path):
    img = tf.io.read_file(image_path)
    img = tf.image.decode_image(img, channels=3)
    img = tf.image.resize(img, [28, 28])
    img = (img / 127.5) - 1.0          # scale to [-1, 1]

    # Generate a style-transfer image

    noise_vec = tf.random.normal([1, 100])
    stylised = generator(noise_vec, training=False)

    # Convert back to uint8 and save

    stylised = (stylised[0] + 1.0) * 127.5
    tf.io.write_file(output_path,
                     tf.image.encode_png(tf.cast(stylised, tf.uint8)))

Running this function on MNIST digits produces stylized outputs that mimic the brush strokes and color palettes of the target WikiArt paintings used during training.

Key Files in the Repository

The AI-For-Beginners lesson ships with multiple implementations to suit different learning preferences:

• lessons/4-ComputerVision/10-GANs/StyleTransfer_Keras.ipynb — Complete Keras implementation with data loading, DCGAN architecture, training loop, and inference utilities.
• lessons/4-ComputerVision/10-GANs/StyleTransfer.ipynb — Pure TensorFlow version using low-level tf.nn operations for educational clarity.
• lessons/4-ComputerVision/10-GANs/GANPyTorch.ipynb — PyTorch equivalent showcasing nn.ConvTranspose2d and custom training loops.
• lessons/4-ComputerVision/10-GANs/GANTF.ipynb — Theoretical background explaining DCGAN fundamentals and architectural choices.
• environment.yml — Dependency specification including tensorflow, torch, torchvision, and keras for reproducible execution.

Summary

• The AI-For-Beginners repository provides complete Generative Adversarial Networks (GANs) for style transfer implementations in both TensorFlow/Keras and PyTorch.
• The DCGAN generator architecture uses Conv2DTranspose layers to upsample 100-dimensional noise vectors into 28×28×3 stylized images.
• Training employs binary cross-entropy adversarial loss with Adam optimizers (learning rate 1e-4) for both generator and discriminator networks.
• After convergence, the trained generator transforms arbitrary content images to match artistic styles from datasets like WikiArt without requiring paired training examples.

Frequently Asked Questions

What is the difference between the Keras and pure TensorFlow implementations?

The Keras version (StyleTransfer_Keras.ipynb) uses high-level tf.keras.Sequential APIs for rapid prototyping and readability, while the pure TensorFlow version (StyleTransfer.ipynb) demonstrates low-level operations using tf.nn modules to expose the underlying mechanics of convolutions and gradient updates.

What dataset is used for training the style transfer GAN?

The notebooks utilize the WikiArt dataset for target artistic styles and MNIST or CIFAR-10 images as content sources. The architecture supports any user-provided image directory, allowing customization for specific artistic styles or content domains.

Why does the generator use tanh activation in the final layer?

The tanh activation constrains output pixel values to the range [-1, 1], which matches the normalization applied to training images (scaled by dividing 127.5 and subtracting 1). This symmetric range provides better gradient flow during adversarial training compared to the [0, 1] range of sigmoid activation.

How long does it take to train a style transfer GAN using these notebooks?

Training typically requires 50-100 epochs on a GPU to achieve convincing style transfer results, though timing varies based on dataset size, image resolution, and the complexity of the target artistic style. The environment.yml file lists all required dependencies to run the training pipeline immediately on compatible hardware.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}