How GANs Enable Artistic Style Transfer in the AI-For-Beginners Curriculum

The Microsoft AI-For-Beginners repository demonstrates artistic style transfer using a Generative Adversarial Network (GAN) with a U-Net generator, PatchGAN discriminator, and combined adversarial, content, and Gram-matrix style losses.

The Microsoft/AI-For-Beginners curriculum provides hands-on lessons in computer vision and deep learning. One advanced module demonstrates how Generative Adversarial Networks (GANs) perform artistic style transfer by repainting ordinary photographs in the visual style of famous artworks. According to the repository's source code in lessons/4-ComputerVision/10-GANs/README.md, this implementation uses an image-to-image translation architecture that balances content preservation with stylistic transformation through carefully designed loss functions.

GAN Architecture for Style Transfer

The system follows a conditional GAN framework where two networks compete during training. The architecture definitions reside in lessons/4-ComputerVision/10-GANs/StyleTransfer_Keras.ipynb (TensorFlow) and lessons/4-ComputerVision/10-GANs/StyleTransfer.ipynb (PyTorch).

Generator (U-Net with Residual Blocks)

The Generator accepts a content image and produces a stylized output. It employs a deep encoder-decoder structure with skip connections that preserve spatial geometry while injecting artistic features. The encoder uses convolutional layers with instance normalization, while the decoder uses transposed convolutions for upsampling. Residual blocks between the encoder and decoder facilitate style blending.

def build_generator(input_shape):
    inputs = tf.keras.Input(shape=input_shape)

    # Encoder

    e1 = tf.keras.layers.Conv2D(64, 7, strides=1, padding='same')(inputs)
    e1 = tf.keras.layers.InstanceNormalization()(e1)
    e1 = tf.keras.layers.Activation('relu')(e1)

    e2 = tf.keras.layers.Conv2D(128, 3, strides=2, padding='same')(e1)
    e2 = tf.keras.layers.InstanceNormalization()(e2)
    e2 = tf.keras.layers.Activation('relu')(e2)

    e3 = tf.keras.layers.Conv2D(256, 3, strides=2, padding='same')(e2)
    e3 = tf.keras.layers.InstanceNormalization()(e3)
    e3 = tf.keras.layers.Activation('relu')(e3)

    # Residual blocks (style-preserving)

    x = e3
    for _ in range(9):
        x = residual_block(x, 256)

    # Decoder (upsampling)

    d1 = tf.keras.layers.Conv2DTranspose(128, 3, strides=2, padding='same')(x)
    d1 = tf.keras.layers.InstanceNormalization()(d1)
    d1 = tf.keras.layers.Activation('relu')(d1)

    d2 = tf.keras.layers.Conv2DTranspose(64, 3, strides=2, padding='same')(d1)
    d2 = tf.keras.layers.InstanceNormalization()(d2)
    d2 = tf.keras.layers.Activation('relu')(d2)

    outputs = tf.keras.layers.Conv2D(3, 7, padding='same', activation='tanh')(d2)
    return tf.keras.Model(inputs, outputs, name='Generator')

PatchGAN Discriminator

The Discriminator evaluates whether image patches are real (matching the target style distribution) or fake. This PatchGAN architecture evaluates local regions rather than global images, forcing the generator to produce high-frequency details. The implementation uses a series of convolutional layers with leaky ReLU activations and instance normalization, outputting a single-channel real-vs-fake map.

def build_discriminator(input_shape):
    inputs = tf.keras.Input(shape=input_shape)

    x = tf.keras.layers.Conv2D(64, 4, strides=2, padding='same')(inputs)
    x = tf.keras.layers.LeakyReLU(0.2)(x)

    x = tf.keras.layers.Conv2D(128, 4, strides=2, padding='same')(x)
    x = tf.keras.layers.InstanceNormalization()(x)
    x = tf.keras.layers.LeakyReLU(0.2)(x)

    x = tf.keras.layers.Conv2D(256, 4, strides=2, padding='same')(x)
    x = tf.keras.layers.InstanceNormalization()(x)
    x = tf.keras.layers.LeakyReLU(0.2)(x)

    x = tf.keras.layers.Conv2D(512, 4, strides=1, padding='same')(x)
    x = tf.keras.layers.InstanceNormalization()(x)
    x = tf.keras.layers.LeakyReLU(0.2)(x)

    outputs = tf.keras.layers.Conv2D(1, 4, padding='same')(x)   # PatchGAN output

    return tf.keras.Model(inputs, outputs, name='Discriminator')

Loss Functions and Training Strategy

The training objective combines three distinct loss terms that guide the generator toward realistic, stylistically accurate outputs.

Adversarial and Content Losses

The adversarial loss trains the generator to fool the discriminator into classifying generated images as real. The content loss preserves the structural integrity of the original scene by computing the L2 distance between VGG-19 feature maps of the content and generated images. This ensures that while the style changes, the spatial arrangement of objects remains intact.

Style Loss via Gram Matrices

Style loss captures texture statistics using Gram matrices computed from intermediate VGG-19 activations. By comparing the correlations between different filter responses, the network transfers the statistical distribution of colors and textures from the style image without copying pixel values directly.

def gram_matrix(feature):
    # feature shape: (batch, height, width, channels)

    batch, h, w, c = tf.shape(feature)[0], tf.shape(feature)[1], tf.shape(feature)[2], tf.shape(feature)[3]
    reshaped = tf.reshape(feature, (batch, h * w, c))
    gram = tf.matmul(reshaped, reshaped, transpose_a=True) / tf.cast(h * w * c, tf.float32)
    return gram

def style_loss(generated, style):
    # Use VGG-19 layers 'block1_conv1', 'block2_conv1', 'block3_conv1', 'block4_conv1'

    vgg = tf.keras.applications.VGG19(include_top=False, weights='imagenet')
    layer_names = ['block1_conv1','block2_conv1','block3_conv1','block4_conv1']
    outputs = [vgg.get_layer(name).output for name in layer_names]
    model = tf.keras.Model(vgg.input, outputs)

    gen_feats = model(generated)
    style_feats = model(style)

    loss = 0
    for gf, sf in zip(gen_feats, style_feats):
        loss += tf.reduce_mean(tf.square(gram_matrix(gf) - gram_matrix(sf)))
    return loss

Training Loop Implementation

The optimization alternates between updating the discriminator and the generator using the Adam optimizer with a learning rate of 2e-4 and β1=0.5, β2=0.999. The training step is wrapped in @tf.function for performance in the Keras implementation.

@tf.function
def train_step(content_img, style_img):
    with tf.GradientTape(persistent=True) as tape:
        generated = generator(content_img, training=True)

        # Discriminator loss

        real_pred = discriminator(style_img, training=True)
        fake_pred = discriminator(generated, training=True)
        d_loss = tf.reduce_mean(tf.square(real_pred - 1)) + tf.reduce_mean(tf.square(fake_pred))

        # Generator loss

        adv_loss = tf.reduce_mean(tf.square(fake_pred - 1))
        c_loss = tf.reduce_mean(tf.square(vgg_features(content_img) - vgg_features(generated)))
        s_loss = style_loss(generated, style_img)
        g_loss = adv_loss + 10 * c_loss + 100 * s_loss   # weighting as in the notebook

    # Apply gradients

    grads_g = tape.gradient(g_loss, generator.trainable_variables)
    grads_d = tape.gradient(d_loss, discriminator.trainable_variables)
    optimizer_g.apply_gradients(zip(grads_g, generator.trainable_variables))
    optimizer_d.apply_gradients(zip(grads_d, discriminator.trainable_variables))

Framework-Specific Implementations

The repository provides complete, runnable examples in both TensorFlow/Keras and PyTorch to accommodate different learner preferences.

TensorFlow/Keras Version

lessons/4-ComputerVision/10-GANs/StyleTransfer_Keras.ipynb contains the full TensorFlow 2 implementation. It defines the U-Net generator, PatchGAN discriminator, and training loop using tf.keras layers and tf.GradientTape for automatic differentiation.

PyTorch Version

lessons/4-ComputerVision/10-GANs/StyleTransfer.ipynb implements identical logic using PyTorch modules. This version demonstrates torch.nn.Module definitions for the generator and discriminator, and uses PyTorch's optimizer and loss computation APIs to achieve the same artistic style transfer results.

Summary

• Generative Adversarial Networks (GANs) in the AI-For-Beginners curriculum perform artistic style transfer through an adversarial training process involving a generator and discriminator.
• The Generator uses a U-Net architecture with residual blocks and skip connections to preserve content structure while applying style.
• The Discriminator employs a PatchGAN design to evaluate local image patches, encouraging high-frequency realism in generated textures.
• Style loss utilizes Gram matrices computed from VGG-19 feature maps to capture and transfer the statistical texture of reference artworks.
• The repository provides dual implementations in lessons/4-ComputerVision/10-GANs/StyleTransfer_Keras.ipynb and lessons/4-ComputerVision/10-GANs/StyleTransfer.ipynb, supporting both TensorFlow and PyTorch workflows.

Frequently Asked Questions

What is the difference between the Generator and Discriminator in style transfer GANs?

The Generator creates the stylized image from the content input, while the Discriminator evaluates whether patches of the generated image match the statistical distribution of real artwork from the target style. The generator tries to minimize the adversarial loss to fool the discriminator, whereas the discriminator tries to maximize its accuracy in distinguishing real style images from generated ones.

Why does the repository use VGG-19 for calculating style loss?

The implementation leverages a pre-trained VGG-19 network because its intermediate convolutional layers capture hierarchical visual features ranging from low-level edges to high-level textures. By computing Gram matrices from these feature maps, the network quantifies the texture correlations that define artistic style without requiring pixel-perfect matches, allowing for flexible artistic expression.

How do the Keras and PyTorch implementations differ in the AI-For-Beginners repo?

Both notebooks in lessons/4-ComputerVision/10-GANs/ implement the same U-Net generator and PatchGAN discriminator architectures, but differ in API specifics. The Keras version uses tf.keras.layers and tf.GradientTape for training steps, while the PyTorch version implements custom nn.Module classes and uses torch.optim.Adam for parameter updates. The underlying mathematical operations and loss weightings remain consistent between frameworks.

What is a PatchGAN discriminator and why is it used here?

A PatchGAN discriminator classifies individual N×N patches of an image as real or fake rather than returning a single scalar score for the entire image. This design is used in the style transfer implementation because it enforces local consistency in texture generation, effectively penalizing blurry or inconsistent regions while allowing the generator flexibility in global composition. The patch-level feedback produces sharper, more visually coherent artistic results.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}