How to Implement Style Transfer and Neural Rendering with the AILIA-Models Repository

The AILIA-Models repository provides ready-to-run ONNX implementations of AdaIN-based style transfer and NeRF/TripoSR-based neural rendering that you can execute via CLI or embed in Python applications using the AILIA SDK.

The axinc-ai/ailia-models repository ships a comprehensive collection of computer-vision demos built on the AILIA SDK, a lightweight wrapper around ONNX and TensorRT runtimes. To implement style transfer and neural rendering, you will leverage pre-trained ONNX models for Adaptive Instance Normalization (AdaIN) artistic stylization and Neural Radiance Fields (NeRF) or TripoSR for 3D view synthesis and mesh extraction.

Implementing Style Transfer with Adaptive Instance Normalization

The style transfer pipeline in style_transfer/adain/adain.py implements the AdaIN algorithm to blend the content of one image with the artistic style of another using VGG-based feature extraction and alpha-controlled blending.

Architecture and Core Components

The implementation relies on three core components working sequentially:

• VGG-19 Encoder – The adain-vgg.onnx model extracts 512-channel feature maps from both content and style images.
• AdaIN Layer – Defined in style_transfer/adain/adain_utils.py, the adaptive_instance_normalization function aligns per-channel means and standard deviations of content features to match style features.
• Decoder Network – The adain-decoder.onnx model reconstructs the final RGB image from the normalized features.

The complete forward pass lives in the style_transfer function within adain.py:

def style_transfer(vgg, decoder, content, style, alpha=args.alpha):
    # 1. Encode content and style

    content_f = vgg.predict(content.astype(np.float32))
    style_f   = vgg.predict(style)
    # 2. Adaptive Instance Normalization

    feat = adain_utils.adaptive_instance_normalization(content_f, style_f)
    # 3. Alpha blending controls stylization strength

    feat = feat * alpha + content_f * (1 - alpha)
    # 4. Decode to image space

    return decoder.predict(feat)

Images are pre-processed using image_utils.load_image to 512×512 resolution with normalize_type='255', converting to AILIA's NCHW tensor layout. The helper model_utils.check_and_download_models automatically pulls ONNX weights from Google Cloud Storage when missing.

Running AdaIN Style Transfer

Execute stylization via CLI:

python style_transfer/adain/adain.py \
    -i path/to/content.jpg \
    -t path/to/style.jpg \
    -a 0.8

For programmatic use, load the VGG and decoder networks via the AILIA SDK and call the utility functions directly:

import ailia
import cv2
import numpy as np
from style_transfer.adain import adain_utils, adain

# Initialize models

vgg = ailia.Net('adain-vgg.onnx.prototxt', 'adain-vgg.onnx')
decoder = ailia.Net('adain-decoder.onnx.prototxt', 'adain-decoder.onnx')

def stylize(content_path, style_path, alpha=0.5):
    # Pre-process to 512x512, normalize to [0,255]

    content = adain.load_image(content_path, (512, 512), normalize_type='255', gen_input_ailia=True)
    style = adain.load_image(style_path, (512, 512), normalize_type='255', gen_input_ailia=True)
    
    # Extract features and blend

    content_f = vgg.predict(content.astype(np.float32))
    style_f = vgg.predict(style)
    feat = adain_utils.adaptive_instance_normalization(content_f, style_f)
    feat = feat * alpha + content_f * (1 - alpha)
    output = decoder.predict(feat)
    
    # Convert back to BGR for OpenCV

    img = cv2.cvtColor(output[0].transpose(1, 2, 0), cv2.COLOR_RGB2BGR)
    return np.clip(img * 255 + 0.5, 0, 255).astype(np.uint8)

result = stylize('photo.jpg', 'painting.jpg', alpha=0.7)
cv2.imwrite('output.png', result)

Implementing Neural Rendering with NeRF and TripoSR

The neural rendering implementations in neural_rendering/nerf/ and neural_rendering/tripo_sr/ provide two distinct approaches: continuous volumetric view synthesis and explicit 3D mesh extraction from single images.

NeRF Volume Rendering

The NeRF implementation in neural_rendering/nerf/nerf.py renders novel views by querying an MLP that predicts RGB color and volume density for 3D coordinates.

Key implementation details:

• MLP Network – nerf.opt.onnx predicts rgb + density for any 3D point and view direction.
• Positional Encoding – The get_embedder function in utils_nerf.py applies sinusoidal embeddings to input coordinates, enabling high-frequency detail capture.
• Ray Marching – The render function in utils_nerf.py casts camera rays, samples N_samples points along each ray (configurable via --N_samples), and integrates colors using the volume rendering equation in raw2outputs.

Run NeRF rendering from the command line:

python neural_rendering/nerf/nerf.py \
    --datadir ./data/nerf_llff_data/ \
    --angle 0 \
    --render_factor 4 \
    -i dummy_input.png \
    -o rendered.png

TripoSR 3D Mesh Reconstruction

TripoSR generates textured 3D meshes from single RGB images using a two-stage neural field architecture implemented in neural_rendering/tripo_sr/tripo_sr.py.

Pipeline stages:

1. Encoder – TripoSR.onnx predicts a latent scene code from a foreground-masked 512×512 image.
2. Neural Field Decoder – TripoSR_decoder.onnx queries density and color at arbitrary 3D points.
3. Mesh Extraction – The TSR wrapper class provides extract_mesh(scene_codes, resolution) which runs marching-cubes on the density field.

Pre-processing includes optional background removal via remove_background and foreground resizing via resize_foreground from util.py.

Execute mesh generation:

python neural_rendering/tripo_sr/tripo_sr.py \
    -i portrait.png \
    -o portrait.obj \
    --mc-resolution 256 \
    --foreground-ratio 0.85

For Python integration:

import cv2
import numpy as np
import ailia
from neural_rendering.tripo_sr.util import remove_background, resize_foreground, TSR

# Load encoder and decoder

net_enc = ailia.Net('TripoSR.onnx.prototxt', 'TripoSR.onnx')
net_dec = ailia.Net('TripoSR_decoder.onnx.prototxt', 'TripoSR_decoder.onnx')

# Initialize TSR wrapper with rendering parameters

model = TSR(net_dec, radius=0.87, feature_reduction='concat',
            density_activation='exp', density_bias=-1.0,
            num_samples_per_ray=128)
model.renderer.set_chunk_size(8192)

def mesh_from_image(img_path, resolution=256):
    img = cv2.imread(img_path, cv2.IMREAD_UNCHANGED)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    
    # Pre-processing

    img = remove_background(img)
    img = resize_foreground(img, 0.85) / 255.0
    img = np.stack([[cv2.resize(img, (512, 512), cv2.INTER_LINEAR)]], axis=0)
    
    # Encode and extract

    scene_code = np.array(net_enc.run(img))[0]
    mesh = model.extract_mesh(scene_code, resolution=resolution)[0]
    return mesh

mesh = mesh_from_image('input.png')
mesh.export('output.obj')

Summary

• Style Transfer in AILIA-Models uses AdaIN in style_transfer/adain/adain.py to align VGG-19 feature statistics between content and style images, with alpha blending controlling the stylization strength.
• Neural Rendering offers two modes: NeRF (neural_rendering/nerf/) for volumetric view synthesis via ray marching and positional encoding, and TripoSR (neural_rendering/tripo_sr/) for explicit mesh extraction using encoder-decoder neural fields with marching-cubes.
• All models are automatically downloaded via model_utils.check_and_download_models and run through the unified AILIA SDK interface, supporting both CLI workflows and programmatic Python integration.

Frequently Asked Questions

What is the difference between AdaIN and other style transfer methods in AILIA-Models?

The AdaIN implementation in style_transfer/adain/ performs real-time arbitrary style transfer by aligning feature statistics in VGG-19 space, requiring no training for new styles. Alternative implementations like animeganv2 or beauty_gan use dedicated GAN architectures trained for specific aesthetic outputs, while pix2pixHD targets high-resolution image-to-image translation with paired training data.

How does NeRF rendering performance scale with resolution?

The NeRF renderer in neural_rendering/nerf/utils_nerf.py casts rays per pixel, so rendering time scales quadratically with image dimensions. Use the --render_factor flag to downsample rays for fast previews (e.g., factor 4 reduces resolution by 4×), or adjust --N_samples to trade quality for speed by reducing point sampling along each ray.

Can TripoSR generate meshes from multiple input images?

The current TripoSR implementation in tripo_sr.py processes single images through the TripoSR.onnx encoder to produce scene codes. While the architecture supports batched inference, multi-view fusion would require custom modifications to aggregate multiple scene codes before calling extract_mesh in the TSR class.

Do I need GPU acceleration to run these models?

While the AILIA SDK supports CPU inference via ONNX Runtime, both style transfer and neural rendering involve heavy matrix operations (512-channel convolutions for AdaIN, millions of MLP queries for NeRF). GPU acceleration is strongly recommended for interactive performance, especially when extracting high-resolution meshes with --mc-resolution above 256 in TripoSR.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}