# TensorFlow vs PyTorch vs ONNX for Production: Architectural Trade-offs and Deployment Workflows

> Compare TensorFlow PyTorch and ONNX for production ML deployment. Explore architectural trade-offs and deployment workflows for seamless cross-platform execution. Optimize your ML models.

- Repository: [Harvard Edge Computing/cs249r_book](https://github.com/harvard-edge/cs249r_book)
- Tags: deep-dive
- Published: 2026-02-19

---

**TensorFlow excels at large-scale production serving with static graphs and XLA optimization, PyTorch dominates research-to-production workflows via TorchScript and dynamic flexibility, while ONNX serves as the interoperability bridge enabling cross-platform deployment across heterogeneous hardware.**

Choosing the right deep learning framework for production requires understanding fundamental architectural trade-offs between static and dynamic computation graphs. This analysis draws from the Harvard Edge Computing course materials (`harvard-edge/cs249r_book`) to compare how TensorFlow, PyTorch, and ONNX handle model export, quantization, and serving in real-world deployment pipelines.

## Architectural Foundations: Static vs Dynamic Graphs

### TensorFlow's Static Graph Paradigm

TensorFlow was designed with a **production-first, static-graph** architecture that enables aggressive compile-time optimizations. In `book/quarto/contents/core/frameworks/frameworks.qmd`, the implementation notes that TensorFlow historically relied on static graphs allowing global optimizations such as operator fusion and memory planning.

Modern TensorFlow 2.x introduces eager execution for debugging while retaining static graph benefits through `tf.function` decorators. The **XLA (Accelerated Linear Algebra)** compiler further optimizes these graphs for specific hardware targets, generating fused kernels that reduce memory bandwidth and latency.

### PyTorch's Dynamic Graph Approach

PyTorch adopts a **research-first, dynamic-graph** "define-by-run" paradigm where the computation graph is built during forward pass execution. This approach prioritizes flexibility and intuitive debugging but requires explicit conversion steps for production deployment.

The framework provides **TorchScript** via `torch.jit.script` and `torch.jit.trace` to convert dynamic models into static graph representations suitable for production. As noted in the source materials, this conversion bridges the gap between PyTorch's research flexibility and the static graph requirements of efficient serving infrastructure.

### ONNX as the Interoperability Standard

ONNX (Open Neural Network Exchange) functions as a **framework-agnostic interoperability layer** rather than a training framework. According to `book/quarto/contents/core/frameworks/frameworks.qmd#L3080`, ONNX defines a standardized graph representation using protocol buffers that captures operations and metadata independent of any specific runtime.

This standardization enables models trained in PyTorch to be exported via `torch.onnx.export` and subsequently executed in TensorFlow Serving, TensorRT, or ONNX Runtime without vendor lock-in. The format supports both floating-point and quantized representations, making it suitable for edge deployment scenarios.

## Production Deployment Workflows

### TensorFlow: Training to TensorFlow Lite

TensorFlow provides end-to-end tooling for converting trained models into optimized deployment artifacts. The workflow typically involves training with Keras APIs, then converting to TensorFlow Lite for mobile and edge deployment.

```python
import tensorflow as tf

# Define and train a simple MNIST classifier

model = tf.keras.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28, 1)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(10)
])

model.compile(
    optimizer='adam',
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy']
)
model.fit(train_ds, epochs=5)

# Export to TensorFlow Lite with post-training quantization

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]  # Enables INT8 PTQ

tflite_model = converter.convert()

with open('mnist.tflite', 'wb') as f:
    f.write(tflite_model)

```

The [`tinytorch/src/15_quantization/ABOUT.md`](https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/15_quantization/ABOUT.md) file explicitly notes that TensorFlow Lite utilizes **symmetric INT8 quantization** with per-channel scaling, a technique shared across production frameworks.

**Edge inference** uses the TensorFlow Lite Interpreter:

```python
import numpy as np
import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path='mnist.tflite')
interpreter.allocate_tensors()
input_idx = interpreter.get_input_details()[0]['index']
output_idx = interpreter.get_output_details()[0]['index']

img = np.expand_dims(test_image, axis=0).astype(np.float32)
interpreter.set_tensor(input_idx, img)
interpreter.invoke()
pred = interpreter.get_tensor(output_idx)
print('Predicted class:', np.argmax(pred))

```

### PyTorch: TorchScript and TorchServe

PyTorch models require conversion from dynamic eager mode to static representations for production. The framework provides TorchScript through tracing or scripting, followed by TorchServe for scalable deployment.

```python
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(28*28, 128)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        return self.fc2(self.relu(self.fc1(self.flatten(x))))

model = Net()
model.eval()

# Convert to TorchScript via scripting

scripted_model = torch.jit.script(model)
scripted_model.save('mnist.pt')

```

**Deployment with TorchServe**:

```bash
torchserve --start --model-store model_store \
           --models mnist=mnist.pt

curl -X POST http://127.0.0.1:8080/predictions/mnist -T sample.npy

```

PyTorch Mobile provides additional optimization for iOS and Android, utilizing the same symmetric INT8 quantization approach noted in [`tinytorch/src/15_quantization/15_quantization.py`](https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/15_quantization/15_quantization.py).

### ONNX: Cross-Platform Export and Runtime

ONNX bridges framework ecosystems by standardizing model representation. Models trained in PyTorch can be exported to ONNX format and executed via ONNX Runtime, often delivering superior CPU inference performance.

```python
import torch
import torch.onnx
import onnxruntime as rt
import numpy as np

# Assume 'model' is the trained PyTorch Net from previous section

dummy_input = torch.randn(1, 1, 28, 28)

# Export to ONNX

torch.onnx.export(
    model,
    dummy_input,
    "mnist.onnx",
    input_names=['input'],
    output_names=['output'],
    opset_version=13,
    dynamic_axes={'input': {0: 'batch_size'}, 'output': {0: 'batch_size'}}
)

# Inference with ONNX Runtime

sess = rt.InferenceSession("mnist.onnx")
input_name = sess.get_inputs()[0].name

# Run inference

outputs = sess.run(None, {input_name: dummy_input.numpy()})
predicted_class = np.argmax(outputs[0])
print(f'Predicted class: {predicted_class}')

```

According to `book/quarto/contents/core/benchmarking/benchmarking.qmd#L1455`, ONNX Runtime consistently outperforms native TensorFlow and PyTorch in CPU inference benchmarks due to aggressive graph fusion and memory pooling optimizations.

## Performance and Quantization Comparison

| Feature | TensorFlow | PyTorch | ONNX |
|---------|------------|---------|------|
| **Graph Type** | Static (with eager fallback) | Dynamic (convertible to static) | Static (framework-agnostic) |
| **Primary Runtime** | TensorFlow Serving, TFX, LiteRT | TorchServe, PyTorch Mobile | ONNX Runtime, TensorRT, OpenVINO |
| **Quantization** | PTQ/QAT with symmetric INT8 | `torch.quantization` with symmetric INT8 | PTQ tools accepting both TF/PyTorch models |
| **CPU Inference** | Good | Good | Superior (graph fusion optimizations) |
| **Hardware Targets** | TPU, GPU, CPU, Edge TPU | CUDA, ROCm, CPU, Mobile | CPU, GPU, TPU, NPU, FPGA |

The repository's `tinytorch/src/15_quantization/ABOUT.md#L355` explicitly confirms that **symmetric INT8 quantization** is the shared standard across TensorFlow Lite, PyTorch Mobile, and ONNX Runtime, enabling identical quantization schemes when converting between frameworks.

## Key Source Files

| File | Purpose | GitHub Link |
|------|---------|-------------|
| `book/quarto/contents/core/frameworks/frameworks.qmd` | Architectural deep-dive into static vs dynamic graphs and ecosystem comparisons | [View on GitHub](https://github.com/harvard-edge/cs249r_book/blob/dev/book/quarto/contents/core/frameworks/frameworks.qmd) |
| `book/quarto/contents/core/benchmarking/benchmarking.qmd` | Performance benchmarks showing ONNX Runtime CPU advantages | [View on GitHub](https://github.com/harvard-edge/cs249r_book/blob/dev/book/quarto/contents/core/benchmarking/benchmarking.qmd) |
| [`tinytorch/src/15_quantization/ABOUT.md`](https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/15_quantization/ABOUT.md) | Documentation on symmetric INT8 quantization shared across frameworks | [View on GitHub](https://github.com/harvard-edge/cs249r_book/blob/dev/tinytorch/src/15_quantization/ABOUT.md) |
| [`tinytorch/src/15_quantization/15_quantization.py`](https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/15_quantization/15_quantization.py) | Educational implementation of quantization techniques | [View on GitHub](https://github.com/harvard-edge/cs249r_book/blob/dev/tinytorch/src/15_quantization/15_quantization.py) |

## Summary

- **TensorFlow** provides production-ready static graphs with XLA compilation and comprehensive serving infrastructure through TensorFlow Serving and TFX, making it ideal for large-scale cloud deployment.
- **PyTorch** offers superior research flexibility through dynamic computation graphs, with production pathways via TorchScript and TorchServe that require explicit conversion but maintain debugging capabilities.
- **ONNX** acts as the interoperability standard, enabling models from any framework to run optimized inference through ONNX Runtime, which consistently outperforms native frameworks on CPU workloads according to the repository benchmarks.
- All three frameworks share **symmetric INT8 quantization** techniques as implemented in the educational TinyTorch reference, ensuring consistent model compression across TensorFlow Lite, PyTorch Mobile, and ONNX Runtime.

## Frequently Asked Questions

### When should I choose TensorFlow over PyTorch for production?

Choose TensorFlow when your deployment requires large-scale distributed training, automatic graph partitioning, or integration with Google Cloud AI services like Vertex AI. TensorFlow's static graph architecture enables aggressive XLA compiler optimizations and robust model versioning through TensorFlow Serving, making it superior for high-throughput cloud inference pipelines that demand millisecond-level latency at scale.

### Can PyTorch models run in TensorFlow Serving?

Native PyTorch models cannot run directly in TensorFlow Serving, but you can bridge this gap by exporting PyTorch models to the **ONNX** format using `torch.onnx.export`, then converting the ONNX model to TensorFlow's SavedModel format using tools like `onnx-tf`. Alternatively, serve PyTorch models directly using **TorchServe**, which provides equivalent functionality to TensorFlow Serving including model versioning, A/B testing, and dynamic batching.

### Does ONNX support quantization for edge deployment?

Yes, ONNX fully supports post-training quantization (PTQ) and quantization-aware training (QAT) for edge deployment. The ONNX Runtime provides quantization tools that accept models from both TensorFlow and PyTorch, applying hardware-specific calibration. According to [`tinytorch/src/15_quantization/ABOUT.md`](https://github.com/harvard-edge/cs249r_book/blob/main/tinytorch/src/15_quantization/ABOUT.md), ONNX Runtime utilizes the same **symmetric INT8 quantization** scheme as TensorFlow Lite and PyTorch Mobile, ensuring consistent numerical behavior across frameworks when deploying to resource-constrained edge devices.

### Which framework offers the best CPU inference performance?

**ONNX Runtime** consistently delivers superior CPU inference performance compared to native TensorFlow and PyTorch implementations. According to benchmarks referenced in `book/quarto/contents/core/benchmarking/benchmarking.qmd#L1455`, ONNX Runtime outperforms native frameworks through aggressive graph fusion, memory pooling optimizations, and kernel implementations specifically tuned for CPU architectures. While TensorFlow and PyTorch require conversion to ONNX for these benefits, the performance gains—particularly for batch inference on Intel and AMD processors—often justify the additional export step.