# How to Work Around Network Restrictions for Downloading External Datasets in AI-For-Beginners

> Bypass network restrictions for AI-For-Beginners dataset downloads. Explore mirroring, cache pre-population, and proxy solutions for restricted environments.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-20

---

**Mirror datasets to internal storage, pre-populate framework caches, or use proxy configurations to bypass firewall blocks when running Microsoft AI-For-Beginners notebooks in restricted environments.**

The **microsoft/AI-For-Beginners** repository provides a comprehensive curriculum for learning artificial intelligence, but many lessons depend on fetching external datasets at runtime. When corporate firewalls, offline labs, or CI pipelines block internet access, notebooks fail with timeout or connection errors. This guide explains exactly where the repository triggers downloads and provides six proven strategies to work around network restrictions while preserving the learning experience.

## Where AI-For-Beginners Fetches External Data

Understanding the download locations helps you target the right intervention points.

### Direct `wget` Calls in Computer Vision Lessons

In [`lessons/4-ComputerVision/08-TransferLearning/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/README.md), the curriculum retrieves the Kaggle Cats-vs-Dogs dataset directly from Microsoft's public CDN:

```bash
!wget -P data https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_5340.zip

```

This pattern appears in several README files that accompany hands-on labs.

### PyTorch torchvision On-Demand Downloads

Notebooks like `lessons/4-ComputerVision/09-Autoencoders/AutoEncodersPyTorch.ipynb` and `lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb` use:

```python
torchvision.datasets.MNIST('.', download=True, ...)

```

The `download=True` parameter triggers automatic retrieval from PyTorch's hosted mirrors when the dataset is not cached locally.

### TensorFlow/Keras Implicit Weight Downloads

When code calls `tf.keras.applications.VGG16(weights='imagenet')` or similar pretrained models, TensorFlow downloads ImageNet weights to `~/.keras/models/` on first use. This occurs in transfer learning lessons throughout `lessons/4-ComputerVision/08-TransferLearning/`.

### Oxford-IIIT Pets Dataset

The transfer learning lab in [`lessons/4-ComputerVision/08-TransferLearning/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/lab/README.md) pulls images directly from the VGG group's servers via public download links embedded in notebook cells.

## Why Direct Downloads Are Architected This Way

The repository maintains `download=True` defaults and `wget` one-liners for three specific reasons:

- **Beginner accessibility** — eliminates manual data preparation steps that could derail new learners
- **Version consistency** — fixed URLs ensure identical dataset versions across all learners
- **Storage efficiency** — the `translations/` folder can be sparse-checked out, while datasets remain on-demand

However, this design assumes unrestricted outbound HTTPS access, which fails in many enterprise environments.

## Six Proven Network Restriction Workarounds

### 1. Mirror Datasets to Internal Storage

**Best for:** Corporate environments with internal object storage or HTTP servers.

Download once from an unrestricted machine, then host internally:

```bash

# On machine with internet access

wget -O kagglecatsanddogs_5340.zip \
  https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_5340.zip

# Upload to Azure Blob Storage

az storage blob upload \
  --account-name <storage_account> \
  --container-name datasets \
  --name kagglecatsanddogs_5340.zip \
  --file kagglecatsanddogs_5340.zip

```

Modify the notebook cell to reference your internal endpoint:

```python
!wget -P data https://<storage_account>.blob.core.windows.net/datasets/kagglecatsanddogs_5340.zip

```

### 2. Configure Proxy-Aware Downloads

**Best for:** Environments with mandatory HTTP/HTTPS proxies.

Set proxy environment variables before download commands:

```bash
export http_proxy=http://proxy.mycompany.com:8080
export https_proxy=$http_proxy
!wget -e use_proxy=yes -e https_proxy=$https_proxy -P data https://download.microsoft.com/...

```

For Python-based downloads, set before importing frameworks:

```python
import os
os.environ['http_proxy'] = 'http://proxy.mycompany.com:8080'
os.environ['https_proxy'] = os.environ['http_proxy']

```

### 3. Pre-Populate Framework Cache Directories

**Best for:** Air-gapped machines or CI runners with no internet access.

Both TensorFlow and PyTorch cache downloaded artifacts in predictable locations:

| Framework | Cache Location | Typical Files |
|-----------|---------------|---------------|
| TensorFlow/Keras | `~/.keras/models/` | `vgg16_weights_tf_dim_ordering_tf_kernels.h5` |
| PyTorch | `~/.cache/torch/` | `checkpoints/mnist/MNIST/raw/*.gz` |

Pre-seed these directories on your restricted machine:

```bash
mkdir -p ~/.keras/models
cp vgg16_weights_tf_dim_ordering_tf_kernels.h5 ~/.keras/models/

```

Now `tf.keras.applications.VGG16(weights='imagenet')` loads from cache without network calls.

### 4. Use Conda Offline Channels

**Best for:** Environments where conda packages are pre-approved but internet is blocked.

Some datasets ship within conda packages. Install from a local channel:

```bash

# On connected machine, download package

conda download torchvision -c pytorch --download-only

# Transfer to restricted environment, then install offline

conda install --offline -c ./local-channel torchvision

```

Note: The repository's [`environment.yml`](https://github.com/microsoft/AI-For-Beginners/blob/main/environment.yml) includes `torchvision` and `tensorflow` but does **not** embed datasets — you'll still need the weights or data files themselves.

### 5. Substitute Synthetic Data for Demonstrations

**Best for:** Debugging notebook logic when no real data is available.

Replace real datasets with API-compatible synthetic versions:

```python
import torch
from torchvision import transforms

class TinyCIFAR(torch.utils.data.Dataset):
    def __init__(self, size=100):
        self.size = size
        self.transform = transforms.ToTensor()
    
    def __len__(self):
        return self.size
    
    def __getitem__(self, idx):
        img = torch.randn(3, 32, 32)
        label = torch.randint(0, 10, (1,)).item()
        return self.transform(img), label

trainset = TinyCIFAR()

```

Swap `torchvision.datasets.CIFAR10(..., download=True, ...)` with `TinyCIFAR()`. The training loop executes normally, demonstrating the workflow without external dependencies.

### 6. Use Cloud Workspaces as Data Gateways

**Best for:** Temporary access to restricted data for offline transfer.

Spin up an Azure ML Compute instance or GitHub Codespace with unrestricted internet:

1. Clone `microsoft/AI-For-Beginners` in the cloud workspace
2. Run cells to trigger all downloads
3. Commit the populated `data/` folder or sync via `rsync` to your restricted environment

```bash

# Sync data back from Codespace to local machine

rsync -avz codespace:/workspaces/AI-For-Beginners/data/ ./data/

```

## Production-Ready Pattern: Graceful Fallback Downloads

Add this header to any notebook for automatic handling of network failures:

```python
import os
import shutil
import urllib.request

DATA_DIR = "./data"
os.makedirs(DATA_DIR, exist_ok=True)

url = "https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_5340.zip"
local_path = os.path.join(DATA_DIR, "kagglecatsanddogs_5340.zip")

if not os.path.exists(local_path):
    try:
        urllib.request.urlretrieve(url, local_path)
        print("Download successful")
    except Exception as e:
        print(f"Download failed: {e}")
        # Fall back to pre-bundled placeholder

        placeholder = "assets/kagglecatsanddogs_placeholder.zip"
        if os.path.exists(placeholder):
            shutil.copyfile(placeholder, local_path)
            print("Using placeholder dataset")
        else:
            raise RuntimeError("No dataset available — network blocked and no fallback found")

```

This pattern preserves the original curriculum intent while automatically handling the most common network restriction scenarios.

## Summary

| Approach | When to Use | Modification Required |
|----------|-------------|----------------------|
| Internal mirroring | Persistent corporate restrictions | Update URLs in notebook cells |
| Proxy configuration | Proxy-required environments | Set `http_proxy`/`https_proxy` |
| Cache pre-population | Fully air-gapped systems | Copy files to `~/.keras/` or `~/.cache/torch/` |
| Conda offline install | Package-approved, data-blocked environments | Build local conda channel |
| Synthetic data | Debugging or demonstrations | Replace dataset constructor |
| Cloud gateway | One-time data acquisition | Sync from cloud workspace |

All techniques maintain compatibility with the original `microsoft/AI-For-Beginners` notebook structure in [`lessons/4-ComputerVision/08-TransferLearning/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/README.md), `lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb`, and related curriculum files.

## Frequently Asked Questions

### Where does AI-For-Beginners store downloaded files?

PyTorch datasets cache to `~/.cache/torch/` and TensorFlow/Keras weights store in `~/.keras/models/`. Notebook-specific downloads typically land in a local `data/` subdirectory as specified by `-P data` in `wget` commands.

### Can I run AI-For-Beginners completely offline?

Yes, after initial setup. Pre-download all datasets on a connected machine, copy the `data/` folder and framework caches to your offline machine, then modify notebooks to use local paths. The [`environment.yml`](https://github.com/microsoft/AI-For-Beginners/blob/main/environment.yml) contains all Python dependencies installable via `conda install --offline`.

### Why do some notebooks fail immediately while others fail mid-execution?

Notebooks fail at different points depending on when the download trigger executes. Explicit `!wget` calls in early cells fail at kernel startup, while `download=True` in dataset constructors fails when that specific cell runs. Pre-populating caches addresses both failure modes.

### Does using synthetic data invalidate the learning experience?

No — synthetic data preserves the API structure and training workflow. While results differ quantitatively from real datasets, the pedagogical value of understanding model architecture, loss functions, and optimization remains intact. Replace with real data when network access becomes available.