Storage Migration Implications When Updating Deployment Manifests: Akash Network Upgrade Guide

Updating deployment manifests in Akash Network requires a state migration during network upgrades because the Cosmos SDK does not automatically rewrite existing KV store entries when protobuf schemas change, necessitating manual migration from DeploymentV1 legacy keys to the new v2-beta2 IndexedMap format.

Akash Network stores deployment specifications (manifests) in the deployment module’s KV store, and any structural change—such as moving the storage field from the top-level deployment object to groups[i].spec.storage in the v2-beta2 specification—triggers a mandatory state migration. Because the Cosmos SDK preserves existing on-chain data layouts, the network must execute a coordinated upgrade that iterates over legacy keys, transforms the data, and writes it to new collection prefixes before new manifest versions can be accepted.

Why Storage Migrations Are Required for Manifest Updates

The Akash node persists deployment data using protobuf-encoded structs in a key-value database. When the manifest schema evolved from DeploymentV1 to DeploymentV2Beta2, the storage location for volume specifications shifted from the root deployment object into nested group specifications, as documented in _docs/adr/adr-002-manifest-v2beta2.md.

The Cosmos SDK does not perform automatic schema migration on existing state. Consequently, old manifests remain encoded with legacy prefixes (deployment/…) and the DeploymentV1 protobuf representation. Without an explicit migration handler, nodes running the new binary would be unable to locate or decode these legacy entries, leading to state divergence and potential consensus failures.

The State Migration Execution Flow

When a network upgrade activates a new manifest version, the node executes a deterministic migration sequence that restructures the KV store without data loss.

Migration Registration and Upgrade Handlers

The upgrade process begins with a governance Plan registered in app/upgrades.go, which specifies the target block height and binary version. Each module migration is registered via utypes.RegisterMigration located in upgrades/types/types.go. For the deployment module’s transition to version 8 (manifest v2-beta2), the registration links the module name to a concrete migration function:

// In upgrades/types/types.go
utypes.RegisterMigration(
    "deployment",           // module name
    8,                     // target version
    func(m utypes.Migrator) utypes.Migration {
        return deploymentMigrations{Migrator: m}
    },
)

KV Store Iterator and Data Transformation

The concrete migration logic resides in upgrades/software/v1.2.0/market.go. The handler constructs a new collections.IndexedMap for the updated schema using new prefixes (mkeys.OrderPrefixNew, BidPrefixNew, LeasePrefixNew). It then iterates over all legacy keys using storetypes.KVStorePrefixIterator with the old prefixes (OrderPrefix, BidPrefix, LeasePrefix).

For each entry, the handler unmarshals the old protobuf, writes the transformed object into the new collection, and deletes the legacy key. After successful migration, it purges obsolete reverse-lookup keys (BidPrefixReverse, LeasePrefixReverse). The handler logs migration counts (orderCount, bidCount, leaseCount) via ctx.Logger().Info("market store migration complete", …), providing auditability.

Atomic Migration and Rollback Safety

The migration operates atomically within the upgrade block. If any step fails—such as a decode error or write failure—the handler returns an error that aborts the entire upgrade, preventing the chain from entering an inconsistent state. Because the old keys remain untouched until successfully written to the new collection, operators can safely re-run the upgrade if a crash occurs mid-migration.

Critical Risks and Prerequisites

Updating deployment manifests without proper network coordination creates fatal compatibility errors.

Network Upgrade Coordination

You must schedule and execute the network upgrade before submitting any v2-beta2 manifests. The SDK validates incoming transactions against the active protobuf definitions; if the on-chain state still contains the old format, new transactions will be rejected. Submitting a v2-beta2 deployment to a pre-upgrade chain results in immediate transaction failure because the new storage keys do not yet exist.

Validator Binary Updates

All full nodes and validators must upgrade to the binary version containing the migration handler before the upgrade height. Nodes running legacy software interpret the store using the old schema and will panic when encountering the new IndexedMap keys, causing the node to crash and potentially halting consensus if a supermajority of validators is affected.

Client SDK Compatibility

Client applications must use SDK versions that include the v2-beta2 protobuf definitions. Submitting manifests using outdated client libraries generates transactions encoded with the deprecated DeploymentV1 layout, which the new chain state will reject post-migration.

Implementing and Testing Migrations

Developers must pair manifest schema changes with rigorous migration testing.

Submitting a V2Beta2 Deployment Manifest

When constructing deployments using the Go SDK, explicitly build DeploymentV2Beta2 objects with storage defined within group specifications:

import (
    akash "github.com/akash-network/go-sdk"
    dtypes "github.com/akash-network/node/x/deployment/types"
)

func submitV2Beta2Deployment(cli *akash.Client, owner sdk.AccAddress) error {
    // Build a DeploymentV2Beta2 protobuf
    dep := &dtypes.DeploymentV2Beta2{
        Groups: []dtypes.Group{
            {
                Spec: &dtypes.GroupSpec{
                    // Storage now lives inside the group spec
                    Volumes: []dtypes.Volume{
                        {
                            Source: dtypes.VolumeSource{
                                Type: dtypes.StorageTypeEphemeral,
                            },
                        },
                    },
                },
            },
        },
    }

    msg := &dtypes.MsgCreateDeployment{
        ID:        dtypes.DeploymentID{Owner: owner.String(), DSeq: 1},
        Deployment: dep,
    }

    _, err := cli.Tx.BroadcastTx(msg)
    return err
}

Ensure the client binary is version v3.0.0 or later to include the necessary protobuf definitions.

Running Local Upgrade Validation

The repository provides an end-to-end test suite to validate migrations before mainnet deployment:

# From the repository root

make test-upgrade

This command executes tests/upgrade/upgrade_test.go, which spins up an in-memory node with pre-upgrade deployment data, triggers the upgrade at a simulated height, and verifies that storage keys are correctly rewritten to the new prefixes and that legacy entries are purged.

Summary

• State migrations are mandatory when deployment manifest schemas change because the Cosmos SDK does not rewrite existing KV entries automatically.
• Migration logic is implemented in upgrades/software/v1.2.0/market.go and registered via upgrades/types/types.go.
• Data transformation uses KVStorePrefixIterator to migrate legacy DeploymentV1 entries into new collections.IndexedMap structures with updated prefixes.
• Upgrade coordination is critical: all validators must upgrade before the target height, and no v2-beta2 manifests can be submitted until the migration completes successfully.
• Atomic safety ensures that failed migrations abort the upgrade, preventing state corruption, while idempotent processing allows restart after crashes.

Frequently Asked Questions

What happens if I submit a v2-beta2 manifest before the network upgrade completes?

The transaction will be rejected. The SDK validates manifests against the active on-chain protobuf definitions. Since the new IndexedMap keys and DeploymentV2Beta2 schema do not exist until the migration runs, the transaction fails validation before entering the mempool.

How does the migration handle crashes or partial failures?

The migration handler records progress counts for each object type (orderCount, bidCount, leaseCount) but leaves original keys untouched until the new entry is successfully written. If the process crashes, operators can restart the node; the upgrade handler will re-execute and skip already-migrated entries. If any irrecoverable error occurs, the handler returns an error that aborts the upgrade, preserving the pre-upgrade state.

Do all validators need to upgrade simultaneously?

Yes. All validators and full nodes must run the binary containing the migration handler before the upgrade height. Nodes running old software lack the new collection prefixes and will panic when attempting to read state written by upgraded nodes, potentially causing consensus failures and chain halts.

How can I verify that my deployment data migrated correctly?

Check the node logs for the migration completion message: ctx.Logger().Info("market store migration complete", …). Additionally, run the built-in upgrade tests using make test-upgrade, which validates that legacy keys are removed and new IndexedMap entries contain correctly transformed data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}