# How to Implement Database Persistence for Module-Specific Data in HoshinoBot

> Learn how to implement database persistence for module-specific data in HoshinoBot using JSON files or the SqliteDao base class. Store your bot's data efficiently.

- Repository: [ice9coffee/hoshinobot](https://github.com/ice9coffee/hoshinobot)
- Tags: how-to-guide
- Published: 2026-03-03

---

**HoshinoBot provides two canonical patterns for module-specific persistence: lightweight JSON files stored in `~/.hoshino/` for simple dictionaries, and a reusable `SqliteDao` base class for structured relational data.**

When building modules for [ice9coffee/hoshinobot](https://github.com/ice9coffee/hoshinobot), you will need to store state that survives bot restarts. The framework offers distinct strategies depending on your data complexity. Simple key-value structures use atomic JSON files, while relational datasets leverage a SQLite abstraction layer that handles connection management and schema migration automatically.

## Understanding HoshinoBot's Persistence Architecture

The repository implements a clear separation between transient and durable storage. Module developers choose between two persistence tiers based on data structure and access patterns:

- **JSON File Persistence**: Ideal for small, dictionary-like data that fits entirely in memory. The bot loads the file at startup, manipulates a Python `dict`, and writes back to disk after mutations.
- **SQLite Persistence**: Designed for structured, relational data with complex queries or high volume. The `SqliteDao` base class in [`hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py) provides connection pooling, table creation, and CRUD wrappers.

Both approaches store files in the `~/.hoshino/` directory, ensuring user-specific data isolation and easy backup.

## Implementing JSON-Based Persistence

The JSON pattern follows a load-modify-dump lifecycle. The canonical implementation resides in [`hoshino/modules/priconne/arena/arena.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/modules/priconne/arena/arena.py), which stores user likes and dislikes for arena entries.

### File Location and Initialization

Define a constant pointing to the user-specific JSON file. The directory is created automatically if absent.

```python
import os

DB_PATH = os.path.expanduser("~/.hoshino/arena_db.json")
os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)

```

### Loading and Deserializing Data

At module import time, load the JSON into a global dictionary. Since JSON cannot natively store Python `set` objects, convert lists back to sets after loading.

```python
import json

try:
    with open(DB_PATH, encoding="utf8") as f:
        DB = json.load(f)
except FileNotFoundError:
    DB = {}

# Reconstruct sets from JSON lists

for k in DB:
    DB[k] = {
        "like": set(DB[k].get("like", [])),
        "dislike": set(DB[k].get("dislike", [])),
    }

```

### Modifying and Saving Data

Manipulate the in-memory dictionary directly. After every mutation, call a helper function that converts sets back to lists and atomically writes the file.

```python
def dump_db():
    j = {}
    for k in DB:
        j[k] = {
            "like": list(DB[k].get("like", [])),
            "dislike": list(DB[k].get("dislike", [])),
        }
    with open(DB_PATH, "w", encoding="utf8") as f:
        json.dump(j, f, ensure_ascii=False)

def add_like(entry_id: str, user_id: int):
    entry = DB.get(entry_id, {})
    likes = entry.get("like", set())
    dislikes = entry.get("dislike", set())
    
    likes.add(user_id)
    dislikes.discard(user_id)
    
    entry["like"] = likes
    entry["dislike"] = dislikes
    DB[entry_id] = entry
    
    dump_db()  # Persist immediately

```

## Implementing SQLite-Based Persistence

For relational data, HoshinoBot provides the `SqliteDao` abstraction in [`hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py). This pattern separates database schema definition from business logic.

### The SqliteDao Base Class

The base class handles connection management and table creation. It stores the database file at `~/.hoshino/clanbattle.db` and enables `datetime` parsing by default.

```python
import os
import sqlite3
import logging

logger = logging.getLogger(__name__)
DB_PATH = os.path.expanduser('~/.hoshino/clanbattle.db')

class SqliteDao:
    def __init__(self, table, columns, fields):
        os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
        self._dbpath = DB_PATH
        self._table = table
        self._columns = columns
        self._fields = fields
        self._create_table()
    
    def _create_table(self):
        with self._connect() as conn:
            conn.execute(f"CREATE TABLE IF NOT EXISTS {self._table} ({self._fields})")
    
    def _connect(self):
        conn = sqlite3.connect(self._dbpath)
        conn.row_factory = sqlite3.Row
        return conn

```

### Creating Concrete DAOs

Each data model inherits from `SqliteDao` and provides its own schema. The `ClanDao` example demonstrates primary key definitions and CRUD operations.

```python
class DatabaseError(Exception):
    pass

class ClanDao(SqliteDao):
    def __init__(self):
        super().__init__(
            table='clan',
            columns='gid, cid, name, server',
            fields='''
                gid INT NOT NULL,
                cid INT NOT NULL,
                name TEXT NOT NULL,
                server INT NOT NULL,
                PRIMARY KEY (gid, cid)
            '''
        )
    
    def add(self, clan):
        with self._connect() as conn:
            try:
                conn.execute(
                    f"INSERT INTO {self._table} ({self._columns}) VALUES (?,?,?,?)",
                    (clan['gid'], clan['cid'], clan['name'], clan['server'])
                )
            except sqlite3.DatabaseError as e:
                logger.error(f'[ClanDao.add] {e}')
                raise DatabaseError('添加公会失败')

```

The same pattern extends to `MemberDao` and `BattleDao` for managing clan battle records with foreign key relationships.

## Service-Level Configuration Persistence

Even core framework data follows the JSON pattern. The `Service` class in [`hoshino/service.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/service.py) persists group-specific configuration (e.g., which groups enable a service) using `_load_service_config()` and `_save_service_config()`.

These helpers read from and write to `~/.hoshino/service_config/<service_name>.json`, using `indent=2` for human-readable output. This demonstrates the canonical location for simple configuration storage outside of modules.

## Practical Implementation Examples

### Adding a New JSON-Backed Module

When creating a module that tracks user preferences or counters, follow the arena pattern:

```python

# mymodule/persistence.py

import os
import json
from hoshino import logger

DB_PATH = os.path.expanduser('~/.hoshino/mymodule_db.json')

# Initialize empty DB if file missing

try:
    with open(DB_PATH, encoding='utf8') as f:
        DB = json.load(f)
except FileNotFoundError:
    DB = {}
    os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)

logger.info(f'mymodule DB loaded from {DB_PATH}')

def _dump():
    """Atomic write of in-memory DB to disk."""
    with open(DB_PATH, 'w', encoding='utf8') as f:
        json.dump(DB, f, ensure_ascii=False, indent=2)

def increment_counter(user_id: str):
    DB[user_id] = DB.get(user_id, 0) + 1
    _dump()

```

### Adding a New SQLite-Backed DAO

For relational data requiring complex queries, extend the DAO framework:

```python

# mymodule/dao.py

import sqlite3
import logging
from hoshino.modules.pcrclanbattle.clanbattle.dao.sqlitedao import SqliteDao, DatabaseError

logger = logging.getLogger(__name__)

class InventoryDao(SqliteDao):
    """Manages user inventory items with quantities."""
    
    def __init__(self):
        super().__init__(
            table='inventory',
            columns='user_id, item_id, quantity',
            fields='''
                user_id INTEGER NOT NULL,
                item_id INTEGER NOT NULL,
                quantity INTEGER NOT NULL DEFAULT 0,
                PRIMARY KEY (user_id, item_id)
            '''
        )
    
    def add_item(self, user_id: int, item_id: int, amount: int = 1):
        with self._connect() as conn:
            try:
                conn.execute('''
                    INSERT INTO inventory (user_id, item_id, quantity)
                    VALUES (?, ?, ?)
                    ON CONFLICT(user_id, item_id) 
                    DO UPDATE SET quantity = quantity + ?
                ''', (user_id, item_id, amount, amount))
            except sqlite3.DatabaseError as e:
                logger.error(f'[InventoryDao.add_item] {e}')
                raise DatabaseError('Failed to add item')
    
    def get_inventory(self, user_id: int):
        with self._connect() as conn:
            cursor = conn.execute(
                'SELECT item_id, quantity FROM inventory WHERE user_id = ?',
                (user_id,)
            )
            return {row['item_id']: row['quantity'] for row in cursor}

```

### Integrating Persistence into Commands

Wire your DAO or JSON store into a Hoshino Service:

```python

# mymodule/__init__.py

from hoshino import Service
from .dao import InventoryDao

sv = Service('inventory')
dao = InventoryDao()

@sv.on_command('giveitem')
async def give_item(session):
    uid = session.ctx['user_id']
    args = session.state['args'].split()
    
    if len(args) < 1:
        await session.send('Usage: giveitem <item_id> [amount]')
        return
    
    item_id = int(args[0])
    amount = int(args[1]) if len(args) > 1 else 1
    
    try:
        dao.add_item(uid, item_id, amount)
        inv = dao.get_inventory(uid)
        await session.send(f'Added {amount} of item {item_id}. You now have {inv.get(item_id, 0)} total.')
    except Exception as e:
        await session.send(f'Error updating inventory: {e}')

```

## Key Files Reference

Understanding the canonical implementations requires examining these specific source files:

- **[`hoshino/modules/priconne/arena/arena.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/modules/priconne/arena/arena.py)** – Demonstrates JSON persistence for user preferences (likes/dislikes) with atomic write patterns.
- **[`hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py)** – Contains the `SqliteDao` base class and concrete implementations (`ClanDao`, `MemberDao`, `BattleDao`) showing relational data management.
- **[`hoshino/service.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/service.py)** – Implements `_load_service_config()` and `_save_service_config()` for service-level JSON configuration at `~/.hoshino/service_config/`.
- **[`run.py`](https://github.com/ice9coffee/hoshinobot/blob/main/run.py)** – Entry point that initializes the `~/.hoshino` directory structure on first startup.

## Summary

HoshinoBot offers two robust patterns for database persistence in module development:

- **JSON File Storage**: Use for simple, dictionary-like data that fits in memory. Store files in `~/.hoshino/<module>_db.json`, load at import time, and call a `dump()` helper after every mutation to ensure atomic writes.
- **SQLite Relational Storage**: Use for structured data requiring ACID compliance or complex queries. Inherit from `SqliteDao` in [`hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py), define your schema in the constructor, and use the `_connect()` context manager for all transactions.

Both approaches automatically create the `~/.hoshino` directory on first use, ensuring seamless deployment across different environments.

## Frequently Asked Questions

### What is the default location for HoshinoBot persistence files?

All module-specific data resides in the `~/.hoshino/` directory under the user's home folder. JSON databases typically use the pattern `~/.hoshino/<module>_db.json`, while the clan battle module stores its SQLite database at `~/.hoshino/clanbattle.db`. Service configurations live in `~/.hoshino/service_config/<service_name>.json`.

### When should I use SQLite instead of JSON for my module?

Choose **SQLite** when your data has relational structure (foreign keys, many-to-many relationships), requires atomic transactions across multiple tables, or will grow beyond what comfortably fits in memory. Use **JSON** for simple key-value stores, user preference flags, or counters where the entire dataset is small enough to load into a Python `dict` at startup.

### How does HoshinoBot handle concurrent writes to JSON files?

The framework uses a simple write-pattern where the in-memory dictionary is the source of truth. As shown in [`hoshino/modules/priconne/arena/arena.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/modules/priconne/arena/arena.py), the `dump_db()` function rewrites the entire file atomically using `json.dump()` after every mutation. While this does not protect against concurrent process access, it ensures that a single bot instance maintains consistency between memory and disk.

### Can I reuse the SqliteDao base class for my own modules?

Yes. The `SqliteDao` class in [`hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py`](https://github.com/ice9coffee/hoshinobot/blob/main/hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py) is designed for inheritance. Your concrete DAO must call `super().__init__()` with `table`, `columns`, and `fields` parameters defining your schema. The base class automatically creates the table if missing and provides the `_connect()` method that returns a `sqlite3.Connection` with `row_factory` set to `sqlite3.Row` for dict-like access.