How to Implement Database Persistence for Module-Specific Data in HoshinoBot

HoshinoBot provides two canonical patterns for module-specific persistence: lightweight JSON files stored in ~/.hoshino/ for simple dictionaries, and a reusable SqliteDao base class for structured relational data.

When building modules for ice9coffee/hoshinobot, you will need to store state that survives bot restarts. The framework offers distinct strategies depending on your data complexity. Simple key-value structures use atomic JSON files, while relational datasets leverage a SQLite abstraction layer that handles connection management and schema migration automatically.

Understanding HoshinoBot's Persistence Architecture

The repository implements a clear separation between transient and durable storage. Module developers choose between two persistence tiers based on data structure and access patterns:

• JSON File Persistence: Ideal for small, dictionary-like data that fits entirely in memory. The bot loads the file at startup, manipulates a Python dict, and writes back to disk after mutations.
• SQLite Persistence: Designed for structured, relational data with complex queries or high volume. The SqliteDao base class in hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py provides connection pooling, table creation, and CRUD wrappers.

Both approaches store files in the ~/.hoshino/ directory, ensuring user-specific data isolation and easy backup.

Implementing JSON-Based Persistence

The JSON pattern follows a load-modify-dump lifecycle. The canonical implementation resides in hoshino/modules/priconne/arena/arena.py, which stores user likes and dislikes for arena entries.

File Location and Initialization

Define a constant pointing to the user-specific JSON file. The directory is created automatically if absent.

import os

DB_PATH = os.path.expanduser("~/.hoshino/arena_db.json")
os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)

Loading and Deserializing Data

At module import time, load the JSON into a global dictionary. Since JSON cannot natively store Python set objects, convert lists back to sets after loading.

import json

try:
    with open(DB_PATH, encoding="utf8") as f:
        DB = json.load(f)
except FileNotFoundError:
    DB = {}

# Reconstruct sets from JSON lists

for k in DB:
    DB[k] = {
        "like": set(DB[k].get("like", [])),
        "dislike": set(DB[k].get("dislike", [])),
    }

Modifying and Saving Data

Manipulate the in-memory dictionary directly. After every mutation, call a helper function that converts sets back to lists and atomically writes the file.

def dump_db():
    j = {}
    for k in DB:
        j[k] = {
            "like": list(DB[k].get("like", [])),
            "dislike": list(DB[k].get("dislike", [])),
        }
    with open(DB_PATH, "w", encoding="utf8") as f:
        json.dump(j, f, ensure_ascii=False)

def add_like(entry_id: str, user_id: int):
    entry = DB.get(entry_id, {})
    likes = entry.get("like", set())
    dislikes = entry.get("dislike", set())
    
    likes.add(user_id)
    dislikes.discard(user_id)
    
    entry["like"] = likes
    entry["dislike"] = dislikes
    DB[entry_id] = entry
    
    dump_db()  # Persist immediately

Implementing SQLite-Based Persistence

For relational data, HoshinoBot provides the SqliteDao abstraction in hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py. This pattern separates database schema definition from business logic.

The SqliteDao Base Class

The base class handles connection management and table creation. It stores the database file at ~/.hoshino/clanbattle.db and enables datetime parsing by default.

import os
import sqlite3
import logging

logger = logging.getLogger(__name__)
DB_PATH = os.path.expanduser('~/.hoshino/clanbattle.db')

class SqliteDao:
    def __init__(self, table, columns, fields):
        os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
        self._dbpath = DB_PATH
        self._table = table
        self._columns = columns
        self._fields = fields
        self._create_table()
    
    def _create_table(self):
        with self._connect() as conn:
            conn.execute(f"CREATE TABLE IF NOT EXISTS {self._table} ({self._fields})")
    
    def _connect(self):
        conn = sqlite3.connect(self._dbpath)
        conn.row_factory = sqlite3.Row
        return conn

Creating Concrete DAOs

Each data model inherits from SqliteDao and provides its own schema. The ClanDao example demonstrates primary key definitions and CRUD operations.

class DatabaseError(Exception):
    pass

class ClanDao(SqliteDao):
    def __init__(self):
        super().__init__(
            table='clan',
            columns='gid, cid, name, server',
            fields='''
                gid INT NOT NULL,
                cid INT NOT NULL,
                name TEXT NOT NULL,
                server INT NOT NULL,
                PRIMARY KEY (gid, cid)
            '''
        )
    
    def add(self, clan):
        with self._connect() as conn:
            try:
                conn.execute(
                    f"INSERT INTO {self._table} ({self._columns}) VALUES (?,?,?,?)",
                    (clan['gid'], clan['cid'], clan['name'], clan['server'])
                )
            except sqlite3.DatabaseError as e:
                logger.error(f'[ClanDao.add] {e}')
                raise DatabaseError('添加公会失败')

The same pattern extends to MemberDao and BattleDao for managing clan battle records with foreign key relationships.

Service-Level Configuration Persistence

Even core framework data follows the JSON pattern. The Service class in hoshino/service.py persists group-specific configuration (e.g., which groups enable a service) using _load_service_config() and _save_service_config().

These helpers read from and write to ~/.hoshino/service_config/<service_name>.json, using indent=2 for human-readable output. This demonstrates the canonical location for simple configuration storage outside of modules.

Practical Implementation Examples

Adding a New JSON-Backed Module

When creating a module that tracks user preferences or counters, follow the arena pattern:

# mymodule/persistence.py

import os
import json
from hoshino import logger

DB_PATH = os.path.expanduser('~/.hoshino/mymodule_db.json')

# Initialize empty DB if file missing

try:
    with open(DB_PATH, encoding='utf8') as f:
        DB = json.load(f)
except FileNotFoundError:
    DB = {}
    os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)

logger.info(f'mymodule DB loaded from {DB_PATH}')

def _dump():
    """Atomic write of in-memory DB to disk."""
    with open(DB_PATH, 'w', encoding='utf8') as f:
        json.dump(DB, f, ensure_ascii=False, indent=2)

def increment_counter(user_id: str):
    DB[user_id] = DB.get(user_id, 0) + 1
    _dump()

Adding a New SQLite-Backed DAO

For relational data requiring complex queries, extend the DAO framework:

# mymodule/dao.py

import sqlite3
import logging
from hoshino.modules.pcrclanbattle.clanbattle.dao.sqlitedao import SqliteDao, DatabaseError

logger = logging.getLogger(__name__)

class InventoryDao(SqliteDao):
    """Manages user inventory items with quantities."""
    
    def __init__(self):
        super().__init__(
            table='inventory',
            columns='user_id, item_id, quantity',
            fields='''
                user_id INTEGER NOT NULL,
                item_id INTEGER NOT NULL,
                quantity INTEGER NOT NULL DEFAULT 0,
                PRIMARY KEY (user_id, item_id)
            '''
        )
    
    def add_item(self, user_id: int, item_id: int, amount: int = 1):
        with self._connect() as conn:
            try:
                conn.execute('''
                    INSERT INTO inventory (user_id, item_id, quantity)
                    VALUES (?, ?, ?)
                    ON CONFLICT(user_id, item_id) 
                    DO UPDATE SET quantity = quantity + ?
                ''', (user_id, item_id, amount, amount))
            except sqlite3.DatabaseError as e:
                logger.error(f'[InventoryDao.add_item] {e}')
                raise DatabaseError('Failed to add item')
    
    def get_inventory(self, user_id: int):
        with self._connect() as conn:
            cursor = conn.execute(
                'SELECT item_id, quantity FROM inventory WHERE user_id = ?',
                (user_id,)
            )
            return {row['item_id']: row['quantity'] for row in cursor}

Integrating Persistence into Commands

Wire your DAO or JSON store into a Hoshino Service:

# mymodule/__init__.py

from hoshino import Service
from .dao import InventoryDao

sv = Service('inventory')
dao = InventoryDao()

@sv.on_command('giveitem')
async def give_item(session):
    uid = session.ctx['user_id']
    args = session.state['args'].split()
    
    if len(args) < 1:
        await session.send('Usage: giveitem <item_id> [amount]')
        return
    
    item_id = int(args[0])
    amount = int(args[1]) if len(args) > 1 else 1
    
    try:
        dao.add_item(uid, item_id, amount)
        inv = dao.get_inventory(uid)
        await session.send(f'Added {amount} of item {item_id}. You now have {inv.get(item_id, 0)} total.')
    except Exception as e:
        await session.send(f'Error updating inventory: {e}')

Key Files Reference

Understanding the canonical implementations requires examining these specific source files:

• hoshino/modules/priconne/arena/arena.py – Demonstrates JSON persistence for user preferences (likes/dislikes) with atomic write patterns.
• hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py – Contains the SqliteDao base class and concrete implementations (ClanDao, MemberDao, BattleDao) showing relational data management.
• hoshino/service.py – Implements _load_service_config() and _save_service_config() for service-level JSON configuration at ~/.hoshino/service_config/.
• run.py – Entry point that initializes the ~/.hoshino directory structure on first startup.

Summary

HoshinoBot offers two robust patterns for database persistence in module development:

• JSON File Storage: Use for simple, dictionary-like data that fits in memory. Store files in ~/.hoshino/<module>_db.json, load at import time, and call a dump() helper after every mutation to ensure atomic writes.
• SQLite Relational Storage: Use for structured data requiring ACID compliance or complex queries. Inherit from SqliteDao in hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py, define your schema in the constructor, and use the _connect() context manager for all transactions.

Both approaches automatically create the ~/.hoshino directory on first use, ensuring seamless deployment across different environments.

Frequently Asked Questions

What is the default location for HoshinoBot persistence files?

All module-specific data resides in the ~/.hoshino/ directory under the user's home folder. JSON databases typically use the pattern ~/.hoshino/<module>_db.json, while the clan battle module stores its SQLite database at ~/.hoshino/clanbattle.db. Service configurations live in ~/.hoshino/service_config/<service_name>.json.

When should I use SQLite instead of JSON for my module?

Choose SQLite when your data has relational structure (foreign keys, many-to-many relationships), requires atomic transactions across multiple tables, or will grow beyond what comfortably fits in memory. Use JSON for simple key-value stores, user preference flags, or counters where the entire dataset is small enough to load into a Python dict at startup.

How does HoshinoBot handle concurrent writes to JSON files?

The framework uses a simple write-pattern where the in-memory dictionary is the source of truth. As shown in hoshino/modules/priconne/arena/arena.py, the dump_db() function rewrites the entire file atomically using json.dump() after every mutation. While this does not protect against concurrent process access, it ensures that a single bot instance maintains consistency between memory and disk.

Can I reuse the SqliteDao base class for my own modules?

Yes. The SqliteDao class in hoshino/modules/pcrclanbattle/clanbattle/dao/sqlitedao.py is designed for inheritance. Your concrete DAO must call super().__init__() with table, columns, and fields parameters defining your schema. The base class automatically creates the table if missing and provides the _connect() method that returns a sqlite3.Connection with row_factory set to sqlite3.Row for dict-like access.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}