Import System Architecture for Manual Data Imports in Maybe Finance

The Maybe finance application implements a layered, extensible import pipeline that processes CSV files through controllers, background jobs, domain models, and mapping systems to create transactions, trades, accounts, or Mint exports.

The open-source Maybe application (maybe-finance/maybe) provides a robust import system architecture for manual data imports that allows users to upload CSV files containing financial data. This architecture handles everything from file parsing and column mapping to background processing and domain object creation, supporting transaction imports, trade imports, account imports, and Mint CSV migrations.

Core Architecture Layers

The import system architecture for manual data imports follows a deliberate layered design that separates concerns between user interface, background processing, and data transformation.

Controller and UI Layer

The ImportsController in app/controllers/imports_controller.rb manages the wizard-style interface for manual data imports. It handles the upload step, column mapping configuration, confirmation screens, and background execution triggers. The controller stitches together the multi-step flow: new displays the import type selection, create persists a pending import record optionally scoped to an existing account, show redirects to the upload page if no CSV exists or to the mapping/confirmation page otherwise, publish triggers the background job, revert initiates rollback operations, and apply_template copies column-mapping configurations from previous successful imports of the same type.

Background Job Processing

Heavy-lifting import work runs asynchronously via ImportJob in app/jobs/import_job.rb, which is queued as high_priority. The job simply delegates to import.publish, which executes the entire import flow within a database transaction. This separation ensures the UI remains responsive while processing large CSV files and complex mapping operations.

Domain Model Layer

The Import class in app/models/import.rb serves as the abstract base orchestrator for all manual data imports. Concrete implementations include TransactionImport in app/models/transaction_import.rb, TradeImport in app/models/trade_import.rb, AccountImport in app/models/account_import.rb, and MintImport in app/models/mint_import.rb, each defining specific behavior for their domain. This layer coordinates CSV parsing, row generation, mapping synchronization, and the final publish operation that creates domain objects.

Row and Mapping Systems

The architecture includes specialized subsystems for data validation and transformation. Import::Row in app/models/import/row.rb represents individual CSV lines, handling validation, number sanitization, and signed amount calculation. The mapping system, centered on Import::Mapping in app/models/import/mapping.rb, bridges raw CSV values to existing domain objects through specialized subclasses: Import::CategoryMapping in app/models/import/category_mapping.rb, Import::TagMapping in app/models/import/tag_mapping.rb, Import::AccountMapping in app/models/import/account_mapping.rb, and Import::AccountTypeMapping in app/models/import/account_type_mapping.rb.

The Import Orchestration Model

The Import base class in app/models/import.rb provides the core machinery that all manual data import types leverage.

CSV Parsing and Row Generation

The parse_csv_str method uses Ruby's standard CSV library with configurable column separators (col_sep) to handle different regional formats. The generate_rows_from_csv method, overridden by each subclass, constructs a hash for every CSV line and bulk-inserts records via Import::Row.insert_all!. This bulk insertion approach minimizes database round-trips when processing thousands of rows.

Mapping Synchronization

The sync_mappings method iterates over mapping_steps defined by each subclass to build or update Import::Mapping records. This process ensures that each raw value from the CSV (such as "Groceries" or "Checking Account") maps to a concrete domain object like Category or Account. The mapping system supports both matching existing records and creating new ones on-the-fly.

Publish and Revert Flows

The publish method validates row counts, executes the subclass-specific import! method inside a database transaction, and updates the import status to complete or failed. Upon successful completion, it triggers a family-wide sync to update balances and calculations. The revert method provides transactional safety by deleting any accounts or entries created during the import and resetting the import status to pending, allowing users to correct errors and retry.

Concrete Import Types

The import system architecture for manual data imports supports four distinct import types, each tailored to specific financial data domains.

Transaction Imports

TransactionImport in app/models/transaction_import.rb creates Transaction and Entry records from CSV files containing personal finance transactions. It requires date and amount columns, with optional support for name, currency, category, tags, account, and notes. The mapping steps include CategoryMapping, TagMapping, and optional AccountMapping. This import type supports both signed amounts and custom column-based signage detection.

Trade Imports

TradeImport in app/models/trade_import.rb handles investment activity by creating Trade and Entry records. It requires date, ticker, qty, and price columns, with optional account mapping. This import type is essential for users tracking brokerage transactions and investment portfolios, allowing bulk entry of buy and sell orders with proper cost basis calculations.

Account Imports

AccountImport in app/models/account_import.rb creates new Account records directly from CSV data, utilizing OpeningBalanceManager to establish initial balances. It requires name and amount columns, with AccountTypeMapping to determine the accountable class (such as Depository or CreditCard). This import type is useful for migrating account lists from other financial systems or setting up initial account structures.

Mint Imports

MintImport in app/models/mint_import.rb provides specialized handling for CSV exports from Mint.com, creating Transaction records while preserving Mint-specific categorization and tagging. It follows similar patterns to TransactionImport but includes defaults optimized for Mint's export format, such as specific signage conventions and column expectations.

Row Processing and Validation

The Import::Row class in app/models/import/row.rb serves as the data validation and transformation layer for individual CSV lines.

Date Parsing and Amount Calculation

Each row stores raw CSV values and provides computed properties for downstream processing. The date_iso method parses the family-specific date_format into ISO-8601 format, handling regional date variations. The signed_amount method applies the import's signage convention—either inflows_positive or inflows_negative—and respects the amount-type strategy, which can be signed_amount or custom_column for complex CSV structures.

The row model also handles tag parsing through tags_list, splitting pipe-delimited strings (such as "groceries|essentials") into arrays for association with transaction records.

The Mapping System

The mapping system bridges the gap between raw CSV strings and typed domain objects, centered on the Import::Mapping base class in app/models/import/mapping.rb.

Category, Tag, and Account Mappings

Specialized mapping subclasses handle different domain associations:

• CategoryMapping (app/models/import/category_mapping.rb): Matches CSV category strings to existing Category records, creates a new category when the user selects "Add as new category".

• TagMapping (app/models/import/tag_mapping.rb): Similar to category mapping but does not require explicit selection because tags can be freely created.

• AccountMapping (app/models/import/account_mapping.rb): Links CSV account strings to existing Account objects; if the import isn't scoped to a specific family account, users can create a new account on-the-fly.

• AccountTypeMapping (app/models/import/account_type_mapping.rb): Used only by AccountImport to map the "entity_type" column to an accountable class (e.g., Depository, CreditCard).

Each mapping class implements mappables_by_key(import) for fast lookup, selectable_values for UI dropdowns, requires_selection? to determine mandatory selection, and create_mappable! to instantiate new domain objects.

Practical Implementation Examples

Creating a Manual Transaction Import

To create a manual transaction import programmatically:

family = Family.find_by(name: "Acme Corp")
import = family.imports.create!(
  type: "TransactionImport",
  account: nil,                     # nil => user will map accounts from CSV

  date_format: "%m/%d/%Y"
)

# Upload CSV content (as a string)

csv = <<~CSV
  date,amount,name,currency,category,tags,account,notes
  05/15/2024,-45.99,Grocery Store,USD,Food,groceries|essentials,Checking Account,Weekly groceries
CSV

import.update!(raw_file_str: csv)               # raw_file_str stores the raw CSV

import.generate_rows_from_csv                  # parses CSV → Import::Row records

import.sync_mappings                           # creates/updates Import::Mapping rows

# Preview mapping UI (in Rails console we can inspect)

import.mappings.accounts.pluck(:key, :mappable_id) # => [["Checking Account", 12]]

import.mappings.categories.pluck(:key, :mappable_id) # => [["Food", 3]]

# Once mappings are set (or left as "add new"), publish the import

import.publish_later   # enqueues ImportJob

Customizing Signage and Amount-Type Strategy

Configure how the system interprets amount signs:

import = family.imports.find(import_id)

# For inflows to be positive (default for Mint imports)

import.update!(signage_convention: "inflows_positive", amount_type_strategy: "signed_amount")

# Or use a custom column to decide inflow/outflow (e.g., a column called "Transaction Type")

import.update!(
  amount_type_strategy: "custom_column",
  amount_type_inflow_value: "credit"   # rows where entity_type == "credit" become positive

)

Adding New Accounts via Mapping

When the import UI shows the Account dropdown, selecting "Add as new account" creates a fresh Account record during import!:

# The mapping row for a new account will have key = "New Savings"

# and CREATE_NEW_KEY = "internal_new_resource"

# During import!:

Import::AccountMapping#create_mappable!  # creates the Account with default balance 0

Summary

The import system architecture for manual data imports in Maybe provides a comprehensive, modular pipeline for CSV data ingestion:

• Layered architecture separates UI concerns (ImportsController), background processing (ImportJob), and domain logic (Import base class and subclasses).
• Four concrete import types handle specific domains: TransactionImport, TradeImport, AccountImport, and MintImport.
• Row-level processing via Import::Row validates data, parses dates according to family-specific formats, and calculates signed amounts based on configurable signage conventions.
• Dynamic mapping system bridges CSV strings to domain objects through CategoryMapping, TagMapping, AccountMapping, and AccountTypeMapping, supporting both existing record matching and on-the-fly creation.
• Transactional safety ensures imports either complete successfully or can be fully reverted via the revert method, which deletes created records and resets import status.

Frequently Asked Questions

How does the Maybe import system handle different CSV formats and date conventions?

The import system architecture for manual data imports accommodates regional variations through configurable parsing options. The Import model stores date_format strings (such as %m/%d/%Y) and col_sep values for different column separators. During processing, Import::Row#date_iso parses dates according to the family-specific format, while Import.parse_csv_str uses Ruby's standard CSV library with the configured separator to handle semicolon or tab-delimited files.

What happens if a CSV contains categories or accounts that don't exist in the Maybe database?

The mapping system automatically handles unknown values through the Import::Mapping subclasses. When sync_mappings runs, it identifies unique keys from the CSV and creates mapping records. For categories and accounts, the UI presents dropdowns with existing values plus an "Add as new" option. If selected, CategoryMapping#create_mappable! or AccountMapping#create_mappable! instantiates new domain objects during the import! phase. Tags work similarly but don't require explicit selection, allowing free-form creation via TagMapping.

Can the import system handle both positive and negative amount conventions?

Yes, the architecture supports configurable signage conventions through the signage_convention attribute, which accepts inflows_positive or inflows_negative values. Additionally, the amount_type_strategy setting allows either signed_amount (using the numeric sign directly) or custom_column (where a separate column like "Transaction Type" determines signage). The Import::Row#signed_amount method applies these rules during processing, ensuring that inflows and outflows are recorded correctly regardless of the source CSV's conventions.

How does Maybe ensure data integrity during large CSV imports?

Data integrity is maintained through a combination of transactional boundaries and reversible operations. The publish method in app/models/import.rb wraps the subclass-specific import! logic in a database transaction, ensuring that either all rows import successfully or none do. For error recovery, the revert method deletes any accounts or entries created during the import and resets the import status to pending. Additionally, Import::Row validates required columns (such as date and amount for transactions) before the publish phase, preventing malformed data from reaching the domain models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}