How CreativeMath Generates Novel Solutions with LLMs: A Technical Deep Dive

CreativeMath generates novel solutions with LLMs by orchestrating a three-stage pipeline that progressively exposes reference solutions to force distinct reasoning paths, utilizing dynamic prompt construction, model abstraction, and iterative generation management.

CreativeMath is an open-source framework hosted at junyiye/creativemath that demonstrates how to generate novel solutions with LLMs for mathematical problems. The system implements a sophisticated prompting strategy that iteratively exposes an increasing number of reference solutions, compelling the model to produce mathematically distinct alternatives rather than variations of existing reasoning.

The Three-Stage Architecture for Novel Solution Generation

CreativeMath generates novel solutions with LLMs through a modular architecture comprising prompt engineering, model abstraction, and pipeline orchestration.

1. Prompt Construction with Progressive Context

The function load_novel_solution_generation_prompt in src/prompts/prompts.py constructs dynamic prompts that force novelty by controlling information exposure.

• Concatenation of references: The function receives a problem statement, the full list of reference solutions, and a counter k. It concatenates the first k reference solutions into a formatted block.
• Novelty criteria injection: The prompt appends a detailed description defining what constitutes a distinct solution—different reasoning paths, intermediate steps, underlying assumptions, generality, or complexity.
• Single-string output: The final output is a unified prompt string ready for LLM consumption.

By incrementing k across iterations, the system progressively expands the "forbidden" solution space, compelling the LLM to explore increasingly distant regions of the reasoning landscape.

2. Model Abstraction via ModelWrapper

The ModelWrapper class in src/models/model_loader.py abstracts away backend complexity, enabling seamless switching between API-based and locally-hosted models.

• Backend detection: During initialization, the wrapper checks the model name against a whitelist to determine whether to use load_api_model for commercial providers (Claude, Gemini, GPT-4) or load_local_model for locally-hosted checkpoints.
• Dynamic loading: It instantiates the appropriate backend based on the detection result.
• Unified interface: The generate_response(prompt) method standardizes interaction across backends. It wraps the prompt using load_messages into the format required by the specific backend, then dispatches to generate_api_response or generate_local_response.

This abstraction ensures that the generation pipeline remains agnostic to whether it is calling OpenAI's API or running inference on a local DeepSeek checkpoint.

3. The Generation Pipeline Orchestration

The script src/generation.py implements the orchestration logic that drives the iterative novelty generation process.

• Dataset parsing: It loads mathematical problems and their existing solution sets using load_json.
• Iterative exposure: For each problem, it iterates k from 1 to n (the total number of reference solutions).
• Prompt generation: For each k, it calls load_novel_solution_generation_prompt to create a tailored prompt exposing exactly k references.
• Model invocation: The prompt is passed to a ModelWrapper instance via model.generate_response(prompt).
• Metadata tracking: Each generated response is collected with metadata (problem_id, k, n) and persisted to JSON via save_json.

By systematically iterating k from 1 to n, the system guarantees that each generated solution is novel relative to an expanding corpus of known approaches, preventing the model from converging on a single reasoning pattern.

Step-by-Step Implementation: Generating Novel Solutions with CreativeMath

You can integrate CreativeMath's approach to generate novel solutions with LLMs using either the Python API for single-problem experimentation or the command-line interface for batch processing.

Python API Example

The following script demonstrates how to generate a novel solution for a single problem by directly invoking the prompt builder and model wrapper:

from prompts import load_novel_solution_generation_prompt
from models import ModelWrapper

# Example data

problem = "Prove that the sum of the first n odd numbers equals n²."
reference_solutions = [
    "Use induction on n.",
    "Observe that the k‑th odd number is 2k‑1 and sum the arithmetic series."
]

# Ask the model to generate a novel solution after seeing the first reference

k = 1  # expose only the first reference solution

prompt = load_novel_solution_generation_prompt(problem, reference_solutions, k)

model = ModelWrapper("gpt-4o")          # can be any supported model name

novel_solution = model.generate_response(prompt)

print("Novel solution:\n", novel_solution)

Batch Generation via CLI

To process an entire dataset and generate novel solutions across all reference counts, use the generation module directly:

python -m src.generation --model_name deepseek-math-7b-rl

This command reads the dataset specified in config.json, iterates through all problems while incrementing k from 1 to n, and writes results to <model_name>.json in the directory defined by config["file_paths"]["generation"].

Core Components and File Structure

Understanding the repository structure helps navigate the implementation when extending or debugging the system:

• src/prompts/prompts.py – Implements load_novel_solution_generation_prompt, which constructs dynamic prompts incorporating the first k reference solutions and explicit novelty criteria.
• src/models/model_loader.py – Defines ModelWrapper, providing a unified interface for both API-based models (Claude, Gemini, GPT-4) and local checkpoints through backend abstraction.
• src/generation.py – The main orchestration script that implements the iterative k-based generation loop, handling dataset I/O and metadata tracking.
• src/config.py – Manages configuration loading, including file paths and model settings referenced by the generation pipeline.
• src/utils.py – Provides utility functions for JSON I/O operations (load_json, save_json) used throughout the system.

Summary

CreativeMath demonstrates a robust methodology to generate novel solutions with LLMs through controlled exposure to existing knowledge:

• Progressive Prompting: The system uses load_novel_solution_generation_prompt to incrementally expose k reference solutions, explicitly defining novelty criteria to guide the LLM toward distinct reasoning paths.
• Backend Agnosticism: The ModelWrapper class abstracts API and local model implementations, enabling seamless switching between providers like GPT-4 and local DeepSeek checkpoints without pipeline modifications.
• Iterative Orchestration: The generation.py pipeline systematically iterates k from 1 to n, ensuring each generated solution is novel relative to an expanding corpus of known approaches.
• Structured Output: All generations include comprehensive metadata (problem_id, k, n) and persist to JSON for downstream analysis and validation.

Frequently Asked Questions

How does CreativeMath ensure that generated solutions are actually novel rather than paraphrased versions of references?

CreativeMath enforces novelty through the load_novel_solution_generation_prompt function in src/prompts/prompts.py, which appends explicit novelty criteria to the prompt. These criteria define distinctiveness as different reasoning paths, intermediate steps, underlying assumptions, generality, or complexity—not merely syntactic variation. By progressively increasing the number of exposed references via the k parameter, the system compels the LLM to explore increasingly distant regions of the solution space.

Can CreativeMath work with local models, or is it limited to commercial APIs?

CreativeMath supports both local and API-based models through the ModelWrapper class in src/models/model_loader.py. During initialization, the wrapper checks the model name against a whitelist to determine whether to instantiate an API backend for commercial providers (Claude, Gemini, GPT-4) or a local backend for locally-hosted checkpoints. The unified generate_response interface ensures the rest of the pipeline remains agnostic to the underlying infrastructure.

What is the purpose of iterating k from 1 to n in the generation pipeline?

The iteration of k from 1 to n in src/generation.py implements a progressive disclosure strategy that forces continual novelty. When k=1, the model sees only one reference and must produce a second distinct solution. When k=2, the model sees two references and must produce a third, and so forth. This expanding context ensures that each generated solution is novel relative to an increasingly comprehensive set of known approaches, preventing convergence on repetitive reasoning patterns.

How do I run the generation pipeline on my own dataset?

To execute the generation pipeline on custom data, configure your dataset path and model settings in config.json, then invoke python -m src.generation --model_name <your_model>. The script src/generation.py loads problems via load_json from the path specified in the configuration, iterates through all k values for each problem, and writes results to <model_name>.json in the directory defined by config["file_paths"]["generation"]. Ensure your dataset follows the expected schema containing problem_id, problem statements, and reference solution lists.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}