Can evaluator.py Detect Plagiarism or Code Similarity?

No, the evaluator.py module in the interviewstreet/hiring-agent repository cannot detect plagiarism or code similarity; it exclusively evaluates candidate résumés using large language model (LLM) analysis.

The evaluator.py module serves as the core résumé screening engine for the hiring-agent project. Despite its general naming, this file reveals a narrow scope focused on parsing résumé text through LLM prompts to generate structured feedback, with no capabilities for analyzing source code or computing similarity metrics.

What evaluator.py Actually Does

Inside evaluator.py, the ResumeEvaluator class orchestrates a specific workflow designed for recruitment automation. The class builds evaluation prompts from résumé text, communicates with an LLM provider, and structures the response into a typed Python object.

The evaluation flow follows this sequence:

1. Prompt Construction: The class uses a template manager (located in prompts/template_manager.py) to render evaluation criteria from templates/resume_evaluation_criteria.
2. LLM Invocation: It calls self.provider.chat to send the constructed prompt to the configured language model.
3. Response Parsing: The raw LLM output passes through extract_json_from_response (defined in llm_utils.py) to extract structured data.
4. Data Validation: The extracted data populates an EvaluationData Pydantic model (defined in models.py), which includes fields for scores and qualitative comments.

Architecture of the Resume Evaluation Pipeline

Understanding why plagiarism detection is absent requires examining the component architecture.

ResumeEvaluator Class

The ResumeEvaluator class acts as the primary interface. When instantiated, it initializes an LLM provider through llm_utils.initialize_llm_provider. The class exposes an evaluate_resume method that accepts résumé text as a string and returns an EvaluationData instance. At no point does this class accept source code files or implement comparison algorithms.

LLM Provider Abstraction

The provider system (accessed via llm_utils.py) abstracts underlying APIs such as Gemini or OpenAI. The provider's .chat method processes natural language prompts only. It performs no intrinsic code analysis, tokenization for similarity, or fingerprinting operations.

Template Management System

Prompt templates stored in prompts/template_manager.py and configuration in prompt.py define static evaluation criteria. These criteria assess résumé quality factors like formatting, achievements, and role relevance—not code originality or syntactic similarity between programming solutions.

EvaluationData Schema

The EvaluationData model in models.py strictly defines expected output fields such as score and comments. The schema contains no plagiarism metrics, similarity scores, or reference corpus identifiers.

Why Plagiarism Detection Is Not Implemented

Detecting code plagiarism requires specific technical components that are entirely absent from the current codebase:

• Reference Corpus: A database of existing solutions to compare against.
• Similarity Algorithms: Implementation of metrics such as Levenshtein distance, token overlap, or abstract syntax tree (AST) comparison.
• Source Code Parsing: Logic to extract and normalize code from submission files.

The interviewstreet/hiring-agent repository contains none of these elements. The module never receives file paths to code submissions, nor does it compute distance metrics between text samples.

How to Use ResumeEvaluator

The following example demonstrates the intended usage for résumé evaluation:

from evaluator import ResumeEvaluator

# Initialize the evaluator (defaults to the configured model)

evaluator = ResumeEvaluator()

# Sample résumé text

resume_text = """
John Doe
Software Engineer
Experience: 5 years at XYZ Corp...
"""

# Run the evaluation – returns a typed EvaluationData object

evaluation = evaluator.evaluate_resume(resume_text)

print("Score:", evaluation.score)
print("Comments:", evaluation.comments)

This snippet illustrates the module's exclusive focus on text-based résumé analysis. Attempting to pass source code through this pipeline would simply result in the LLM evaluating the code as if it were biographical text, without any similarity comparison to existing solutions.

Summary

• evaluator.py processes only résumé text, not source code.
• The ResumeEvaluator class relies on LLM prompts for qualitative assessment, not algorithmic similarity detection.
• No plagiarism detection infrastructure exists in llm_utils.py, models.py, or template files.
• Adding plagiarism detection would require a separate pipeline with corpus management and algorithmic comparison tools.

Frequently Asked Questions

Does evaluator.py support code similarity checks?

No, evaluator.py does not support code similarity checks. The module is hardcoded to evaluate résumé content through LLM analysis. It lacks the algorithms, reference databases, and file parsing logic required for plagiarism detection.

What file types does evaluator.py process?

The module processes plain text strings representing résumé content. According to the source code in evaluator.py, the evaluate_resume method accepts text input directly and passes it to the LLM provider without file extension checks or syntax parsing.

How does evaluator.py validate résumé content?

Validation occurs through the EvaluationData Pydantic model in models.py. The extract_json_from_response function in llm_utils.py parses the LLM's JSON output, and the model validates field types and constraints. This ensures structured scoring data, not code originality verification.

Can I modify evaluator.py to detect plagiarism?

While technically possible, modifying evaluator.py for plagiarism detection would require significant architectural changes. You would need to integrate a code corpus database, implement similarity algorithms (such as AST matching or token-based fingerprinting), and modify the input pipeline to accept source code rather than résumé text. This would effectively create a separate module beyond the current résumé-focused scope.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}