From Reading Papers to Evaluating Startup Claims

This post builds on the first version of Gap Finder, introducing rule-based evaluations and source-verified quotations.

Overview

Deep-tech founders often point to scientific papers or patents as evidence that their technology is better than the alternatives. But unless you’re an expert in the field, it’s difficult to determine whether that evidence actually supports what the company is claiming today.

Gap Finder, which I built during the AI for Science Fellowship, addresses this gap. You provide the papers cited by the founders, the startup’s public claims, and relevant patents. Gap Finder identifies what’s missing from the research, evaluates whether each claim is supported by the evidence, and turns those gaps into questions to ask during founder meetings.

When I tested it on a solar startup, three findings stood out:

• Device mismatch: The stability data came from simpler research cells, rather than the commercial product.
• Testing mismatch: The durability claim involved testing three times longer and at higher temperatures than those reported in the paper.
• Patent ownership: A key patent was owned by a university, rather than the company.

General-purpose AI models are also good at reading scientific papers, but Claude, ChatGPT, and Gemini reached different conclusions when given the same questions. To address this, Gap Finder uses AI to answer factual questions, while code-based rules incorporating domain expertise determine the verdicts. Every quotation is checked against the original source.

In a small test involving three papers and 36 evaluation items, Gap Finder achieved approximately 94% agreement with expert assessments.

The next step is to have VCs test Gap Finder and refine its features based on their needs.

How It Works

Gap Finder separates reading from decision-making.

1. The LLM extracts the evidence

It examines papers, claims, and patents, answers factual questions such as “Is the measurement area reported?” or “Was this value measured in a tandem device?”, and quotes the exact sentences supporting its answers.

2. Rules determine the verdicts

Domain experts define 12 essential reporting items—a “spec sheet”—and evaluation rules in advance. Code applies these rules to the LLM’s answers to determine whether information is reported or missing, or whether claims are supported or unsupported.

3. Code verifies the quotations

Every sentence quoted by the LLM is checked against the original source to confirm that it actually exists.

This makes the rule-based judgments reproducible: the same extracted evidence leads to the same verdict, and every judgment comes with a traceable explanation.