README.md

ReviewAid uniquely combines full-text AI screening, customizable extraction, multi-provider LLM configuration, explicit confidence scoring, and local inference in one open-source tool.

⚡ Usage (Online)

1. Launch Online Streamlit hosted web app

2. Select Mode:

   • Full-text Paper Screener: Choose this mode to screen papers based on PICO (Population, Intervention, Comparison, Outcome) criteria.
   • Full-text Data Extractor: Choose this mode to extract specific fields (Author, Year, Conclusion, etc.) from research papers.

3. Workflow (Screener):

   • Enter your PICO criteria (Inclusion/Exclusion) in the input fields.
   • Upload your PDF papers (Batch upload supported).
   • Click "Screen Papers".
   • Monitor the "System Terminal" for real-time logs of extraction, API calls, and processing status.
   • View the "Screening Dashboard" for a pie chart of Included/Excluded/Maybe decisions.
   • Download results as CSV, XLSX, or DOCX.

4. Workflow (Extractor):

   • Enter the fields you want to extract (comma-separated).
   • Upload your PDF papers.
   • Click "Process Papers".
   • Monitor the "System Terminal" for logs.
   • View extracted data in the dashboard.
   • Download extracted data as CSV, XLSX, or DOCX.

5. Configuration:

   • For using API key, you can select the respective AI model in either Screener/Extractor.

⚡ Usage (run streamlit Locally)

1. Clone the repository

   git clone https://github.com/aurumz-rgb/ReviewAid.git
   cd ReviewAid
   

2. Create and activate a virtual environment (recommended)

   python -m venv venv
   source venv/bin/activate        # macOS / Linux
   venv\Scripts\activate           # Windows
   

3. Install dependencies

   pip install -r requirements.txt
   

4. Start the Streamlit application

   streamlit run app.py
   

5. Configure Ai model along with API key inside the UI

   • Select AI model as the provider
   • Use API key

Seamlessly switch between AI models in the default setup without an API key, and also connect external providers using API keys like:

• OpenAI
• Anthropic (Claude)
• DeepSeek
• Cohere
• Z.ai
• Ollama (local execution)

Privacy Note To protect user privacy, API keys are never stored at any time.

✅ Tested & Supported Models

The following models were successfully tested for the supported tasks:

• OpenAI - GPT-4o
• DeepSeek - deepseek-chat
• Cohere - command-a-03-2025
• Z.ai - GLM-4.6V-Flash, GLM-4.5V-Flash , GLM-4.7-Flash
• Anthropic - Claude-Sonnet-4-20250514
• Ollama (local) - Llama3

Default model: GLM-4.6V-Flash

⚙️ Configuration File

ReviewAid does not require a separate configuration file.

All API keys and AI model selections are securely entered directly within the Streamlit user interface during runtime.

No .env, YAML, or JSON configuration files are required.

ReviewAid supports local inference using Ollama, allowing you to run the application without any external API keys. This is ideal for users who prefer offline usage, enhanced privacy, or full local control.

✅ Prerequisites

Ensure the following are installed on your system:

• Python 3.12+

• Ollama (installed and running locally)

  • Download: https://ollama.com

• At least one supported Ollama model (e.g., llama3)

Pull a model (example):

ollama pull llama3

Verify Ollama is running:

ollama list

▶️ Running ReviewAid with Ollama

1. Clone the repository

   git clone https://github.com/aurumz-rgb/ReviewAid.git
   cd ReviewAid
   

2. Create and activate a virtual environment (recommended)

   python -m venv venv
   source venv/bin/activate        # macOS / Linux
   venv\Scripts\activate           # Windows
   

3. Install dependencies

   pip install -r requirements.txt
   

4. Start the Streamlit application

   streamlit run app.py
   

5. Configure Ollama inside the UI

   • Select Ollama (Local) as the provider
   • Choose a local model (e.g., llama3)
   • No API key is required

🔒 Privacy Advantage

When using Ollama:

• All inference runs entirely on your local machine
• No data is sent to external servers
• No API keys are required or stored

This makes Ollama the most privacy-preserving configuration supported by ReviewAid.

⚠️ Notes

• Performance depends on your local hardware (CPU/GPU/RAM)
• Large PDFs or batch sizes may take longer on CPU-only systems
• For best results, ensure Ollama is running before launching Streamlit

This system implements a hierarchical four-tier model designed to maximize precision and minimize false classifications during automated paper screening and data extraction. The logic prioritizes deterministic rule-based decisions before progressively falling back to algorithmic and heuristic estimation only when necessary.

🔄 How ReviewAid Works (v4.0.0)

End-to-end walkthrough of both engines - what happens at every step, what goes where, and why. The decision architecture is: Tier-1 deterministic gate → Tier-2 LLM screening → deterministic override. Humans stay in the loop throughout.

The Life of a Paper - Screener

1. Upload & deduplicate. Every PDF is hashed (SHA-256); an identical file reuses the cached decision from earlier in the batch - no repeated API calls. Why: same paper, same answer, zero cost.

2. Text extraction. PyMuPDF pulls the full text plus candidate title, author and year; the text is cleaned and token-capped before any AI call. Why: models see a bounded, clean input, and evidence quotes are checked against this exact text so they stay verifiable.

3. Tier-1 guarded gate - 0 API calls. Keyword scanning with the v4.0.0 guards (word-boundary matching, negation guard, background-context guard, corroboration for one-word criteria). Fires only on exclusion evidence that is unanimous across its own checks and grounded. Why: obvious exclusions cost nothing; weak keyword evidence refers instead of deciding - the v3.0.0 lesson.

4. Per-criterion LLM stage - 3 API calls. Each call sees every criterion plus the full text and returns, per criterion, a verdict (yes/no/unsure) plus a verbatim supporting quote. Quotes are verified against the text: an unfindable quote downgrades the judgment to unsure. Why: judging criteria one-by-one beats one holistic verdict, and quotes make every decision checkable in seconds.

5. Majority vote + agreement. The three samples are majority-voted per criterion; the agreement rate across samples becomes the paper's score. Why: three independent readings outvote single-sample errors and model refusals; agreement is a measured property instead of a self-reported guess.

6. Recall-first decision. Exclude only on exclusion evidence that is unanimous and grounded, with no inclusion criterion met. Include only when the driving inclusion criteria clear the agreement floor (0.67) with no exclusion met. Everything else - conflicts, shaky agreement, unusable samples - is a Maybe referral. Why: wrongly excluding a study is the cardinal sin; weak evidence must refer, never decide.

7. Tiebreaker adjudication. If the samples split on a criterion, one extra senior-reviewer call sees the competing votes and quotes and issues a grounded ruling. Why: contested papers deserve a second opinion, not a coin flip.

8. override (unchanged from v3). If confidence is high but the Tier-1 deterministic score is low, confidence is overridden downward and the paper is flagged. Why: the deterministic layer catches confident hallucination.

9. Decision routing. Papers land in the Include / Exclude / Maybe tables, each row carrying confidence, a priority score, the reason and the full per-criterion trail.

What Each of the 3 Screening Calls Is For

• Call #1, #2, #3 - the same prompt shape (all criteria + full text, temperature 0): three independent readings of the paper, each returning per-criterion verdicts with verbatim quotes. Majority vote per criterion; agreement across the three becomes the confidence.

• Call #4 (conditional tiebreaker) - runs only when the samples split on a criterion; it receives the split votes and competing quotes and issues a final ruling.

• 0 calls - papers the Tier-1 gate auto-excludes.

Why three calls: three independent readings outvote single-sample errors and model refusals (self-consistency, Wang et al. 2022). Voting gains are front-loaded in the first samples - which is why v4.0.0 standardises on three instead of one, or six.

The Life of a Paper - Extractor

1. Fields. The user lists the fields to extract; "Paper Title" is added automatically if missing. Why: extraction is user-defined - ReviewAid extracts what your review needs.

2. Prompt contract. Every field gets a description; the response is a single JSON object (extracted + confidence) with "Not Found" for missing data, at temperature 0.

3. Effect Direction contract. The direction must be one closed label (significantly increases / significantly decreases / no significant difference / unclear) and Effect Direction Evidence must quote the paper verbatim. Why: free-text directions scored at chance in validation; labels make it strictly scoreable and the evidence quote keeps it grounded.

4. Tier-1 verification. Each extracted field is checked against the source text: exact string match → token overlap for paraphrases → negation windows. Ungrounded fields drop the confidence score. Why: this is the hallucination guard - in validation, 95–99.6% of ~26,000 extracted fields were verifiably present in the source papers.

5. Reliability accounting. Every regex-fallback use is counted (parser.fallback_uses()) so a degraded model arm is visible instead of silent.

Where Results Go

• Include / Exclude / Maybe tables - every row carries confidence, priority score, reason and the criteria trail.
• Exports - each table downloads as DOCX, CSV or XLSX.
• System Terminal - per-call labels (Paper N [LLM Call #k]), screening stage, usable samples, agreement, and every Tier-1 discarded keyword hit.
• Priority queue - sort the export by priority and work the review queue highest-first; workload saved at a fixed recall is measurable from the export alone.

Providers & Privacy

• Default mode uses ReviewAid's own GLM keys - nothing to configure.
• Bring your own key for OpenAI, Anthropic, Cohere, DeepSeek or GLM.
• Ollama (local) runs fully offline - paper text never leaves your machine. The local context window defaults to 16,384 tokens (OLLAMA_NUM_CTX to tune) so long papers are never silently truncated.
• Paper text is sent only to the provider you select for that run.

Why It Is Built This Way

Recall-first (weak evidence refers, never decides) · grounding everywhere (quotes and fields must exist in the source text) · measured confidence (sample agreement, not self-report) · full auditability (every decision carries its evidence and every discarded hit is logged) · permanent human oversight. ReviewAid is a third reference for reviewers - not a replacement.

❓ Overview

The confidence score reflects how reliably a paper has been classified or extracted. Scores range from 0.0 to 1.0, where higher values indicate stronger certainty and lower values explicitly flag the need for manual review.

The system operates in the following order:

1. Deterministic Rule-Based Classification & Verification (Screener & Extractor)
2. Per-Criterion LLM Screening with Evidence Quotes (v4.0.0)
3. Heuristic Keyword Estimation
4. Low-Confidence Default

Text: Tier 1 (Deterministic) → Tier 2 (LLM + Override) → Tier 3 (Heuristic) → Tier 4 (Default).

Each tier is only activated if the previous tier fails to produce a valid and reliable result.

✔️ Tier 1: Deterministic Rule-Based Classification & Verification (Highest Priority)

Purpose: Eliminate ambiguity using explicit mathematical rules and verify AI outputs against the source text.

Screener Logic (v4.0.0 gate, tier1_gate.py):

• The system performs a preliminary scan for exclusion and inclusion keywords.

• An exclusion keyword only counts when its mention survives deterministic guards:

  • Word-boundary matching - "men" no longer matches inside "women", "rat" no longer matches "ratio".
  • Negation guard - "no acute LBP", "non-pregnant" or "free of ..." mentions are not population hits.
  • Background-context guard - mentions sitting in background/related-work discourse, far from any eligibility language, are ignored.

• A lone single-word exclusion criterion ("adults", "children", "pregnant") can never decide a paper on its own; it needs a second qualifying exclusion criterion behind it (corroboration). Multi-word phrases ("acute LBP", "cross-sectional studies") still decide alone.

• If qualifying exclusion keywords are detected without any corresponding inclusion keywords, the paper is:

  • Automatically classified as Excluded
  • Assigned a confidence score of 1.0 (100%)

• Everything else - including any paper where both exclusion and inclusion keywords are present - is deferred to the AI, exactly as before. Inclusion keywords are deliberately matched loosely (raw substring), because they only ever defer a paper, never exclude one.

• Every keyword mention a guard throws out is reported in the System Terminal (negated / background mention), so each auto-exclusion stays fully auditable after the fact.

Extractor Logic:

• The system deterministically verifies the AI's extracted data against the source text using Exact String Matching and Token Overlap for paraphrased text.

• Negation Detection is applied to ensure the AI didn't miss a "not" or "failed" that changes the meaning of the extracted data.

Rationale:
Explicit rules provide deterministic certainty and override probabilistic inference when applicable. Mathematical verification ensures the AI actually saw what it claimed to see. Since v4.0.0, the gate also refuses to decide on incidental keyword mentions: the architecture validation against human gold standards showed unguarded substring matching was the single largest error source, so borderline papers are deferred to the LLM tier instead.

✔️ Tier 2: Per-Criterion LLM Screening (With Override Logic)

Purpose: Leverage the model’s internal reasoning and evidence-based judgment.

Logic (v4.0.0 per-criterion pipeline, pico_screen.py):

• The LLM no longer emits one holistic include/exclude verdict with a self-reported confidence. It judges each criterion separately, returning a verdict (yes / no / unsure) and a verbatim supporting quote per criterion.

• Every quote is verified against the paper text; a quote that is not actually in the paper downgrades that judgment to unsure (grounding, applied to screening).

• Each paper is judged k = 3 independent samples and the samples are majority-voted per criterion. The sample agreement rate is the confidence - a measurable property of the judgments, not a self-assessment.

• Three API calls per screened paper, flat. The paper's full text is judged three independent times (temperature 0) and the samples are majority-voted per criterion. Three is the accuracy-per-cost sweet spot: majority voting gains are front-loaded in the first samples, and the vote absorbs single-sample errors and refusals.

• Include requires the driving inclusion criteria to clear the agreement floor (0.67) with no exclusion met; exclude fires only on exclusion evidence that is unanimous across all samples and grounded in a quote - weak evidence is referred, never decided. Everything else is a Maybe referral with the full per-criterion trail visible to the reviewer.

• Tiebreaker adjudication: when the samples split on a criterion, one senior-reviewer call settles it against the competing quotes; the ruling is grounded like every other judgment.

• Priority queue: every paper leaves with a priority score (inclusion strength, quote coverage, agreement) so the human review queue is worked highest-first - workload-saved at a fixed recall is measurable straight from the export.

• Override Logic (unchanged): if the agreement-based confidence is high but the Tier 1 Deterministic Check fails, the confidence is overridden downward and the paper is flagged for human review.

Rationale:
This tier captures nuanced contextual understanding that deterministic rules cannot, while the override logic ensures mathematical grounding prevents false confidence. Per-criterion judgments with quotes make every decision auditable criterion-by-criterion, and agreement-based confidence reflects how stable the model's reading actually is.

Effect-direction extraction (v4.0.0): the extractor accepts an Effect Direction field with a closed label set (significantly increases / significantly decreases / no significant difference / unclear) plus an Effect Direction Evidence field that must quote the paper verbatim. Strict label scoring becomes possible, and the label's confidence is verified through its evidence sentence rather than literal text matching.

✔️ Tier 3: Heuristic Keyword Estimation (Fallback)

Purpose: Provide a probabilistic estimate when LLM confidence is unavailable.

Triggered when:

• The LLM fails to return a valid confidence value (e.g., formatting or JSON parsing errors)

Screener Logic:

• The system analyzes the users input Inclusions and Exclusions criteria and matches with the paper's full-text and determines the confidence level.

Extractor Logic:

• The system analyzes Extracted data with the paper's full-text and determines the confidence level.

Rationale:
This tier offers a best-effort estimate derived from text structure rather than semantic certainty.

✔️ Tier 4: Low-Confidence Default (Last Resort)

Purpose: Explicitly flag unreliable outputs.

Triggered when:

• Data extraction fails entirely (e.g., Regex failure or missing sections)

Logic:

• Assigns a baseline low confidence score (e.g., 0.2)
• Automatically flags the result for mandatory manual review

Rationale:
Prevents silent failures by clearly signaling unreliability.

This layered approach ensures that high-confidence decisions are automated safely, while ambiguous or unreliable cases are clearly flagged for human oversight.

⭐ Confidence Score Interpretation

Purpose: Safely parse API/AI responses, even if the JSON is broken or missing.

Flow

1. If raw_result is None → Use regex to extract data locally.

2. Clean the response → Remove Markdown, comments, and trailing commas.

3. Try standard JSON parsing → json.loads (using strict=False to safely handle unescaped newlines without hanging).

4. If that fails, use AI repair → Ask AI to fix the JSON syntax.

5. Final fallback → Extract known keys using regex.

Guarantee

• Never crashes
• Always attempts to recover usable data

Purpose: Extract structured data from images using OCR before parsing.

The system uses Optical Character Recognition (OCR) to convert image content into structured text for downstream processing.

Initially, the project was built with PaddleOCR for higher accuracy. However, due to deployment limitations on Streamlit servers, it now uses pytesseractOCR, a Python wrapper for Google Tesseract-OCR.

For local deployment with higher accuracy, you can use the PaddleOCR version here: ReviewAid-OCR

OCR Workflow (Tesseract)

• Image is processed using the Tesseract OCR engine
• Text is extracted and cleaned for improved accuracy
• Output is structured and forwarded to the parsing pipeline
• Handles noisy, scanned, and complex images reliably

Official pytesseract repository: madmaze/pytesseract