🧠 Plugging Schema Graph into Multi-Table QA

A Human-Guided Framework for Reducing LLM Reliance

This repository contains the source code, datasets, and evaluation scripts for our paper on multi-table question answering using graph-enhanced reasoning. Our method integrates human-curated schema graphs to assist large language models (LLMs), improving performance and reducing reliance on model memorization.

---

📁 Directory Structure

TableQA-Graph/
├── data/                        # Datasets, prompts, and SQL resources
│   ├── dev/                    # Financial dataset & testing scripts
│   ├── train/                  # Olympics dataset & testing scripts
│   ├── outputs/               # Result logs and output files
│   ├── processed_data/        # Preprocessed CISS tables
│   ├── prompts/               # Prompt templates for LLMs
│   ├── datasetCISS.xlsx       # Raw CISS dataset
│   └── questions_and_comments_v2.txt

├── notebooks/                  # Table schemas and notes
├── docs/                       # Paper and figures (optional)
├── src/                        # Core logic
│   ├── data/                  # Data loading and processing
│   └── llm/                   # LLM querying and reasoning modules

├── tests/                      # Evaluation scripts on CISS
├── find.py                    # Global task orchestration
├── find_nocode_question.py   # Non-SQL QA generation
├── tasks.py                   # End-to-end main pipeline

├── requirements.txt           # Runtime dependencies
├── requirements_dev.txt       # Dev dependencies
├── pyproject.toml             # Python build and packaging
├── README.md

---

🧠 Key Modules

📦 src/llm/ – Model & Reasoning Engine

• llm_loader.py
  Unified interface for GPT-4o, Qwen, LLaMA, etc. Enables .query() abstraction.

• prompt_manager.py
  Loads, formats, and manages prompt templates for LLM input.

• question_decompose.py
  Decomposes complex questions into constraints + target for multi-hop logic.

• TableRAG_keywords.py
  Keyword and embedding-based attribute-level retrieval for table QA.

• TableRAG_keywords_decompQuestion.py
  Enhanced version that handles decomposition and retrieval in stages.

• variable_Graph.py
  Builds and queries schema-level graphs across multi-table environments to guide information extraction.

---

🚀 Model Inference Setup

We support both API-based and local inference:

• 🔹 GPT-4o (via OpenAI API)
• 🔹 LLaMA-3 70B (local on A100 GPUs)
• 🔹 Embedding: text-embedding-3-small (OpenAI)

---

🧪 Running Experiments

You can evaluate the system on different datasets using SLURM scripts:

# CISS dataset (real-world crash investigation)
sbatch tests/TableRAG_test.sh

# Olympics dataset (text-to-SQL benchmark)
sbatch data/train/Olympics_test.sh

# Financial dataset (text-to-SQL benchmark)
sbatch data/dev/financial_test.sh

---

📊 Evaluation

To summarize hop-type accuracy and correctness:

python tests/results_summary.py

---

⚙️ Requirements

We recommend using conda or virtualenv.

# Basic dependencies
pip install -r requirements.txt

# Developer tools and notebook support
pip install -r requirements_dev.txt

📌 Key Dependencies

• transformers >= 4.37
• openai >= 1.10
• pandas, scikit-learn, numpy
• torch >= 2.0
• sentence-transformers