CSV File Support

NeuroLink provides seamless CSV file support as a multimodal input type - attach CSV files directly to your AI prompts for data analysis, insights, and processing.

Overview

CSV support in NeuroLink works just like image support - it's a multimodal input that gets automatically processed and injected into your prompts. The system:

1. Auto-detects CSV files using FileDetector (magic bytes, MIME types, extensions, content heuristics)
2. Parses CSV data using a streaming parser for memory efficiency
3. Formats CSV content into LLM-optimized text (markdown/json)
4. Injects formatted CSV data into your prompt text
5. Works with ALL AI providers (not limited to vision models)

Delimiter auto-detection: the delimiter is detected from the content (comma, tab / .tsv, semicolon, or pipe) — so tab- and semicolon-separated files parse into the correct columns instead of collapsing into one. Comma remains the default on ambiguity, and parsing is RFC-4180 quote-aware (a delimiter inside a "…" quoted field, e.g. "Smith, John", does not split the field). The detected delimiter is reported in metadata.detectedDelimiter.

Quick Start

SDK Usage

import { NeuroLink } from "@juspay/neurolink";

const neurolink = new NeuroLink();

// Basic CSV analysis
const result = await neurolink.generate({
  input: {
    text: "What are the key trends in this sales data?",
    csvFiles: ["sales-2024.csv"],
  },
});

// Multiple CSV files
const comparison = await neurolink.generate({
  input: {
    text: "Compare Q1 vs Q2 performance and identify growth areas",
    csvFiles: ["q1-sales.csv", "q2-sales.csv"],
  },
});

// Auto-detect file types (mix CSV and images)
const multimodal = await neurolink.generate({
  input: {
    text: "Analyze this data and compare with the chart",
    files: ["data.csv", "chart.png"], // Auto-detects which is CSV vs image
  },
});

// Customize CSV processing
const custom = await neurolink.generate({
  input: {
    text: "Summarize the top 100 customers by revenue",
    csvFiles: ["customers.csv"],
  },
  csvOptions: {
    maxRows: 100, // Limit to first 100 rows
    formatStyle: "markdown", // Use markdown table format
    includeHeaders: true, // Include CSV headers
  },
});

CLI Usage

# Attach CSV files to your prompt
neurolink generate "Analyze this sales data" --csv sales.csv

# Multiple CSV files
neurolink generate "Compare these datasets" --csv q1.csv --csv q2.csv

# Auto-detect file types
neurolink generate "Analyze data and image" --file data.csv --file chart.png

# Customize CSV processing
neurolink generate "Summarize trends" \
  --csv large-dataset.csv \
  --csv-max-rows 500 \
  --csv-format json

# Stream mode also supports CSV
neurolink stream "Explain this data in detail" --csv data.csv

# Batch processing with CSV
echo "Summarize sales data" > prompts.txt
echo "Find top performers" >> prompts.txt
neurolink batch prompts.txt --csv sales.csv

API Reference

GenerateOptions

type GenerateOptions = {
  input: {
    text: string;
    images?: Array<Buffer | string>;
    csvFiles?: Array<Buffer | string>; // Explicit CSV files
    files?: Array<Buffer | string>; // Auto-detect file types
  };

  csvOptions?: {
    maxRows?: number; // Default: 1000
    formatStyle?: "raw" | "markdown" | "json"; // Default: "raw"
    includeHeaders?: boolean; // Default: true
    encoding?: string; // Override; else auto-detected (#362)
    sanitizeColumnNames?: boolean; // Default: false (#378)
    columnNameCase?: "camelCase" | "snake_case"; // Default: "snake_case"
    parseTimeoutMs?: number; // Default: 30s strings / 5min files (#379)
  };

  // ... other options
};

CSV Input Types

CSV files can be provided as:

• File paths: "./data.csv" or "/absolute/path/data.csv"
• URLs: "https://example.com/data.csv"
• Buffers: Buffer.from("name,age\nAlice,30")
• Data URIs: "data:text/csv;base64,..."

// File path
await neurolink.generate({
  input: {
    text: "Analyze this",
    csvFiles: ["./data.csv"],
  },
});

// URL
await neurolink.generate({
  input: {
    text: "Analyze this",
    csvFiles: ["https://example.com/data.csv"],
  },
});

// Buffer
const csvBuffer = Buffer.from("name,age\nAlice,30\nBob,25");
await neurolink.generate({
  input: {
    text: "Analyze this",
    csvFiles: [csvBuffer],
  },
});

CSV Processing Options

maxRows

Limit the number of rows processed (default: 1000). Useful for large datasets.

csvOptions: {
  maxRows: 100; // Only process first 100 rows
}

formatStyle

Control how CSV data is formatted for the LLM:

• raw (default, RECOMMENDED): Original CSV format with proper escaping

  • Best for large files and minimal token usage
  • Preserves original structure
  • Handles commas, quotes, newlines correctly
  • File size stays minimal (63KB stays 63KB, not 199KB)

• json: JSON array format

  • Best for structured data processing
  • Easy to parse programmatically
  • Higher token usage (can expand 3x for large files)

• markdown: Markdown table format

  • Best for small datasets (<100 rows)
  • More readable for humans
  • Takes most tokens

// Raw CSV (recommended for large files)
csvOptions: {
  formatStyle: "raw",
}
// Output: name,age\nAlice,30\nBob,25

// JSON array
csvOptions: {
  formatStyle: "json",
}
// Output: [{"name":"Alice","age":30},{"name":"Bob","age":25}]

// Markdown table
csvOptions: {
  formatStyle: "markdown",
}
// Output: | name | age |
//         | ---- | --- |
//         | Alice | 30 |

includeHeaders

Include CSV headers in output (default: true).

csvOptions: {
  includeHeaders: false; // Skip headers
}

encoding

Character-encoding override (#362). When omitted, the encoding is detected in this order (see decodeBuffer in src/lib/utils/textEncoding.ts):

1. BOM — authoritative when present (UTF-8/UTF-16 byte-order mark).
2. Pure-ASCII fast path — if every byte in the peeked content is < 0x80, it's reported as UTF-8 immediately (ASCII decodes identically to UTF-8), without ever invoking chardet.
3. chardet statistical detection — only reached for non-ASCII content with no BOM.
4. UTF-8 fallback — used if chardet can't identify anything.

So Windows-1252 / Latin-1 / UTF-16 files no longer decode as mojibake, while the common plain-ASCII case never pays the chardet cost. Accepts any label iconv-lite supports.

csvOptions: {
  encoding: "windows-1252", // force a specific encoding
}

CLI: --csv-encoding windows-1252. The detected (or overridden) encoding is reported on result.metadata.detectedEncoding (plus encodingConfidence).

Streaming detection limitation: for on-disk files, encoding is detected from the initial ~64 KiB only and then committed for the rest of the stream (so the parse-timeout guard can keep working against a single streaming pass). A file whose leading ~64 KiB is pure ASCII but which switches to a legacy encoding later on can therefore still be misdecoded as UTF-8. If your files may do this, pass encoding explicitly rather than relying on auto-detection.

sanitizeColumnNames / columnNameCase

Rewrite column headers into valid identifiers (#378). Opt-in — the default (false) preserves the raw header strings as object keys. columnNameCase selects "snake_case" (default) or "camelCase".

csvOptions: {
  formatStyle: "json",
  sanitizeColumnNames: true,
  columnNameCase: "snake_case",
}

Original	Sanitized (snake_case)
Price ($)	price
1st Place	col_1st_place
Name/Title	name_title

Original names are preserved: result.metadata.columnNameMapping lists each renamed { original, sanitized } pair (columns left unchanged by sanitization are omitted), and each renamed columnMetadata entry carries originalName. CLI: --csv-sanitize-names [--csv-name-case camelCase]. The same option is available on the RAG CSVLoader (sanitizeColumnNames).

parseTimeoutMs

Wall-clock cap for the streaming parse (#379). Defaults: 30s for in-memory strings, 5min for on-disk files. On timeout the parser returns the rows collected so far and sets result.metadata.parseTimedOut = true instead of hanging forever.

csvOptions: {
  parseTimeoutMs: 60000, // 60s cap
}

CLI: --csv-parse-timeout-ms 60000.

File Detection System

NeuroLink uses a multi-strategy detection system with confidence scores:

Detection Strategies (in priority order)

1. Magic Bytes (95% confidence)

   • Detects file type from binary headers
   • Works for images (PNG, JPEG, GIF, WebP)
   • PDFs and binary formats

2. MIME Type (85% confidence)

   • Uses HTTP Content-Type headers for URLs
   • Detects text/csv, image/*, etc.

3. Extension (70% confidence)

   • File extension-based detection
   • Supports: .csv, .tsv, .jpg, .png, etc.

4. Content Heuristics (75% confidence)

   • Analyzes file content patterns
   • Detects CSV by checking consistent comma-separated columns

The system stops at the first strategy with 80%+ confidence.

// Example: FileDetector workflow
// 1. Check magic bytes -> Not binary (0% confidence)
// 2. Check MIME type (if URL) -> text/csv (85% confidence) ✓ STOP
// Result: Detected as CSV with 85% confidence

How It Works

Internal Processing Flow

// When you call generate() with CSV files:
await neurolink.generate({
  input: {
    text: "Analyze this data",
    csvFiles: ["data.csv"],
  },
});

// Internal flow:
// 1. messageBuilder.ts detects csvFiles array
// 2. Calls FileDetector.detectAndProcess("data.csv")
// 3. FileDetector runs detection strategies
// 4. Loads file content (from path/URL/buffer)
// 5. Routes to CSVProcessor.process(buffer)
// 6. CSV parsed using streaming csv-parser library
// 7. Formatted to LLM-optimized text (raw/markdown/json)
// 8. Appends to prompt text:
//    "Analyze this data
//
//    ## CSV Data from "data.csv":
//    ```csv
//    name,age,city
//    Alice,30,New York
//    Bob,25,London
//    ```"
// 9. Sends to AI provider

Memory Efficiency

CSV files are parsed using streaming for memory efficiency:

// CSVProcessor uses Readable streams
Readable.from([csvString])
  .pipe(csvParser())
  .on("data", (row) => {
    if (count < maxRows) rows.push(row);
  });

Large CSV files are handled efficiently:

• Streaming parser: Processes line-by-line
• Row limit: Configurable maxRows (default: 1000)
• Memory bounded: Only holds limited rows in memory

Examples

Data Analysis

const result = await neurolink.generate({
  input: {
    text: `Analyze this customer data and provide:
    1. Total customers
    2. Average age
    3. Top 5 cities by customer count
    4. Any notable patterns or insights`,
    csvFiles: ["customers.csv"],
  },
});

Data Comparison

const result = await neurolink.generate({
  input: {
    text: "Compare Q1 vs Q2 sales data. What changed? Which products improved?",
    csvFiles: ["q1-sales.csv", "q2-sales.csv"],
  },
});

Data Cleaning

const result = await neurolink.generate({
  input: {
    text: `Review this data for:
    - Missing values
    - Duplicate entries
    - Data quality issues
    - Suggested corrections`,
    csvFiles: ["raw-data.csv"],
  },
  csvOptions: {
    maxRows: 100,
    formatStyle: "markdown",
  },
});

Schema Generation

const result = await neurolink.generate({
  input: {
    text: "Generate a JSON schema for this CSV data with appropriate types and constraints",
    csvFiles: ["sample-data.csv"],
  },
  csvOptions: {
    maxRows: 50,
    formatStyle: "json",
  },
});

Multimodal Analysis

const result = await neurolink.generate({
  input: {
    text: "Compare the sales chart with the actual CSV data. Do they match?",
    files: ["sales-chart.png", "sales-data.csv"],
  },
});

TypeScript Types

Only types are exposed from the package (not classes):

import type {
  FileType,
  FileInput,
  FileSource,
  FileDetectionResult,
  FileProcessingResult,
  CSVProcessorOptions,
  FileDetectorOptions,
  CSVContent,
} from "@juspay/neurolink";

// FileType union
type FileType = "csv" | "image" | "pdf" | "text" | "unknown";

// CSV processing options
type CSVProcessorOptions = {
  maxRows?: number;
  formatStyle?: "raw" | "markdown" | "json";
  includeHeaders?: boolean;
  encoding?: string; // #362 — override; else BOM/chardet auto-detect
  sanitizeColumnNames?: boolean; // #378 — opt-in identifier-safe headers
  columnNameCase?: "camelCase" | "snake_case"; // #378
  parseTimeoutMs?: number; // #379 — wall-clock parse cap
};

// File detector options
type FileDetectorOptions = {
  maxSize?: number;
  timeout?: number;
  allowedTypes?: FileType[];
};

Best Practices

1. Use Raw Format for Large Files

The raw format is recommended for large files and best token efficiency:

csvOptions: {
  formatStyle: "raw",
} // ✅ RECOMMENDED for large files

// Use json for smaller datasets or when you need structured parsing
csvOptions: {
  formatStyle: "json",
} // ✅ Good for small-medium files

2. Limit Rows for Large Files

For large datasets, limit rows to avoid token limits:

csvOptions: {
  maxRows: 500,
} // Process first 500 rows

3. Use Markdown for Small Datasets

For <100 rows, markdown tables are more readable:

csvOptions: {
  maxRows: 50,
  formatStyle: "markdown"
}

4. Provide Clear Instructions

Give the AI clear instructions about what to analyze:

input: {
  text: `Analyze this sales data and provide:
  1. Total revenue
  2. Top 5 products
  3. Revenue trend
  4. Recommendations`,
  csvFiles: ["sales.csv"],
}

5. Use Auto-Detection

Let FileDetector handle mixed file types:

files: ["data.csv", "chart.png", "report.pdf"]; // Auto-detects each type

Limitations

• Max file size: 10MB by default (configurable)
• Max rows: 1000 by default (configurable)
• Encoding: auto-detected via BOM + chardet (UTF-8 / UTF-16 / Windows-1252 / Latin-1 …), or forced with encoding (#362)
• Per-row size: a single row is capped at 10MB to bound memory; larger rows fail fast with a clear error (#371)
• Parse timeout: parsing is time-bounded (30s strings / 5min files); on timeout partial rows are returned with metadata.parseTimedOut (#379)
• Row shape: parsed rows are validated to be string-keyed objects with string values; a malformed row aborts the parse with [CSVProcessor] Invalid CSV row <n> (#384)
• Token limits: Large CSV files may exceed provider token limits
• Streaming: CSV content is parsed and formatted before sending (not streamed to LLM)

Error Handling

try {
  const result = await neurolink.generate({
    input: {
      text: "Analyze this",
      csvFiles: ["data.csv"],
    },
  });
} catch (error) {
  if (error.message.includes("File too large")) {
    // Handle file size error
  } else if (error.message.includes("not allowed")) {
    // Handle file type restriction
  } else if (error.message.includes("CSV")) {
    // Handle CSV parsing error. Failures now carry context, e.g.:
    //   [CSVProcessor] Failed to open CSV file (/path/x.csv): ENOENT ...
    //   [CSVProcessor] CSV parsing failed after 3 row(s) : <cause>
    //     | headerCount: 3 | lastRow columns: 3 | Expected RFC 4180 CSV ...
    // (only structural context — counts, never raw header or cell string
    // values, which for headerless input may literally be first-row data —
    // is included, so error messages can't leak CSV data (#1199))
    //   [CSVProcessor] Invalid CSV row 2: expected a string-keyed object ...
    // A bad/missing path is wrapped (not a raw ENOENT), and both the source
    // and parser streams are always destroyed on any error path.
  }
}

Related Features

• Office Documents: DOCX, PPTX, XLSX processing
• PDF Support: PDF document processing
• Image Support: Similar multimodal input for images
• File Detection: Auto-detect file types with confidence scores
• Memory Efficient: Streaming parser for large files
• Provider Agnostic: Works with all AI providers
• CLI Integration: Full CLI support with options

Summary

• CSV support is multimodal input (like images)
• Use csvFiles array or files array (auto-detect)
• Customize with csvOptions (maxRows, formatStyle, includeHeaders)
• Works with ALL providers (not just vision models)
• Memory efficient streaming parser
• CLI support with --csv, --file, --csv-max-rows, --csv-format
• Only types exposed from package (not classes)