describegpt command

describegpt allows users to infer extended metadata about a CSV dataset using large language models, in particular GPT chat completion models from OpenAI's API, Ollama, or an API compatible with the OpenAI API specification such as Jan.

describegpt uses qsv stats and qsv frequency in the background to provide context to the model. It does not send the entire dataset to the model. This "zero-copy" approach is not only ideal for reducing costs (as detailed metadata, even for very large dataset is "tiny" in comparison) and increasing performance, it also protects the privacy of sensitive data.

Note that this command uses LLMs for inferencing and is therefore prone to inaccurate information being produced. Verify output results before using them.

Basic Inference Options

describegpt provides several options to infer different types of metadata about your dataset:

• --dictionary - Creates a Data Dictionary using a hybrid "neuro-symbolic" pipeline. The Data Dictionary is deterministically populated using Summary Statistics and Frequency Distribution data, and only the human-friendly Label and Description (and the Content Type, when --infer-content-type is set) are populated by the LLM using the same statistical context.

• --description - Infers a general Description of the dataset based on detailed statistical context.

• --tags - Infers Tags that categorize the dataset based on detailed statistical context. Useful for grouping datasets and filtering.

• -A, --all - Shortcut for --dictionary --description --tags. Generates all three types of metadata in a single run.

Dictionary Options

When using the --dictionary option, the following option controls what the Data Dictionary includes:

--infer-content-type

By default, the Data Dictionary's LLM pass only infers a human-friendly Label and Description for each field. With --infer-content-type, the LLM additionally infers a semantic Content Type for each field — a single token describing what the field actually contains — and a "Content Type" column/field is added to the Data Dictionary output (Markdown, JSON, TSV, and TOON).

The Content Type is chosen from a fixed, curated vocabulary so the value stays stable and machine-mappable:

first_name, last_name, full_name, username, password, email, phone, street_address, building_number, secondary_address, city, state, state_abbr, zip_code, country, country_code, latitude, longitude, time_zone, company_name, job_title, uuid, credit_card, currency_code, isbn, ip_address, mac_address, url, user_agent, file_name, file_path, mime_type, color_hex, time, category, lorem_word, lorem_sentence, lorem_paragraph, free_text, unknown

Primitive types (integer, decimal, boolean, date, datetime) are intentionally not in the vocabulary — they are already covered by the Data Dictionary's deterministic Type column. Plain numeric, temporal, or boolean fields with no specific semantic meaning are classified as unknown, and any value the LLM returns outside the vocabulary is coerced to unknown.

When --infer-content-type is not set, the Data Dictionary output is unchanged.

qsv describegpt data.csv --dictionary --infer-content-type

The curated vocabulary is designed to map cleanly onto fake-data generators, so a Data Dictionary produced with --infer-content-type can later drive synthetic test-data generation.

Tag Options

When using the --tags option, you can control how tags are inferred:

--num-tags <n>

The maximum number of tags to infer when the --tags option is used. The value must be between 1 and 50. The default is 10.

--tag-vocab <file>

The CSV file containing the tag vocabulary to use for inferring tags. If no tag vocabulary file is provided, the model will use free-form tags.

The CSV file must have two columns with headers:

• First column: tag name
• Second column: tag description

Example CSV format:

tag,description
alphabetical_data,Data containing letters or alphabetical characters
numerical_data,Data containing numbers or numerical values
test_data,Sample or test data used for demonstration

The tag vocabulary supports multiple sources:

• Local files: Provide a path to a local CSV file
• Remote URLs: HTTP/HTTPS URLs pointing to CSV files (e.g., https://example.com/tags.csv)
• CKAN resources: Use the ckan:// scheme to reference CKAN dataset resources (e.g., ckan://dataset-id/resource-id)
• dathere:// scheme: Reference GitHub raw content using the dathere:// scheme

Remote resources (HTTP/HTTPS, CKAN, and dathere://) are automatically cached locally to avoid repeated downloads. The cache TTL is 1 hour by default.

Note: qsvlite only supports local files. Remote URLs, CKAN resources, and dathere:// schemes are not available in qsvlite.

--cache-dir <dir>

The directory to use for caching downloaded tag vocabulary resources. If the directory does not exist, qsv will attempt to create it. The default is ~/.qsv-cache.

If the QSV_CACHE_DIR environment variable is set, it will be used instead of this option.

--ckan-api <url>

The URL of the CKAN API to use for downloading tag vocabulary resources with the ckan:// scheme. The default is https://data.dathere.com/api/3/action.

If the QSV_CKAN_API environment variable is set, it will be used instead of this option.

--ckan-token <token>

The CKAN API token to use. Only required if downloading private CKAN resources. If the QSV_CKAN_TOKEN environment variable is set, it will be used instead of this option.

Examples

Using tag vocabulary with a local CSV file:

qsv describegpt data.csv --tags --tag-vocab tags.csv

Using tag vocabulary with a remote URL:

qsv describegpt data.csv --tags --tag-vocab https://example.com/tags.csv

Using tag vocabulary with CKAN resources:

qsv describegpt data.csv --tags --tag-vocab ckan://dataset-id/resource-id --ckan-api https://data.example.com/api/3/action

Using tag vocabulary with a custom cache directory:

qsv describegpt data.csv --tags --tag-vocab https://example.com/tags.csv --cache-dir /tmp/qsv-cache

Limiting the number of tags inferred:

qsv describegpt data.csv --tags --tag-vocab tags.csv --num-tags 5

QSV_LLM_APIKEY

When working with a cloud-based LLM, describegpt requires an API key. You can set this key using the QSV_LLM_APIKEY environment variable. Check /docs/ENVIRONMENT_VARIABLES.md for more info.

When working with a Local LLM (i.e. if --base-url or QSV_LLM_BASE_URL contains "localhost"), describegpt will NOT require an API key.

--api-key <key>

You can also specify your API key directly in your CLI using the --api-key option.

Note that the precedence is: CLI flag > environment variable > error. If you pass --api-key, it overrides the QSV_LLM_APIKEY environment variable. Pass --api-key NONE to suppress the API key entirely.

--format <format>

Use the --format option to set the output format. Supported values are Markdown (the default), TSV, JSON, TOON, JSONSchema, and SemanticMd.

• Markdown - human-readable Markdown (default).
• TSV - tab-separated values.
• JSON - JSON output, useful for piping to other commands.
• TOON - a compact, human-readable encoding of the JSON data model, optimized for LLM prompts.
• JSONSchema - emits the Data Dictionary as a JSON Schema (draft 2020-12). Requires the dictionary inference phase. See --allow-extra-cols and --strict-dates for JSONSchema-specific behavior.
• SemanticMd - emits the Data Dictionary as a Semantic Markdown document. Implies --infer-content-type and also requires the dictionary inference phase. See --ds-source, --ds-updated, and --ds-license.

Note that --format only controls the structure of the command's output. It does not instruct the LLM to generate a particular content shape - that is driven by your prompts.

--max-tokens <value>

--max-tokens is an option that allows you to specify the maximum number of tokens in the completion output. This is limited by the maximum number of tokens allowed by the model including the input tokens.

Input tokens may include the output of qsv stats and qsv frequency from your dataset, which can be large based on your dataset's size. Therefore we use gpt-oss-20b as the default model for describegpt as it has a maximum token limit of 131,072.

It is highly recommended to set the --max-tokens option to set the maximum number of tokens in the completion output. Your output may be truncated if you set this value too low or you may receive errors depending on your options. The default is set to 10000 as a safety measure.

When running a Local LLM (detected if the base_url contains localhost), the max token limit is automatically disabled. Your completions are only limited by the LLM model you're using.

--prompt-file

With describegpt you can use a prompt file to add your own custom prompts and as an alternative to specifying certain options through the CLI. You can use the --prompt-file option to specify a prompt file to use.

If you do not specify a prompt file, default prompts will be used.

Field	Description
name	The name of your prompt file.
description	A description of your prompt file.
author	Your name.
version	The version of your prompt file.
tokens	The maximum number of tokens in the completion output.
format	Output format: Markdown, TSV, JSON, TOON, JSONSchema, or SemanticMd (default markdown). The --format CLI flag takes precedence over this field.
system_prompt	Overall guidance prompt to the LLM.
dictionary_prompt	The prompt for the --dictionary option.
description_prompt	The prompt for the --description option.
tags_prompt	The prompt for the --tags option.
base_url	The URL of the LLM API. When it contains "localhost", automatically sets tokens to 0.
model	The LLM model to use.
timeout	The timeout in seconds to use when waiting for LLM prompt completions.
custom_prompt_guidance	The guidance used to generate SQL queries in SQL RAG mode.
duckdb_sql_guidance	DuckDB-specific SQL generation guidelines.
polars_sql_guidance	Polars-specific SQL generation guidelines.
dd_fewshot_examples	DuckDB "few-shot" examples. See https://www.promptingguide.ai/techniques/fewshot
p_fewshot_examples	Polars "few-shot" examples.

All fields must be present in your prompt file. If you do not want to use a certain prompt, you can set it to an empty string.

Within your prompts, you can use the following MiniJinja variables (use {{ variable }} syntax). These are replaced in the prompt sent to the LLM with their respective values at run-time:

• {{ stats }} - summary stats generated by qsv stats (CSV format)
• {{ frequency }} - frequency distribution generated by qsv frequency (CSV format)
• {{ dictionary }} - the Data Dictionary that was inferred by the LLM using Stats & Frequency (JSON format)
• {{ json_add }} - inserts (in JSON format) when JSON output is requested. Note the leading space.
• {{ input_table_name }} - replaced by the name of the input file
• {{ generated_by_signature }} - replaced with model name and current timestamp
• {{ duckdb_version }} - DuckDB version (up to the minor version)
• {{ context }} - additional, user-supplied context provided via --context-file. The file type is sniffed from its contents (not its extension). Supported types: plain text, Markdown, CSV, Excel/ODS spreadsheets (extracted to CSV), and PDF or image files (JPEG/PNG/WebP/GIF) — the latter are sent to the LLM as a multimodal attachment (needs a multimodal model & endpoint; ~32 MB max), and here the variable renders a short marker. Word/PowerPoint (docx/pptx) are not supported — convert to PDF or text first. By default qsv injects this context into the user message; if a custom template references {{ context }} in any prompt, that explicit placement is honored instead. Empty when not set, so guard usage with {% if context %}.

See the header of resources/describegpt_defaults.toml for the full list of available template variables.

See resources/describegpt_defaults.toml for the default values.

Running LLMs locally with Ollama

Since the release of Ollama v0.2.0, Ollama provides the necessary OpenAI compatible endpoints to work with describegpt. You may find the Ollama OpenAI compatibility documentation here: https://github.com/ollama/ollama/blob/main/docs/openai.md.

An example command for getting an inferred description is as follows:

qsv describegpt <filepath> --base-url http://localhost:11434/v1 --model <model> --max-tokens <number> --description

Remove the arrow brackets <> and replace filepath with your file's path, <model> with the model you want to use, and number with the max tokens you want to set based on your model's context size.

SQL Query Generation and Execution ("SQL RAG" mode)

When using the --prompt option, describegpt can automatically generate and execute SQL queries to answer questions that cannot be answered using just the summary statistics and frequency distribution data. This is called "SQL RAG" mode.

Using SQL with Polars (default)

By default, when the polars feature is enabled, describegpt uses qsv's sqlp command to execute SQL queries. You can specify the --sql-results option to save the query results to a CSV file:

qsv describegpt data.csv --prompt "What's the breakdown of complaint types by borough?" --sql-results results.csv

Using SQL with DuckDB

You can also use DuckDB to execute SQL queries by setting the QSV_DUCKDB_PATH environment variable to the fully qualified path of the DuckDB binary. When set and non-empty, DuckDB will be used instead of Polars:

export QSV_DUCKDB_PATH=/usr/local/bin/duckdb
qsv describegpt data.csv --prompt "What's the breakdown of complaint types by borough?" --sql-results results.csv

When DuckDB is used, the SQL query generation guidelines are automatically modified to use DuckDB generation guidelines. See resources/describegpt_defaults.toml for the default guidelines for DuckDB and Polars.