Data Format & Packing Pipeline

中文阅读

Overview

Training expects packed JSONL: each line contains multiple original samples concatenated into a single sequence up to pack_length tokens. This maximizes GPU utilization by removing padding waste.

The pipeline: raw JSONL → tokenize + count → pack → training-ready JSONL.

---

1. Raw OCR JSONL Schema

Each line in a raw JSONL file is one training sample:

{
  "image_path": ["/absolute/path/to/image.png"],
  "conversations": [
    {
      "from": "human",
      "value": "<image>\n提取文档图片中正文的所有信息用markdown格式表示..."
    },
    { "from": "gpt", "value": "# Title\n\nBody text ..." }
  ]
}

Fields:

Field	Type	Required	Notes
image_path	list[str]	✅	Absolute paths. Usually 1 image; multi-image supported.
conversations	list[dict]	✅	Alternating human / gpt turns. <image> placeholder in human value indicates image insertion point.

2. Packing Pipeline

The tools/pipeline_count_and_pack.py script does two things:

1. Count phase (parallel): tokenizes each sample and writes token counts to count_output_dir/*.jsonl
2. Pack phase: greedily packs samples up to pack_length tokens per output line (First-Fit Decreasing)

Config file: configs/data_list.txt

Plain text file, one path per line:

/path/to/dataset_A.jsonl
/path/to/dataset_B.jsonl
/path/to/dataset_C.jsonl
# Comments starting with # are ignored

Run

MODEL_PATH=/path/to/HunyuanOCR \
INPUT_LIST=./configs/data_list.txt \
PACK_OUTPUT=./data/parsing_packed_20480.jsonl \
    bash scripts/pack_data.sh

Tunable parameters (env vars)

Env var	Default	Description
MODEL_PATH	(required)	HunyuanOCR base model dir (for tokenizer + processor)
INPUT_LIST	./configs/data_list.txt	Path to file listing raw JSONLs
COUNT_OUTPUT_DIR	./data/parsing_jsonl_count	Temp dir for count-phase output
PACK_OUTPUT	./data/parsing_packed_{PACK_LEN}.jsonl	Final packed JSONL
PACK_LEN	20480	Max sequence length per packed line
NUM_PROCESSES	32	Multiprocess count workers
THREADS_PER_PROCESS	8	Threads per count worker
LOG_FILE	pack_data.log	Progress log path

Output schema (packed JSONL)

Each output line contains:

{
    "packed_samples": [
        {
            "image_path": ["/abs/path/1.png"],
            "conversations": [...]
        },
        {
            "image_path": ["/abs/path/2.png"],
            "conversations": [...]
        },
        ...
    ],
    "cu_seqlens": [0, 4123, 8567, ..., 20351],
    "total_tokens": 20351
}

Fields:

• packed_samples: original raw samples concatenated in this pack
• cu_seqlens: cumulative token boundaries (for FlashAttention varlen)
• total_tokens: sum ≤ pack_length

Performance tips

• CPU-bound: scales linearly with NUM_PROCESSES. On 128-core machine, use NUM_PROCESSES=32~64.
• Image data does not need to be pre-loaded — the count phase only tokenizes text; images are loaded lazily during training.
• First run generates count cache; re-running with the same input list skips the count phase.

3. Verifying Packed Data

Sanity check the packed JSONL:

python -c "
import json
with open('./data/parsing_packed_20480.jsonl') as f:
    for i, line in enumerate(f):
        r = json.loads(line)
        print(f'pack {i}: {len(r[\"packed_samples\"])} samples, '
              f'{r[\"total_tokens\"]} tokens')
        if i >= 3: break
"

You should see something like:

pack 0: 7 samples, 20438 tokens
pack 1: 5 samples, 19821 tokens
pack 2: 12 samples, 20301 tokens
pack 3: 3 samples, 18654 tokens

Each pack is close to pack_length=20480 — that's the goal.