s3downloadfile_mapper

February 4, 2026 · View on GitHub

Mapper to download files from S3 to local files or load them into memory.

This operator downloads files from S3 URLs (s3://...) or handles local files. It supports:

  • Downloading multiple files concurrently
  • Saving files to a specified directory or loading content into memory
  • Resume download functionality
  • S3 authentication with access keys
  • Custom S3 endpoints (for S3-compatible services like MinIO)

The operator processes nested lists of URLs/paths, maintaining the original structure in the output.

用于从 S3 下载文件到本地或加载到内存的 Mapper。

该算子可从 S3 URL（s3://...）下载文件，也支持处理本地文件。功能包括：

  • 并发下载多个文件
  • 将文件保存到指定目录或将内容加载到内存
  • 支持断点续传
  • 使用访问密钥进行 S3 身份验证
  • 支持自定义 S3 端点（适用于 MinIO 等 S3 兼容服务）

该算子可处理嵌套的 URL/路径列表，并在输出中保持原始结构。

Type 算子类型: mapper

Tags 标签: cpu

🔧 Parameter Configuration 参数配置

name 参数名type 类型default 默认值desc 说明
download_field<class 'str'>NoneThe field name to get the URL/path to download.
save_dir<class 'str'>NoneThe directory to save downloaded files.
save_field<class 'str'>NoneThe field name to save the downloaded file content.
resume_download<class 'bool'>FalseWhether to resume download. If True, skip the sample if it exists.
timeout<class 'int'>30(Deprecated) Kept for backward compatibility, not used for S3 downloads.
max_concurrent<class 'int'>10Maximum concurrent downloads.
aws_access_key_id<class 'str'>NoneAWS access key ID for S3.
aws_secret_access_key<class 'str'>NoneAWS secret access key for S3.
aws_session_token<class 'str'>NoneAWS session token for S3 (optional).
aws_region<class 'str'>NoneAWS region for S3.
endpoint_url<class 'str'>NoneCustom S3 endpoint URL (for S3-compatible services).
args''extra args
kwargs''extra args