cleanlinksmapper

September 22, 2025 · View on GitHub

Mapper to clean links like http/https/ftp in text samples.

This operator removes or replaces URLs and other web links in the text. It uses a regular expression pattern to identify and remove links. By default, it replaces the identified links with an empty string, effectively removing them. The operator can be customized with a different pattern and replacement string. It processes samples in batches and modifies the text in place. If no links are found in a sample, it is left unchanged.

映射器用于清理文本样本中的http/https/ftp等链接。

此算子删除或替换文本中的URL和其他网络链接。它使用正则表达式模式来识别和删除链接。默认情况下，它将识别到的链接替换为空字符串，从而删除它们。可以通过不同的模式和替换字符串自定义算子。它以批量方式处理样本并在原地修改文本。如果样本中没有找到链接，则保持不变。

Type 算子类型: mapper

Tags 标签: cpu, text

🔧 Parameter Configuration 参数配置

name 参数名	type 类型	default 默认值	desc 说明
`pattern`	typing.Optional[str]	`None`	regular expression pattern to search for within text.
`repl`	<class 'str'>	`''`	replacement string, default is empty string.
`args`		`''`	extra args
`kwargs`		`''`	extra args

📊 Effect demonstration 效果演示

test_mixed_https_links_text

CleanLinksMapper()

📥 input data 输入数据

Sample 1: text

This is a test,https://www.example.com/file.html?param1=value1&param2=value2

Sample 2: text

这是个测试,https://example.com/my-page.html?param1=value1&param2=value2

Sample 3: text

这是个测试,https://example.com

📤 output data 输出数据

Sample 1: text

This is a test,

Sample 2: text

这是个测试,

Sample 3: text

这是个测试,

✨ explanation 解释

This example shows the operator removing HTTPS links from text that contains both plain text and a link. The operator identifies and removes the links, leaving the rest of the text intact. For example, 'This is a test,https://www.example.com/file.html?param1=value1&param2=value2' becomes 'This is a test,' after processing. 这个示例展示了算子从同时包含纯文本和链接的文本中移除HTTPS链接。算子识别并移除这些链接，而保留其余文本不变。例如，'This is a test,https://www.example.com/file.html?param1=value1&param2=value2' 在处理后变为 'This is a test,'。

test_replace_links_text

CleanLinksMapper(repl='<LINKS>')

📥 input data 输入数据

Sample 1: text

ftp://user:password@ftp.example.com:21/

Sample 2: text

This is a sample for test

Sample 3: text

abcd://ef is a sample for test

Sample 4: text

HTTP://example.com/my-page.html?param1=value1&param2=value2

📤 output data 输出数据

Sample 1: text

<LINKS>

Sample 2: text

This is a sample for test

Sample 3: text

<LINKS> is a sample for test

Sample 4: text

<LINKS>

✨ explanation 解释

This example demonstrates the operator replacing different types of links with a custom string ''. If a sample contains a link, it will be replaced by '', while samples without links remain unchanged. For instance, 'ftp://user:password@ftp.example.com:21/' is transformed into '', whereas 'This is a sample for test' stays as it is because it doesn't contain any links. 这个示例展示了算子使用自定义字符串''替换不同类型的链接。如果一个样本包含链接，它将被替换为''，而不含链接的样本则保持不变。例如，'ftp://user:password@ftp.example.com:21/' 被转换为 ''，而 'This is a sample for test' 保持不变，因为它不包含任何链接。