===== w3lib

.. image:: https://github.com/scrapy/w3lib/actions/workflows/tests-ubuntu.yml/badge.svg :target: https://github.com/scrapy/w3lib/actions/workflows/tests-ubuntu.yml

.. image:: https://img.shields.io/codecov/c/github/scrapy/w3lib/master.svg :target: http://codecov.io/github/scrapy/w3lib?branch=master :alt: Coverage report

Overview

This is a Python library of web-related functions, such as:

• remove comments, or tags from HTML snippets
• extract base url from HTML snippets
• translate entites on HTML strings
• convert raw HTTP headers to dicts and vice-versa
• construct HTTP auth header
• converting HTML pages to unicode
• sanitize urls (like browsers do)
• extract arguments from urls

Requirements

Python 3.10+

Install

pip install w3lib

Documentation

See http://w3lib.readthedocs.org/

License

The w3lib library is licensed under the BSD license.