toml-bench

June 25, 2024 · View on GitHub

Which toml package to use in python?

Report

Version

The verions of the packages tested in this report.

	Version
toml	0.10.2
tomli/tomli_w	2.0.1; tomli_w: 1.0.0
tomlkit	0.12.5
rtoml	0.11.0
qtoml	0.3.1
tomllib	(Python 3.12.2)

Dumping `None` value

How the package dumps None value in python

Literally <package>.dumps(None)

	Dumped value or error
toml	'NoneType' object is not iterable
tomli/tomli_w	'NoneType' object has no attribute 'items'
tomlkit	Expecting Mapping or TOML Container, <class 'NoneType'> given
rtoml	"null" --- rtoml v0.11+ supports dumping None to a desired string: `rtoml.dumps(data, none_value='@None')`: "@None"
qtoml	'NoneType' object has no attribute 'items'
tomllib	module 'tomllib' has no attribute 'dumps'

Dumping key-`None` pair

How the package dumps key-value pair with value None

Literally <package>.dumps({"key": None})

	Dumped value or error
toml
tomli/tomli_w	Object of type <class 'NoneType'> is not TOML serializable
tomlkit	Invalid type <class 'NoneType'>
rtoml	key = "null" --- rtoml v0.11+ supports dumping None to a desired string: `rtoml.dumps(data, none_value='@None')`: key = "@None"
qtoml	TOML cannot encode None
tomllib	module 'tomllib' has no attribute 'dumps'

Dumping list with `None` value

How the package dumps a list with None value in it.

Literally <package>.dumps({"key": [1, 2, 3, None, 5]})

	Dumped value or error
toml	key = [ 1, 2, 3, "None", 5,]
tomli/tomli_w	Object of type <class 'NoneType'> is not TOML serializable
tomlkit	Invalid type <class 'NoneType'>
rtoml	key = [1, 2, 3, "null", 5] --- rtoml v0.11+ supports dumping None to a desired string: `rtoml.dumps(data, none_value='@None')`: key = [1, 2, 3, "@None", 5]
qtoml	bad type '<class 'NoneType'>' for dump_value
tomllib	module 'tomllib' has no attribute 'dumps'

Loading `None`-like values

How the package loads None-like value in string

Literally <package>.loads('v1 = "null" v2 = "None"')

	Loaded as
toml	{'v1': 'null', 'v2': 'None'}
tomli/tomli_w	module 'tomli_w' has no attribute 'loads'
tomlkit	{'v1': 'null', 'v2': 'None'}
rtoml	{'v1': 'null', 'v2': 'None'} --- rtoml v0.11+ supports loading custom None values: `rtoml.loads(data, none_value='None')`: {'v1': 'null', 'v2': None} `rtoml.loads(data, none_value='null')`: {'v1': None, 'v2': 'None'}
qtoml	{'v1': 'null', 'v2': 'None'}
tomllib	{'v1': 'null', 'v2': 'None'}

Dumping a heterogenous array

How the package dumps a python dictionary with a heterogenous array.

Literally <package>.dumps({"v": [1, 1.2, True, "string"]})

	Dumped value or error
toml	v = [ 1, 1.2, true, "string",]
tomli/tomli_w	v = [ 1, 1.2, true, "string", ]
tomlkit	v = [1, 1.2, true, "string"]
rtoml	v = [1, 1.2, true, "string"]
qtoml	v = [1, 1.2, true, 'string']
tomllib	Dumping not supported

Loading a heterogenous array

How the package loads a toml string with a heterogenous array.

Literally <package>.loads('v = [1, 1.2, True, "string"]')

	Loaded as
toml	Not a homogeneous array (line 2 column 1 char 1)
tomli/tomli_w	`{'v': [1, 1.2, True, 'string']}`
tomlkit	`{'v': [1, 1.2, True, 'string']}`
rtoml	`{'v': [1, 1.2, True, 'string']}`
qtoml	`{'v': [1, 1.2, True, 'string']}`
tomllib	`{'v': [1, 1.2, True, 'string']}`

Dumping a nested array

How the package dumps a python dictionary with a nested array.

Literally <package>.dumps({"v": [[1], [1, 2]]})

	Dumped value or error
toml	v = [ [ 1,], [ 1, 2,],]
tomli/tomli_w	v = [ [ 1, ], [ 1, 2, ], ]
tomlkit	v = [[1], [1, 2]]
rtoml	v = [[1], [1, 2]]
qtoml	v = [[1], [1, 2]]
tomllib	Dumping not supported

Loading a nested array

How the package loads a toml string with a nested array.

Literally <package>.loads('v = [[1], [1, 2]]')

	Loaded as
toml	`{'v': [[1], [1, 2]]}`
tomli/tomli_w	`{'v': [[1], [1, 2]]}`
tomlkit	`{'v': [[1], [1, 2]]}`
rtoml	`{'v': [[1], [1, 2]]}`
qtoml	`{'v': [[1], [1, 2]]}`
tomllib	`{'v': [[1], [1, 2]]}`

Dumping keeps order of keys?

Whether the package preserves the order of the keys while dumps a python dictionary.

Thus, whether <package>.dumps({"c": 1, "a": 2, "b": 3}) yields a string like c = 1\na = 2\nb = 3\n.

	Order kept?
toml	Kept
tomli/tomli_w	Kept
tomlkit	Kept
rtoml	Kept
qtoml	Kept
tomllib	Dumping not supported

Loading keeps order of keys?

Whether the package preserves the order of the keys while loads a TOML string.

Thus, whether <package>.loads('c = 1\na = 2\nb = 3\n') yields a dictionary with keys in the order of ['c', 'a', 'b'].

	Order kept?
toml	Kept
tomli/tomli_w	Kept
tomlkit	Kept
rtoml	Kept
qtoml	Kept
tomllib	Kept

Dumping unicode

How the package dumps Unicode in python

Literally, <package>.dumps({"你好": "世界"})

	Dumped value
toml	"你好" = "世界"
tomli/tomli_w	"你好" = "世界"
tomlkit	"你好" = "世界"
rtoml	"你好" = "世界"
qtoml	'你好' = '世界'
tomllib	Dumping not supported

Loaded unicode

How the package loads a file with unicode.

The file was created by:

## Create a file with unicode content
with open(self.datafile, "w", encoding="utf-8") as f:
    f.write('"你好" = "世界"\n')

## Use `<package>.load()` to load the file
with open(self.datafile, "r", encoding="utf-8") as f:
    loaded = <package>.load(f)

	Loaded as
toml	{'你好': '世界'}
tomli/tomli_w	File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')` When loaded with `rb`: {'你好': '世界'}
tomlkit	{'你好': '世界'}
rtoml	{'你好': '世界'}
qtoml	{'你好': '世界'}
tomllib	File must be opened in binary mode, e.g. use `open('foo.toml', 'rb')` When loaded with `rb`: {'你好': '世界'}

Compliance with valid tests in toml-test

Test the compliance with the standard test suites for valid toml files here:

https://github.com/BurntSushi/toml-test/

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

	Result (toml-test v1.5.0)
toml	spec/array-0.toml Not a homogeneous array (line 8 column 1 char 261) spec/keys-4.toml Found invalid character in key name: 'c'. Try quoting the key name. (line 2 column 8 char 57) spec/local-time-0.toml Parsed as unexpected data. datetime/no-seconds.toml invalid literal for int() with base 0: '13:37' (line 2 column 1 char 46) datetime/local-time.toml Parsed as unexpected data. datetime/datetime.toml Parsed as unexpected data. comment/tricky.toml Parsed as unexpected data. key/dotted-1.toml Parsed as unexpected data. key/unicode.toml Found invalid character in key name: '‍'. Try quoting the key name. (line 5 column 2 char 67) key/dotted-2.toml Found invalid character in key name: '"'. Try quoting the key name. (line 7 column 11 char 166) key/quoted-unicode.toml Duplicate keys! (line 3 column 1 char 19) key/dotted-empty.toml Duplicate keys! (line 2 column 1 char 17) key/escapes.toml Parsed as unexpected data. table/empty-name.toml Can't have a keygroup with an empty name (line 1 column 1 char 0) string/raw-multiline.toml Unbalanced quotes (line 20 column 50 char 532) string/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28) string/hex-escape.toml Reserved escape sequence used (line 3 column 1 char 35) string/escape-esc.toml Reserved escape sequence used (line 1 column 1 char 0) string/multiline-quotes.toml Unterminated string found. Reached end of file. (line 27 column 1 char 664) float/zero.toml Weirdness with leading zeroes or underscores in your number. (line 4 column 1 char 47) array/mixed-int-string.toml Not a homogeneous array (line 1 column 1 char 0) array/nested-double.toml Not a homogeneous array (line 1 column 1 char 0) array/string-with-comma-2.toml string index out of range array/mixed-int-float.toml Not a homogeneous array (line 1 column 1 char 0) array/mixed-string-table.toml list index out of range array/mixed-int-array.toml Not a homogeneous array (line 1 column 1 char 0) inline-table/multiline.toml Invalid inline table value encountered (line 1 column 1 char 0) inline-table/key-dotted-1.toml Parsed as unexpected data. inline-table/key-dotted-5.toml Not a homogeneous array (line 2 column 1 char 20) inline-table/newline.toml Key name found without value. Reached end of line. (line 5 column 2 char 98) 157/187 (83.96%) passed
tomli/tomli_w	datetime/no-seconds.toml Expected newline or end of document after a statement (at line 2, column 23) key/unicode.toml Invalid statement (at line 3, column 1) string/hex-escape.toml Unescaped '' in a string (at line 3, column 22) string/escape-esc.toml Unescaped '' in a string (at line 1, column 10) inline-table/newline.toml Invalid initial character for a key part (at line 3, column 21) 182/187 (97.33%) passed
tomlkit	datetime/no-seconds.toml Invalid number at line 2 col 25 key/unicode.toml Empty key at line 3 col 0 string/hex-escape.toml Invalid character 'x' in string at line 3 col 20 string/escape-esc.toml Invalid character 'e' in string at line 1 col 8 inline-table/newline.toml Empty key at line 3 col 20 182/187 (97.33%) passed
rtoml	spec/table-9.toml duplicate key: `apple` for key `fruit` at line 8 column 1 datetime/no-seconds.toml expected a colon, found a newline at line 2 column 26 key/unicode.toml unexpected character found: `\u{20ac}` at line 3 column 1 table/array-within-dotted.toml duplicate key: `apple` for key `fruit` at line 4 column 1 string/hex-escape.toml invalid escape character in string: `x` at line 3 column 21 string/escape-esc.toml invalid escape character in string: `e` at line 1 column 9 inline-table/newline.toml expected a table key, found a newline at line 3 column 21 180/187 (96.26%) passed
qtoml	spec/string-4.toml Didn't find expected newline (line 7, column 62) spec/string-7.toml Didn't find expected newline (line 7, column 50) datetime/no-seconds.toml can't parse type (line 2, column 20) datetime/milliseconds.toml Didn't find expected newline (line 2, column 27) datetime/datetime.toml Didn't find expected newline (line 4, column 18) comment/after-literal-no-ws.toml can't parse type (line 1, column 4) comment/tricky.toml can't parse type (line 11, column 7) key/unicode.toml '€' cannot begin key (line 3, column 0) string/raw-multiline.toml Didn't find expected newline (line 22, column 3) string/hex-escape.toml \x not a valid escape (line 3, column 43) string/escape-esc.toml \e not a valid escape (line 1, column 33) string/multiline-quotes.toml Didn't find expected newline (line 4, column 26) inline-table/newline.toml ' ' cannot begin key (line 3, column 20) 174/187 (93.05%) passed
tomllib	datetime/no-seconds.toml Expected newline or end of document after a statement (at line 2, column 23) key/unicode.toml Invalid statement (at line 3, column 1) string/hex-escape.toml Unescaped '' in a string (at line 3, column 22) string/escape-esc.toml Unescaped '' in a string (at line 1, column 10) inline-table/newline.toml Invalid initial character for a key part (at line 3, column 21) 182/187 (97.33%) passed

Compliance with invalid tests in toml-test

Test the compliance with the standard test suites for invalid toml files here:

https://github.com/BurntSushi/toml-test/

Not OK: The toml file is parsed without error, but expected to fail.
OK: All files are failed to parse, as expected. Showing the last parsing error.

	Result (toml-test v1.5.0)
toml	Not OK: integer/double-sign-plus.toml incorrectly parsed. Not OK: integer/us-after-bin.toml incorrectly parsed. Not OK: integer/double-sign-nex.toml incorrectly parsed. Not OK: integer/us-after-hex.toml incorrectly parsed. Not OK: integer/us-after-oct.toml incorrectly parsed. Not OK: spec/inline-table-2-0.toml incorrectly parsed. Not OK: datetime/offset-overflow-minute.toml incorrectly parsed. Not OK: datetime/offset-overflow-hour.toml incorrectly parsed. Not OK: control/comment-del.toml incorrectly parsed. Not OK: control/string-del.toml incorrectly parsed. Not OK: 63 more items incorrectly parsed. 298/371 (80.32%) passed
tomli/tomli_w	OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18) 371/371 (100%) passed
tomlkit	Not OK: control/comment-cr.toml incorrectly parsed. Not OK: control/multi-cr.toml incorrectly parsed. Not OK: control/rawmulti-cd.toml incorrectly parsed. Not OK: control/bare-cr.toml incorrectly parsed. Not OK: table/append-with-dotted-keys-1.toml incorrectly parsed. Not OK: table/overwrite-array-in-parent.toml incorrectly parsed. Not OK: table/append-to-array-with-dotted-keys.toml incorrectly parsed. Not OK: table/append-with-dotted-keys-2.toml incorrectly parsed. Not OK: array/extend-defined-aot.toml incorrectly parsed. Not OK: inline-table/overwrite-09.toml incorrectly parsed. 361/371 (97.30%) passed
rtoml	Not OK: integer/positive-hex.toml incorrectly parsed. Not OK: integer/positive-bin.toml incorrectly parsed. Not OK: integer/positive-oct.toml incorrectly parsed. Not OK: datetime/offset-overflow-minute.toml incorrectly parsed. Not OK: datetime/offset-overflow-hour.toml incorrectly parsed. Not OK: control/comment-del.toml incorrectly parsed. Not OK: control/comment-cr.toml incorrectly parsed. Not OK: control/multi-cr.toml incorrectly parsed. Not OK: control/rawmulti-cd.toml incorrectly parsed. Not OK: control/bare-cr.toml incorrectly parsed. 361/371 (97.30%) passed
qtoml	Not OK: spec/inline-table-2-0.toml incorrectly parsed. Not OK: spec/table-9-1.toml incorrectly parsed. Not OK: spec/table-9-0.toml incorrectly parsed. Not OK: datetime/offset-overflow-minute.toml incorrectly parsed. Not OK: control/comment-del.toml incorrectly parsed. Not OK: control/comment-lf.toml incorrectly parsed. Not OK: control/comment-null.toml incorrectly parsed. Not OK: control/comment-ff.toml incorrectly parsed. Not OK: control/comment-cr.toml incorrectly parsed. Not OK: control/multi-cr.toml incorrectly parsed. Not OK: 14 more items incorrectly parsed. 347/371 (93.53%) passed
tomllib	OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18) 371/371 (100%) passed

Compliance with valid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for valid toml files here:

https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

	Result (cpython tag 3.12.4)
toml	apostrophes-in-literal-string.toml Unbalanced quotes (line 1 column 50 char 49) five-quotes.toml Unterminated string found. Reached end of file. (line 7 column 1 char 97) dates-and-times/datetimes.toml Parsed as unexpected data. multiline-basic-str/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28) 8/12 (66.67%) passed
tomli/tomli_w	OK, 12/12 (100%) passed
tomlkit	OK, 12/12 (100%) passed
rtoml	OK, 12/12 (100%) passed
qtoml	apostrophes-in-literal-string.toml Didn't find expected newline (line 3, column 3) five-quotes.toml Didn't find expected newline (line 3, column 3) dates-and-times/datetimes.toml Didn't find expected newline (line 1, column 19) 9/12 (75.00%) passed
tomllib	OK, 12/12 (100%) passed

Compliance with invalid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for invalid toml files here:

https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid

Not OK: The toml file is parsed without error, but expected to fail.
OK: All files are failed to parse, as expected. Showing the last parsing error.

	Result (cpython tag 3.12.4)
toml	Not OK: invalid-comment-char.toml incorrectly parsed. Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed. Not OK: array/unclosed-empty.toml incorrectly parsed. Not OK: array/file-end-after-val.toml incorrectly parsed. Not OK: array/unclosed-after-item.toml incorrectly parsed. Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed. Not OK: inline-table/unclosed-empty.toml incorrectly parsed. 41/50 (82.00%) passed
tomli/tomli_w	OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21) 50/50 (100%) passed
tomlkit	Not OK: array-of-tables/overwrite-array-in-parent.toml incorrectly parsed. Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-aot.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed. Not OK: inline-table/override-val-in-table.toml incorrectly parsed. 44/50 (88.00%) passed
rtoml	Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed. 49/50 (98.00%) passed
qtoml	Not OK: non-scalar-escaped.toml incorrectly parsed. Not OK: invalid-comment-char.toml incorrectly parsed. Not OK: table/redefine-2.toml incorrectly parsed. Not OK: table/redefine-1.toml incorrectly parsed. Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed. Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed. Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed. Not OK: inline-table/override-val-with-table.toml incorrectly parsed. 41/50 (82.00%) passed
tomllib	OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21) 50/50 (100%) passed

Running speed with data provided by `rtoml`

Test the speed of loading and dumping the loaded using data provided by rtoml

https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml

	Loading speed	Dumping speed
toml	Excluded (heterogeneous arrays not supported)	Excluded (heterogeneous arrays not supported)
tomli/tomli_w	2.14s (5000 iterations)	0.73s (5000 iterations)
tomlkit	39.78s (5000 iterations)	0.98s (5000 iterations)
rtoml	0.37s (5000 iterations)	0.08s (5000 iterations)
qtoml	4.99s (5000 iterations)	1.87s (5000 iterations)
tomllib	2.04s (5000 iterations)	Dumping not supported

Running speed with data provided by `tomli`

Test the speed of loading and dumping the loaded using data provided by tomli

https://github.com/hukkin/tomli/raw/master/benchmark/data.toml

	Loading speed	Dumping speed
toml	Excluded (heterogeneous arrays not supported)	Excluded (heterogeneous arrays not supported)
tomli/tomli_w	1.41s (5000 iterations)	0.46s (5000 iterations)
tomlkit	24.55s (5000 iterations)	0.51s (5000 iterations)
rtoml	0.32s (5000 iterations)	0.16s (5000 iterations)
qtoml	3.63s (5000 iterations)	1.25s (5000 iterations)
tomllib	1.44s (5000 iterations)	Dumping not supported

pip install -U toml-bench

Generate your own report

toml-bench

Use a different data directory than the default one

toml-bench --datadir /tmp/toml-bench

Write the report to a markdown file

toml-bench --report ./README.md

Test with a different version of compliance set (`BurntSushi/toml-test`)

toml-bench --comver 1.0.0

Use a different number of iterations in speed tests

toml-bench --iter 5000

Test with different versions of packages

git clone https://github.com/pwwang/toml-bench.git
cd toml-bench
# See https://python-poetry.org/docs/cli/#add
# for how to specify a version constraint
poetry add "tomli=2.0.0"
poetry update
poetry install
poetry run toml-bench