This tests various parsers agains the toml-test suite.
"Decoder tests" check if valid TOML documents are processed correctly, "encoder tests" checks writing TOML files, and "invalid input" checks that the parser rejects invalid TOML documents. Implementations are marked as "compliant" (✅) if both the "valid" and "encoder" tests pass, otherwise they're marked as non-compliant (❌). Usually it's not a big deal to accept some technically invalid TOML; almost all failing "invalid" tests are uncommon edge cases.
The benchmarks are a simple decode of a 15k and 5M file. Results are displayed as "good", "slow", and "very slow" only. Doing a fully fair comparison is not straight-forward as some implementations will parse numbers, dates, etc. when reading the file, and others delay this parsing until the data is used. It's also not a contest on who can write the fastest parser – it's just to establish that performance is within reasonable bounds. Some parsers seem to be O(n²) or thereabouts and scale very poorly to large files.
Name | Decoder tests | Encoder tests | Invalid input | 15k file | 5M file | |||
---|---|---|---|---|---|---|---|---|
✅ | C++ | toml++ | fail: 0 | pass: 182fail: 0 | pass: 182fail: 0 | pass: 371good | good | output |
✅ | C++ | toml11 | fail: 0 | pass: 182(TODO) | fail: 0 | pass: 371good | slow | output |
✅ | C | toml-c | fail: 0 | pass: 182(doesn't include encoder) | fail: 21 | pass: 350good | slow | output |
❌ | C | tomlc99 | fail: 7 | pass: 175(doesn't include encoder) | fail: 78 | pass: 293good | slow | output |
❌ | C# | tomlyn | fail: 3 | pass: 179(TODO) | fail: 5 | pass: 366slow | good | output |
✅ | Dart | toml.dart | fail: 0 | pass: 182fail: 0 | pass: 182fail: 39 | pass: 332good | good | output |
❌ | Fortran | toml-f | fail: 11 | pass: 171fail: 12 | pass: 170fail: 32 | pass: 339good | very slow | output |
✅ | Go | go-toml | fail: 0 | pass: 182fail: 0 | pass: 182fail: 0 | pass: 371good | slow | output |
✅ | Go | toml | fail: 0 | pass: 182fail: 0 | pass: 182fail: 15 | pass: 356good | good | output |
❌ | Guile | toml | fail: 5 | pass: 177fail: 4 | pass: 178fail: 15 | pass: 356(TODO) | (TODO) | output |
❌ | Haskell | toml-parser | fail: 1 | pass: 181fail: 0 | pass: 182fail: 2 | pass: 369good | good | output |
❌ | Haskell | toml-reader | fail: 2 | pass: 177(doesn't include encoder) | fail: 2 | pass: 366good | good | output |
✅ | JS | j-toml | fail: 0 | pass: 182fail: 0 | pass: 182fail: 14 | pass: 357good | good | output |
❌ | JS | smol-toml | fail: 0 | pass: 181fail: 7 | pass: 174fail: 14 | pass: 357(error) | (error) | output |
✅ | JS | toml-eslint-parser | fail: 0 | pass: 182(doesn't include encoder) | fail: 9 | pass: 362good | good | output |
❌ | Lisp | clop | fail: 3 | pass: 179(doesn't include encoder) | fail: 0 | pass: 371(TODO) | (TODO) | output |
❌ | OCAML | otoml | fail: 1 | pass: 181fail: 8 | pass: 174fail: 52 | pass: 319(TODO) | (TODO) | output |
❌ | Python | toml | fail: 18 | pass: 164fail: 14 | pass: 168fail: 73 | pass: 298good | good | output |
✅ | Python | tomli | fail: 0 | pass: 182fail: 0 | pass: 182fail: 0 | pass: 371good | good | output |
✅ | Python | tomlkit | fail: 0 | pass: 182fail: 0 | pass: 182fail: 6 | pass: 365slow | very slow | output |
✅ | Python | tomllib | fail: 0 | pass: 182(doesn't include encoder) | fail: 0 | pass: 371good | good | output |
❌ | Racket | toml-racket | fail: 182 | pass: 0fail: 7 | pass: 175fail: 371 | pass: 0(TODO) | (TODO) | output |
❌ | Ruby | perfect_toml | fail: 0 | pass: 182fail: 1 | pass: 181fail: 9 | pass: 362good | good | output |
❌ | Ruby | toml-rb | fail: 27 | pass: 155fail: 28 | pass: 154fail: 90 | pass: 281(TODO) | (TODO) | output |
✅ | Ruby | tomlrb | fail: 0 | pass: 182(doesn't include encoder) | fail: 20 | pass: 351good | slow | output |
❌ | Rust | basic-toml | fail: 21 | pass: 161(TODO) | fail: 4 | pass: 367good | good | output |
❌ | Rust | taplo | fail: 4 | pass: 178(TODO) | fail: 3 | pass: 368good | very slow | output |
✅ | Rust | toml | fail: 0 | pass: 182fail: 0 | pass: 182fail: 0 | pass: 371good | good | output |
✅ | Rust | toml_edit | fail: 0 | pass: 182fail: 0 | pass: 182fail: 0 | pass: 371good | good | output |