Simplemma: a simple multilingual lemmatizer for Python

Purpose

Lemmatization is the process of grouping together the inflected forms of a word so they can be analysed as a single item, identified by the word's lemma, or dictionary form. Unlike stemming, lemmatization outputs word units that are still valid linguistic forms.

In modern natural language processing (NLP), this task is often indirectly tackled by more complex systems encompassing a whole processing pipeline. However, it appears that there is no straightforward way to address lemmatization in Python although this task can be crucial in fields such as information retrieval and NLP.

Simplemma provides a simple and multilingual approach to look for base forms or lemmata. It may not be as powerful as full-fledged solutions but it is generic, easy to install and straightforward to use. In particular, it does not need morphosyntactic information and can process a raw series of tokens or even a text with its built-in tokenizer. By design it should be reasonably fast and work in a large majority of cases, without being perfect.

With its comparatively small footprint it is especially useful when speed and simplicity matter, in low-resource contexts, for educational purposes, or as a baseline system for lemmatization and morphological analysis.

Currently, 50 languages are partly or fully supported (see the list of supported languages).

Installation

The current library is written in pure Python with no dependencies: pip install simplemma

• pip install -U simplemma for updates
• pip install git+https://github.com/adbar/simplemma for the cutting-edge version

The last version supporting Python 3.6 and 3.7 is simplemma==1.0.0.

Usage

Word-by-word

Simplemma is used by selecting a language of interest and then applying the data on a list of words.

>>> import simplemma
# get a word
myword = 'masks'
# decide which language to use and apply it on a word form
>>> simplemma.lemmatize(myword, lang='en')
'mask'
# apply it on a list of tokens
>>> mytokens = ['Hier', 'sind', 'Vaccines']
>>> [simplemma.lemmatize(t, lang='de') for t in mytokens]
['hier', 'sein', 'Vaccines']

Chaining languages

Chaining several languages can improve coverage, they are used in sequence:

>>> from simplemma import lemmatize
>>> lemmatize('Vaccines', lang=('de', 'en'))
'vaccine'
>>> lemmatize('spaghettis', lang='it')
'spaghettis'
>>> lemmatize('spaghettis', lang=('it', 'fr'))
'spaghetti'
>>> lemmatize('spaghetti', lang=('it', 'fr'))
'spaghetto'

Greedier decomposition

For certain languages a greedier decomposition is activated by default as it can be beneficial, mostly due to a certain capacity to address affixes in an unsupervised way. This can be triggered manually by setting the greedy parameter to True.

This option also triggers a stronger reduction through an additional iteration of the search algorithm, e.g. "angekündigten" → "angekündigt" (standard) → "ankündigen" (greedy). In some cases it may be closer to stemming than to lemmatization.

>>> simplemma.lemmatize('angekündigten', lang='de', greedy=False)
'angekündigt' # 1 step: reduction to past participle
>>> simplemma.lemmatize('angekündigten', lang='de', greedy=True)
'ankündigen' # 2 steps: further reduction to infinitive verb

is_known()

The additional function is_known() checks if a given word is present in the language data:

>>> from simplemma import is_known
>>> is_known('spaghetti', lang='it')
True

Tokenization

A simple tokenization function is provided for convenience:

>>> from simplemma import simple_tokenizer
>>> simple_tokenizer('Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.')
['Lorem', 'ipsum', 'dolor', 'sit', 'amet', ',', 'consectetur', 'adipiscing', 'elit', ',', 'sed', 'do', 'eiusmod', 'tempor', 'incididunt', 'ut', 'labore', 'et', 'dolore', 'magna', 'aliqua', '.']
# for an iterator instead of a list, use the RegexTokenizer directly
>>> from simplemma import RegexTokenizer
>>> RegexTokenizer().split_text('Lorem ipsum dolor sit amet')
<generator object ...>

The functions text_lemmatizer() and lemma_iterator() chain tokenization and lemmatization. They accept the same greedy argument as lemmatize():

>>> from simplemma import text_lemmatizer
>>> sentence = 'Sou o intervalo entre o que desejo ser e os outros me fizeram.'
>>> text_lemmatizer(sentence, lang='pt')
# caveat: desejo is also a noun, should be desejar here
['ser', 'o', 'intervalo', 'entre', 'o', 'que', 'desejo', 'ser', 'e', 'o', 'outro', 'me', 'fazer', '.']
# same principle, returns a generator and not a list
>>> from simplemma import lemma_iterator
>>> lemma_iterator(sentence, lang='pt')

Caveats

# don't expect too much though
# this diminutive form isn't in the model data
>>> simplemma.lemmatize('spaghettini', lang='it')
'spaghettini' # should read 'spaghettino'
# the algorithm cannot choose between valid alternatives yet
>>> simplemma.lemmatize('son', lang='es')
'ser' # 3rd-person plural of 'ser'; but 'son' is also a noun (a Cuban music genre)

As the focus lies on overall coverage, some short frequent words (typically: pronouns and conjunctions) may need post-processing, this generally concerns a few dozens of tokens per language.

The current absence of morphosyntactic information is an advantage in terms of simplicity. However, it is also an impassable frontier regarding lemmatization accuracy, for example when it comes to disambiguating between past participles and adjectives derived from verbs in Germanic and Romance languages. In most cases, simplemma often does not change such input words.

The greedy algorithm seldom produces invalid forms. It is designed to work best in the low-frequency range, notably for compound words and neologisms. Aggressive decomposition is only useful as a general approach in the case of morphologically-rich languages, where it can also act as a linguistically motivated stemmer.

Bug reports over the issues page are welcome.

Language detection

Language detection works by providing a text and a tuple lang consisting of a series of languages of interest. Each score is a proportion between 0 and 1. The proportions are computed independently per language, so a token recognized in several languages counts towards each of them and the scores need not sum to 1.

The langdetect() function returns a list of language codes along with their corresponding scores, appending "unk" for the proportion of unknown or out-of-vocabulary tokens. The proportion of tokens that belong to the target language(s) can also be obtained directly with the in_target_language() function, which returns a single ratio.

# import necessary functions
>>> from simplemma import in_target_language, langdetect
# language detection
>>> langdetect('"Exoplaneta, též extrasolární planeta, je planeta obíhající kolem jiné hvězdy než kolem Slunce."', lang=("cs", "sk"))
[("cs", 0.75), ("sk", 0.125), ("unk", 0.25)]
# proportion of known words
>>> in_target_language("opera post physica posita (τὰ μετὰ τὰ φυσικά)", lang="la")
0.5

The greedy argument (extensive in past software versions) triggers use of the greedier decomposition algorithm described above, thus extending word coverage and recall of detection at the potential cost of a lesser accuracy.

Advanced usage via classes

The functions described above are suitable for simple usage, but you can have more control by instantiating Simplemma classes and calling their methods instead. Lemmatization is handled by the Lemmatizer class, while language detection is handled by the LanguageDetector class. These in turn rely on different lemmatization strategies, which are implementations of the LemmatizationStrategy protocol. The DefaultStrategy implementation uses a combination of different strategies, one of which is DictionaryLookupStrategy. It looks up tokens in a dictionary created by a DictionaryFactory.

For example, it is possible to conserve RAM by limiting the number of cached language dictionaries (default: 8) by creating a custom DefaultDictionaryFactory with a specific cache_max_size setting, creating a DefaultStrategy using that factory, and then creating a Lemmatizer and/or a LanguageDetector using that strategy:

# import necessary classes
>>> from simplemma import LanguageDetector, Lemmatizer
>>> from simplemma.strategies import DefaultStrategy
>>> from simplemma.strategies.dictionaries import DefaultDictionaryFactory

LANG_CACHE_SIZE = 5  # How many language dictionaries to keep in memory at once (max)
>>> dictionary_factory = DefaultDictionaryFactory(cache_max_size=LANG_CACHE_SIZE)
>>> lemmatization_strategy = DefaultStrategy(dictionary_factory=dictionary_factory)

# lemmatize using the above customized strategy
>>> lemmatizer = Lemmatizer(lemmatization_strategy=lemmatization_strategy)
>>> lemmatizer.lemmatize('doughnuts', lang='en')
'doughnut'

# detect languages using the above customized strategy
>>> language_detector = LanguageDetector('la', lemmatization_strategy=lemmatization_strategy)
>>> language_detector.proportion_in_target_languages("opera post physica posita (τὰ μετὰ τὰ φυσικά)")
0.5

For more information see the extended documentation.

Reducing memory usage

Simplemma provides an alternative solution for situations where low memory usage is more important than lemmatization and language detection performance. The quickest way to opt in is the low_memory flag, available on lemmatize, text_lemmatizer, lemma_iterator, is_known, langdetect and in_target_language:

>>> from simplemma import lemmatize
>>> lemmatize('doughnuts', lang='en', low_memory=True)
'doughnut'

This selects the stdlib-only StreamDictionaryFactory (see below): the most memory-frugal backend, reading the dictionary stream directly with no full-dict build spike and no on-disk cache. TrieDictionaryFactory reaches a lower steady-state footprint but spikes and writes to disk on first use, so it is not auto-selected — request it explicitly (see below) when its RAM/speed trade-off suits you. For explicit control over which backend is used — including with Lemmatizer and LanguageDetector instances — build a strategy directly, as described in the rest of this section. DefaultStrategy also accepts the same low_memory flag, but not together with an explicit dictionary_factory:

>>> from simplemma import Lemmatizer
>>> from simplemma.strategies import DefaultStrategy

>>> strategy = DefaultStrategy(low_memory=True)
>>> Lemmatizer(lemmatization_strategy=strategy).lemmatize('doughnuts', lang='en')
'doughnut'

The three backends trade memory against speed as follows (measured on German, ~1.1M dictionary entries; exact figures vary by language and hardware):

Backend	Peak RAM	Load time	Uncached lookup²	Cached lookup³	Extra dependency
DefaultDictionaryFactory	~175 MB	~0.6 s	fastest (baseline)	fastest (baseline)	none
TrieDictionaryFactory	~30 MB	~1 ms (warm)¹	~2.5× slower	~1.2× slower	marisa-trie
StreamDictionaryFactory	~50 MB	~0.6 s	~18× slower	~6× slower	none

Choosing between them: pick DefaultDictionaryFactory when throughput matters most and memory is not a constraint; TrieDictionaryFactory for the best RAM/speed trade-off if installing marisa-trie is an option; StreamDictionaryFactory for the same low RAM with no extra dependency, at a bigger speed cost. The RAM saving compounds with every additional language kept loaded at once, since DefaultDictionaryFactory holds each language's full dict in memory for as long as it stays cached — though German is near the largest shipped dictionary and includes a fixed Python baseline, so smaller languages add less than the table's absolute numbers suggest.

¹ Warm-load time only; see below for the (one-time, per language) cost of building the trie. ² Per single lookup, bypassing any cache. ³ End-to-end through Lemmatizer's result cache, measured over the German UD-HDT treebank (3.5M tokens, 200k unique). The gap shrinks toward parity on smaller texts whose vocabulary fits the cache, and widens toward the uncached figure on large, low-repetition corpora.

To force a specific backend instead of relying on low_memory=True, pass it explicitly — DefaultStrategy(dictionary_factory=TrieDictionaryFactory()) or DefaultStrategy(dictionary_factory=StreamDictionaryFactory()), both importable from simplemma.strategies.dictionaries.

TrieDictionaryFactory needs the marisa-trie extra dependency (pip install simplemma[marisa-trie], available from version 1.1.0). The first use of a language builds its trie from the shipped dictionary — taking a few seconds and briefly using as much memory as the DefaultDictionaryFactory would — then caches it on disk for later invocations. If the machine running Simplemma doesn't have enough memory to build the trie, it can also be built on another machine with the same CPU architecture and the cache directory copied over.

StreamDictionaryFactory needs no extra dependency: it reads the shipped dictionary files directly instead of loading them into a Python dict, at the cost of much slower lookups (see the table above). There's no on-disk cache to warm up, so throughput is consistent from the first call.

Supported languages

The following languages are available, identified by their BCP 47 language tag, which typically corresponds to the ISO 639-1 code. If no such code exists, a ISO 639-3 code is used instead.

Available languages (2026-05-29):

The Forms column counts the inflected word forms stored in the dictionary, while Lemmata counts the distinct base forms they map to (both in thousands). A large gap between the two reflects rich morphology rather than a data error.

Code	Language	Forms (10³)	Lemm. (10³)	Acc.	Comments
ar	Arabic	297	49	0.77	on UD AR-PADT; real-world (unsegmented) input scores ≈0.74, see note below
ast	Asturian	154	36
bg	Bulgarian	139	18	0.85	on UD BG-BTB
ca	Catalan	640	63	0.89	on UD CA-AnCora
cs	Czech	355	44	0.91	on UD CS-FicTree
cy	Welsh	402	21	0.91	on UD CY-CCG
da	Danish	778	115	0.93	on UD DA-DDT, alternative: lemmy
de	German	1,115	334	0.95	on UD DE-GSD, see also German-NLP list
el	Greek	248	27	0.91	on UD EL-GDT
en	English	181	77	0.95	on UD EN-LinES, alternative: LemmInflect
enm	Middle English	43	6
eo	Esperanto	191	18	0.95	on UD EO-PraGo
es	Spanish	823	88	0.91	on UD ES-AnCora
et	Estonian	2,662	92	0.84	on UD ET-EWT, low coverage
fa	Persian	17	4	0.89	on UD FA-Seraji
fi	Finnish	3,547	125	0.86	on UD FI-FTB, see this benchmark
fr	French	248	37	0.93	on UD FR-Sequoia
ga	Irish	444	48	0.89	on UD GA-IDT
gd	Gaelic	72	15	0.84	on UD GD-ARCOSG
gl	Galician	426	43	0.88	on UD GL-CTG
grc	Ancient Greek	849	22	0.76	on UD GRC-PROIEL (best available; no general-register grc treebank exists)
gv	Manx	77	14	0.84	on UD GV-Cadhan
hbs	Serbo-Croatian	610	49	0.87	on UD HR-SET + SR-SET (token-weighted); Croatian and Serbian lists to be added later
he	Hebrew	104	10	0.88	on UD HE-HTB; real-world (unsegmented) input scores ≈0.77, see note below
hi	Hindi	53	11	0.93	on UD HI-HDTB
hu	Hungarian	492	36	0.85	on UD HU-Szeged
hy	Armenian	467	17	0.88	on UD HY-BSUT
id	Indonesian	21	4	0.93	on UD ID-CSUI
is	Icelandic	208	17	0.78	on UD IS-GC
it	Italian	357	28	0.93	on UD IT-ISDT
ka	Georgian	448	16	0.82	on UD KA-GLC
la	Latin	1,144	63	0.85	on UD LA-PROIEL
lb	Luxembourgish	306	79		only a <1k-token UD treebank available
lt	Lithuanian	365	28	0.84	on UD LT-ALKSNIS
lv	Latvian	177	14	0.78	on UD LV-LVTB
mk	Macedonian	551	39	0.73	on UD MK-MTB
ml	Malayalam	746	64	0.69	on UD ML-UFAL (small treebank), experimental
ms	Malay	17	4
nb	Norwegian (Bokmål)	633	138	0.81	on UD NO-Bokmaal
nl	Dutch	369	125	0.92	on UD NL-Alpino, excl. underscore-joined compound lemmas
nn	Norwegian (Nynorsk)	137	36	0.76	on UD NO-Nynorsk
pl	Polish	3,670	264	0.93	on UD PL-LFG
pt	Portuguese	926	95	0.92	on UD PT-GSD
ro	Romanian	342	36	0.92	on UD RO-RRT
ru	Russian	1,357	128	0.89	on UD RU-SynTagRus, alternative: pymorphy2
se	Northern Sámi	115	7	0.95	on UD SME-Giella
sk	Slovak	908	73	0.93	on UD SK-SNK
sl	Slovene	147	30	0.92	on UD SL-SSJ
sq	Albanian	96	10	0.72	on UD SQ-STAF
sv	Swedish	871	114	0.91	on UD SV-Talbanken, alternative: lemmy
sw	Swahili	4,869	4		experimental
tl	Tagalog	71	18	0.84	on UD TL-TRG
tr	Turkish	1,236	40	0.91	on UD TR-KeNet
uk	Ukrainian	502	35	0.90	on UD UK-IU, alternative: pymorphy2

Languages marked as having low coverage may be better suited to language-specific libraries, but Simplemma can still provide limited functionality. Where possible, open-source Python alternatives are referenced.

Experimental mentions indicate that the language remains untested or that there could be issues with the underlying data or lemmatization process.

The scores are calculated on Universal Dependencies treebanks on single word tokens (including some contractions but not merged prepositions), they describe to what extent simplemma can accurately map tokens to their lemma form. For each language the figure is the accuracy on its best-performing general-purpose treebank (parallel, spoken, learner, historical and other narrow-domain treebanks are excluded). The Dutch (nl) figure excludes gold lemmas that are underscore-joined compounds (e.g. klooster_orde), a UD-Alpino annotation convention that simplemma's single-token output cannot match; without that exclusion it is ≈0.88. Hebrew (he) and Arabic (ar) proclitics/articles fuse onto their host word in real (unsegmented) text but are scored as pre-split sub-tokens by the standard UD protocol above; on whole, unsegmented input their accuracy is ≈0.77 and ≈0.74 respectively. See the training/ folder of the code repository for more information.

This library is particularly relevant as regards the lemmatization of less frequent words. Its performance in this case is only incidentally captured by the benchmark above. In some languages, a fixed number of words such as pronouns can be further mapped by hand to enhance performance.

Speed

The following orders of magnitude are provided for reference only and were measured on an old laptop to establish a lower bound:

• Tokenization: > 1 million tokens/sec
• Lemmatization: > 250,000 words/sec

Using the most recent Python version (i.e. with pyenv) can make the package run faster.

Roadmap

• Add further lemmatization lists
• Grammatical categories as option
• Function as a meta-package?
• Integrate optional, more complex models?

Credits and licenses

The software is licensed under the MIT license. For information on the licenses of the linguistic information databases, see the licenses folder.

The surface lookups (non-greedy mode) rely on lemmatization lists derived from the following sources, listed in order of relative importance:

• Lemmatization lists by Michal Měchura (Open Database License)
• Wiktionary entries packaged by the Kaikki project
• FreeLing project
• spaCy lookups data
• Unimorph Project
• Wikinflection corpus by Eleni Metheniti (CC BY 4.0 License)

Contributions

This package has been first created and published by Adrien Barbaresi. It has then benefited from extensive refactoring by Juanjo Diaz (especially the new classes). See the full list of contributors to the repository.

Feel free to contribute, notably by filing issues for feedback, bug reports, or links to further lemmatization lists, rules and tests.

Contributions by pull requests ought to follow the following conventions: code style and linting with ruff, type hinting with mypy, included tests with pytest.

Running pytest after a plain pip install ".[dev]" skips the marisa-trie test module and under-reports coverage; install the extra as well (pip install ".[dev,marisa-trie]" or uv sync --extra dev --extra marisa-trie) to run the full suite and match CI's coverage numbers.

Other solutions

See lists: German-NLP and other awesome-NLP lists.

For another approach in Python see Spacy's edit tree lemmatizer.

References

To cite this software:

Barbaresi A. (year). Simplemma: a simple multilingual lemmatizer for Python [Computer software] (Version version number). Available from https://github.com/adbar/simplemma DOI: 10.5281/zenodo.4673264

This work draws from lexical analysis algorithms used in:

• Barbaresi, A., & Hein, K. (2017). Data-driven identification of German phrasal compounds. In International Conference on Text, Speech, and Dialogue Springer, pp. 192-200.
• Barbaresi, A. (2016). An unsupervised morphological criterion for discriminating similar languages. In 3rd Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2016), Association for Computational Linguistics, pp. 212-220.
• Barbaresi, A. (2016). Bootstrapped OCR error detection for a less-resourced language variant. In 13th Conference on Natural Language Processing (KONVENS 2016), pp. 21-26.