Language Identification Transform

February 27, 2025 · View on GitHub

The Language Identification transforms serves as a simple exemplar to demonstrate the development of a simple 1:1 transform.
Please see the set of transform project conventions for details on general project conventions, transform configuration, testing and IDE set up.

Summary

This transform will identify language of each text with confidence score with fasttext language identification model. ref

Contributors

Configuration and command line Options

The set of dictionary keys holding LangIdentificationTransform configuration for values are as follows:

Key nameDefaultDescription
lang_id_model_credentialunsetspecifies the credential you use to get model. This will be huggingface token. Guide to get huggingface token
lang_id_model_kindunsetspecifies what kind of model you want to use for language identification. Currently, only fasttext is available.
lang_id_model_urlunsetspecifies url that model locates. For fasttext, this will be repo nme of the model, like facebook/fasttext-language-identification
lang_id_content_column_namecontentsspecifies name of the column containing documents
lang_id_output_lang_column_namelangspecifies name of the output column to hold predicted language code
lang_id_output_score_column_namescorespecifies name of the output column to hold score of prediction

Running

Launched Command Line Options

The following command line arguments are available in addition to the options provided by the launcher.

  --lang_id_model_credential LANG_ID_MODEL_CREDENTIAL   the credential you use to get model. This will be huggingface token.
  --lang_id_model_kind LANG_ID_MODEL_KIND   what kind of model you want to use for language identification. Currently, only `fasttext` is available.
  --lang_id_model_url LANG_ID_MODEL_URL   url that model locates. For fasttext, this will be repo name of the model, like `facebook/fasttext-language-identification`
  --lang_id_content_column_name LANG_ID_CONTENT_COLUMN_NAME   A name of the column containing documents
  --lang_id_output_lang_column_name LANG_ID_OUTPUT_LANG_COLUMN_NAME   Column name to store identified language
  --lang_id_output_score_column_name LANG_ID_OUTPUT_SCORE_COLUMN_NAME   Column name to store the score of language identification

These correspond to the configuration keys described above.

Code example

Here is a sample notebook

Troubleshooting guide

For M1 Mac user, if you see following error during make command, error: command '/usr/bin/clang' failed with exit code 1, you should follow this step

Transforming data using the transform image

To use the transform image to transform your data, please refer to the running images quickstart, substituting the name of this transform image and runtime as appropriate.

Language Identification Ray Transform

Please see the set of transform project conventions for details on general project conventions, transform configuration, testing and IDE set up.

Summary

This project wraps the language identification transform with a Ray runtime.

Configuration and command line Options

Language Identification configuration and command line options are the same as for the base python transform.

Launched Command Line Options

In addition to those available to the transform as defined here, the set of launcher options are available.

Transforming data using the transform image

To use the transform image to transform your data, please refer to the running images quickstart, substituting the name of this transform image and runtime as appropriate.

Supported Languages

AfrikaansAlbanianAlemannicAmharicArabicAragoneseArmenianAsaro'o
AssameseAsturianAvaricAzerbaijaniBashkirBasqueBavarianBelarusian
BengaliBihari languagesBishnupriya ManipuriBosnianBretonBulgarianBuriatBurmese
CatalanCebuanoCentral BikolCentral KhmerCentral KurdishChavacanoChechenChinese
ChuvashCornishCorsicanCroatianCzechDanishDimliDivehi
DoteliDutchEastern MariEgyptian ArabicEmilian-RomagnolEnglishEsperantoEstonian
Fiji HindiFinnishFrenchGalicianGeorgianGermanGoan KonkaniGreek
GuaraniGujaratiHaitian CreoleHebrewHill MariHindiHungarianIcelandic
IdoIlokanoIndonesianInterlinguaInterlingueIrishItalianJapanese
JavaneseKalmykKannadaKapampanganKarachay-BalkarKazakhKomiKorean
KurdishKyrgyzLaoLatinLatvianLezgianLimburgishLithuanian
LojbanLombardLow GermanLower SorbianLuxembourgishMacedonianMaithiliMalagasy
MalayMalayalamMalteseManxMarathiMazandaraniMinMirandese
MongolianNahuatlNeapolitanNepal BhasaNepaliNorthern FrisianNorthern LuriNorwegian
Norwegian NynorskOccitanOdiaOssetianPalatinePashtoPersianPiedmontese
PolishPortuguesePunjabiQuechuaRomanianRomanshRussianRusyn
SanskritSardinianScotsScottish GaelicSerbianSerbo-CroatianSicilianSindhi
SinhalaSlovakSlovenianSomaliSouth AzerbaijaniSpanishSundaneseSwahili
SwedishTagalogTajikTamilTatarTeluguThaiTibetan
TurkishTurkmenTuvanUighurUkrainianUpper SorbianUrduUzbek
VenetianVepsVietnameseVolapükWalloonWarayWelshWest Flemish
Western FrisianWestern PunjabiWu ChineseYakutYiddishYorubaYue