NLP for Punjabi
August 7, 2020 ยท View on GitHub
This repository contains State of the Art Language models and Classifier for Punjabi language (spoken in Indian sub-continent)
The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)
Dataset
Created as part of this project
Open Source Datasets
Results
Language Model Perplexity (on validation set)
| Architecture/Dataset | Punjabi Wikipedia Articles |
|---|---|
| ULMFiT | 24.40 |
| TransformerXL | 14.03 |
Classification Metrics
ULMFiT
| Dataset | Accuracy | MCC | Notebook to Reproduce results |
|---|---|---|---|
| IndicNLP News Article Classification Dataset - Punjabi | 97.12 | 96.17 | Link |
Visualizations
Word Embeddings
| Architecture | Visualization |
|---|---|
| ULMFiT | Embeddings projection |
| TransformerXL | Embeddings projection |
Sentence Embeddings
| Architecture | Visualization |
|---|---|
| ULMFiT | Encodings projection |
Pretrained Models
Language Models
Download pretrained Language Models from here
Tokenizer
Unsupervised training using Google's sentencepiece
Download the trained model and vocabulary from here