Compare model versions

Diff

Before

**GlotLID** is a Fasttext language identification (LID) model that supports more than **2000 labels**.


**Latest:** GlotLID is now updated to **V3**. V3 supports **2102 labels** (three-letter ISO codes with script). For more details on the supported languages and performance, as well as significant changes from previous versions, please refer to [https://github.com/cisnlp/GlotLID/blob/main/languages-v3.md](https://github.com/cisnlp/GlotLID/blob/main/languages-v3.md).


- **Demo:** [huggingface](https://huggingface.co/spaces/cis-lmu/glotlid-space)
- **Repository:** [github](https://github.com/cisnlp/GlotLID)
- **Paper:** [paper](https://arxiv.org/abs/2310.16248) (EMNLP 2023)
- **Point of Contact:** amir@cis.lmu.de

After

**GlotLID** is a Fasttext language identification (LID) model that supports more than **2000 labels**.


**Latest:** GlotLID **v4 alpha** (`model_v4_alpha.bin`) is out as a pre-release of the next version: **2056 labels**, a smaller model (783 MB instead of 1.69 GB), trained with [fastText-turbo](https://github.com/CommonGlot/fastText-turbo). It is not final yet; its labels and scores may still change, so `model.bin` stays v3 for now. See [versions](https://github.com/cisnlp/GlotLID#versions).

**Stable:** GlotLID **V3** supports **2102 labels** (three-letter ISO codes with script). For more details on the supported languages and performance, as well as significant changes from previous versions, please refer to [https://github.com/cisnlp/GlotLID/blob/main/languages-v3.md](https://github.com/cisnlp/GlotLID/blob/main/languages-v3.md).


- **Demo:** [huggingface](https://huggingface.co/spaces/cis-lmu/glotlid-space) (language identification, and code-switching with [MaskLID](https://github.com/cisnlp/MaskLID))
- **Repository:** [github](https://github.com/cisnlp/GlotLID)
- **Paper:** [paper](https://arxiv.org/abs/2310.16248) (EMNLP 2023)
- **Point of Contact:** amir@cis.lmu.de

Diff

Before

Here is how to use this model to detect the language of a given text:

```python
>>> import fasttext
>>> from huggingface_hub import hf_hub_download

# model.bin is the latest version always
>>> model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin")
>>> model = fasttext.load_model(model_path)
>>> model.predict("Hello, world!")
```

If you are not a fan of huggingface_hub, then download the model directyly:

```python
>>> ! wget https://huggingface.co/cis-lmu/glotlid/resolve/main/model.bin
```

```python
>>> import fasttext

>>> model = fasttext.load_model("/path/to/model.bin")
>>> model.predict("Hello, world!")
```

After

Here is how to use this model to detect the language of a given text:

```python
# pip install fasttext-turbo huggingface_hub
>>> import fasttext
>>> from huggingface_hub import hf_hub_download

# model.bin is the latest version always
>>> model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin")
>>> model = fasttext.load_model(model_path)
>>> model.predict("Hello, world!")
```

We recommend [fastText-turbo](https://github.com/CommonGlot/fastText-turbo) (`pip install fasttext-turbo`): it is a drop-in replacement for fastText (same `import fasttext`, same model files and API, prebuilt wheels) that predicts with GlotLID about 3 times faster than the original fastText in our tests (2.7x with v4 alpha, 3.4-4.3x with v3), with the same predictions. The original `fasttext` package works too.

To use the v4 pre-release, download `model_v4_alpha.bin` instead of `model.bin`.

To limit the prediction to a smaller set of languages without training another model, use [CustomLID](https://github.com/cisnlp/GlotLID/blob/main/assets/inference/customlid.py) (see the [GitHub README](https://github.com/cisnlp/GlotLID#limit-the-model-to-a-smaller-set-of-languages-python)).

If you are not a fan of huggingface_hub, then download the model directly:

```python
>>> ! wget https://huggingface.co/cis-lmu/glotlid/resolve/main/model.bin
```

```python
>>> import fasttext

>>> model = fasttext.load_model("/path/to/model.bin")
>>> model.predict("Hello, world!")
```

Diff

Before

We always maintain the previous version of GlotLID in our repository.

To access a specific version, simply append the version number to the `filename`.

- For v1: `model_v1.bin` (introduced in the GlotLID [paper](https://arxiv.org/abs/2310.16248) and used in all experiments).
- For v2: `model_v2.bin` (an edited version of v1, featuring more languages, and cleaned from noisy corpora based on the analysis of v1).
- For v3: `model_v3.bin` (an edited version of v2, featuring more languages, excluding macro languages, further cleaned from noisy corpora and incorrect metadata labels based on the analysis of v2, supporting "zxx" and "und" series labels)


`model.bin` always refers to the latest version (v3).

After

We always maintain the previous version of GlotLID in our repository.

To access a specific version, simply append the version number to the `filename`.

- For v1: `model_v1.bin` (introduced in the GlotLID [paper](https://arxiv.org/abs/2310.16248) and used in all experiments).
- For v2: `model_v2.bin` (an edited version of v1, featuring more languages, and cleaned from noisy corpora based on the analysis of v1).
- For v3: `model_v3.bin` (an edited version of v2, featuring more languages, excluding macro languages, further cleaned from noisy corpora and incorrect metadata labels based on the analysis of v2, supporting "zxx" and "und" series labels)
- For v4 alpha: `model_v4_alpha.bin` (a pre-release of v4; not final, its labels and scores may still change). It supports 2056 labels (1899 three-letter ISO codes, with the "zxx" and "und" series). Trained with [fastText-turbo](https://github.com/CommonGlot/fastText-turbo); it loads in fastText and fastText-turbo like the previous versions. It is smaller (96-dimensional, 783 MB) and uses character n-grams of 2 to 5 with word n-grams of 1, so nothing is shared between neighbouring words: the scores of a text add up from the scores of its words, which makes the per-word scores that [MaskLID](https://github.com/cisnlp/MaskLID) uses for code-switching exact.


`model.bin` always refers to the latest stable version (v3).