Running
š
SIG on Iranian languages
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
sigiranic's activity
kargaranamirĀ
updated
a
Space
1 day ago
kargaranamirĀ
published
a
Space
1 day ago
kargaranamirĀ
authored
3
papers
3 months ago
How Transliterations Improve Crosslingual Alignment
Paper
ā¢
2409.17326
ā¢
Published
ā¢
1
GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages
Paper
ā¢
2410.23825
ā¢
Published
ā¢
3
MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment
Paper
ā¢
2410.05873
ā¢
Published
ā¢
3
kargaranamirĀ
posted
an
update
8 months ago
Post
1199
Introducing GlotCC: a new 2TB corpus based on an early 2024 CommonCrawl snapshot with data for 1000+ languages.
š¤ corpus v1: cis-lmu/GlotCC-V1
š± pipeline v3: https://github.com/cisnlp/GlotCC
More details? Stay tuned for our upcoming paper.
More data? In the next version, we plan to include additional snapshots of CommonCrawl.
Limitation: Due to the lower frequency of low-resource languages compared to others, there are sometimes only a few sentences available for very low-resource languages. However, the data volume for English in this version stands at 750GB, and the top 200 languages still have a strong presence in our data (see plot attached; we write the index for every 20 languages, meaning the 10th index is the 200th language).
š¤ corpus v1: cis-lmu/GlotCC-V1
š± pipeline v3: https://github.com/cisnlp/GlotCC
More details? Stay tuned for our upcoming paper.
More data? In the next version, we plan to include additional snapshots of CommonCrawl.
Limitation: Due to the lower frequency of low-resource languages compared to others, there are sometimes only a few sentences available for very low-resource languages. However, the data volume for English in this version stands at 750GB, and the top 200 languages still have a strong presence in our data (see plot attached; we write the index for every 20 languages, meaning the 10th index is the 200th language).
kargaranamirĀ
posted
an
update
12 months ago
Post
A Text Language Identification Model with Support for +2000 Labels:
space: cis-lmu/glotlid-space
model: cis-lmu/glotlid
github: https://github.com/cisnlp/GlotLID
paper: GlotLID: Language Identification for Low-Resource Languages (2310.16248)
space: cis-lmu/glotlid-space
model: cis-lmu/glotlid
github: https://github.com/cisnlp/GlotLID
paper: GlotLID: Language Identification for Low-Resource Languages (2310.16248)