18 MB

Ctrl+K

2 contributors

History: 5 commits

DongHyeok-Seo

Record verified publication

d57c43f about 2 months ago

provenance
Record verified publication about 2 months ago
reports
Release KorByte-128K v2 tokenizer about 2 months ago
src
Release KorByte-128K v2 tokenizer about 2 months ago
tests
Release KorByte-128K v2 tokenizer about 2 months ago
.gitattributes

1.57 kB
Publish KorByte-128K tokenizer and reproducibility evidence about 2 months ago
.gitignore

143 Bytes
Publish KorByte-128K tokenizer and reproducibility evidence about 2 months ago
.python-version

5 Bytes
Publish KorByte-128K tokenizer and reproducibility evidence about 2 months ago
DATA_SOURCES.md

1.33 kB
Release KorByte-128K v2 tokenizer about 2 months ago
LICENSE

11.4 kB
Publish KorByte-128K tokenizer and reproducibility evidence about 2 months ago
NOTICE

264 Bytes
Publish KorByte-128K tokenizer and reproducibility evidence about 2 months ago
README.md

3.83 kB
Release KorByte-128K v2 tokenizer about 2 months ago
added_tokens.json

7.64 kB
Publish KorByte-128K tokenizer and reproducibility evidence about 2 months ago
merges.txt

2.45 MB
Release KorByte-128K v2 tokenizer about 2 months ago
pyproject.toml

936 Bytes
Release KorByte-128K v2 tokenizer about 2 months ago
special_tokens_map.json

6.18 kB
Publish KorByte-128K tokenizer and reproducibility evidence about 2 months ago
tokenizer.json

11.7 MB
xet

Release KorByte-128K v2 tokenizer about 2 months ago
tokenizer_config.json

6.35 kB
Publish KorByte-128K tokenizer and reproducibility evidence about 2 months ago
uv.lock

326 kB
Release KorByte-128K v2 tokenizer about 2 months ago
vocab.json

3.36 MB
Release KorByte-128K v2 tokenizer about 2 months ago