Sudachi Language Resources
Japanese dictionaries and pre-trained models (word embeddings and language models) for natural language processing. SudachiDict is the dictionary for a Japanese tokenizer (morphological analyzer) Sudachi. chiVe is Japanese pretrained word embeddings (word vectors), trained using the ultra-large-scale web corpus NWJC by National Institute for Japanese Language and Linguistics, analyzed by Sudachi. chiTra is a library for using large-scale pre-trained language models with the Japanese tokenizer SudachiPy. Managed by Works Applications. Updated the dictionaries are updated every few months to include neologism and fixes for the existing words.. Tags: natural language processing.
- Category
- Natural Language Processing
- Provider
- Works Applications
- License
- Apache-2.0
- Hosted on
- Amazon S3, bucket sudachi, ap-northeast-1
- Access
- Free and public. No account with the data provider is needed.
- Source
- https://registry.opendata.aws/roda/sudachi/
Data structure and loading code
Files
Example file: chitra/nlp2022/
Path pattern: {id}/{seg1}/{object}
Analyze this dataset without downloading it
On Scigantic, Sudachi Language Resources opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
Related Natural Language Processing datasets
- ExploreToM
- Shopping Humor Generation
- Common Screens
- Phrase Clustering Dataset (PCD)
- Nemotron Instruction Following Chat V 1
- Low Context Name Entity Recognition (NER) Datasets with Gazetteer
- Provision of Web-Scale Parallel Corpora for Official European Languages (ParaCrawl)
- Multilingual Name Entity Recognition (NER) Datasets with Gazetteer