Sudachi Language Resources

Japanese dictionaries and pre-trained models (word embeddings and language models) for natural language processing. SudachiDict is the dictionary for a Japanese tokenizer (morphological analyzer) Sudachi. chiVe is Japanese pretrained word embeddings (word vectors), trained using the ultra-large-scale web corpus NWJC by National Institute for Japanese Language and Linguistics, analyzed by Sudachi. chiTra is a library for using large-scale pre-trained language models with the Japanese tokenizer SudachiPy. Managed by Works Applications. Updated the dictionaries are updated every few months to include neologism and fixes for the existing words.. Tags: natural language processing.

Category
Natural Language Processing
Provider
Works Applications
License
Apache-2.0
Hosted on
Amazon S3, bucket sudachi, ap-northeast-1
Access
Free and public. No account with the data provider is needed.
Source
https://registry.opendata.aws/roda/sudachi/
Data structure and loading code

Files

Example file: chitra/nlp2022/

Path pattern: {id}/{seg1}/{object}

Analyze this dataset without downloading it

On Scigantic, Sudachi Language Resources opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

Related Natural Language Processing datasets