diff --git a/docs/docs/tokenizer/thai_tokenizer.md b/docs/docs/tokenizer/thai_tokenizer.md index 5186034..efbef7e 100644 --- a/docs/docs/tokenizer/thai_tokenizer.md +++ b/docs/docs/tokenizer/thai_tokenizer.md @@ -1,6 +1,13 @@ The `ThaiTokenizer` is a Rasa compatible tokenizer for Thai, using [`PyThaiNLP`](https://github.com/PyThaiNLP/pythainlp) under the hood. -In order to use the `ThaiTokenizer` the language **must** be set to `th` - no other languages are supported by this tokenizer. +In order to use the `ThaiTokenizer` the language **must** be set to `th` - no +other languages are supported by this tokenizer. + +## Configurable Variables + +None + +## Base Usage The `ThaiTokenizer` can be used in a Rasa configuration like below: @@ -15,4 +22,6 @@ pipeline: max_ngram: 4 - name: DIETClassifier epochs: 100 -``` \ No newline at end of file +``` + +If there are any issues with this tokenizer, please [let us know](https://github.com/RasaHQ/rasa-nlu-examples/issues). diff --git a/docs/index.md b/docs/index.md index dc799cc..8ec9acd 100644 --- a/docs/index.md +++ b/docs/index.md @@ -24,6 +24,13 @@ The following components are implemented. This component will print what each featurizer adds to the NLU message. Very useful for debugging. You can find a tutorial on it [here](https://blog.rasa.com/custom-printer-component/). +### Tokenizers + +#### ThaiTokenizer [docs](docs/tokenizer/thai_tokenizer/) + +We support a Thai tokenizier based on PyThaiNLP [link](https://github.com/PyThaiNLP/pythainlp). + + ### Dense Featurizers #### FastTextFeaturizer diff --git a/setup.py b/setup.py index 4f45da9..af3f62e 100644 --- a/setup.py +++ b/setup.py @@ -23,7 +23,7 @@ setup( name="rasa_nlu_examples", - version="0.1.1", + version="0.1.2", packages=find_packages(exclude=["notebooks"]), install_requires=base_packages, extras_require={"dev": dev_packages},