Custom token delimiter should be allowed not required for CJK texts

hasloe
Fri, 01 Dec 2023 14:24:39 +0000

PS.

I had a look whether there are tokenizers for CJK languages available in the WebLicht infrastructure. One tokenizer produces output like this:

Not sure if this is useful, but if you are interested you could maybe first try a few texts in the web interface of WebLicht. You may have to request an account, if your institution is not listed as an identity provider.

If the online results still seem useful, I can assist in trying to run a so called toolchain from within ELAN.