Custom token delimiter should be allowed not required for CJK texts

wzzhang
Wed, 29 Nov 2023 04:25:00 +0000

For Chinese/Japanese/Korean texts, there are no space between words. So it should be allowed to no input in the cutom delimiter on the Tier->Tokenize Tier dialogue. If no input, the texts should be tokenized character by character.
For your testing, provide some Chinese texts as below:
所以ELAN 软件应该是分析多模态 隐喻中不同模态的比较理想的工具。使用ELAN 之前,首先要对分析的模态进行赋码。