Custom token delimiter should be allowed not required for CJK texts

hasloe
Wed, 29 Nov 2023 13:00:25 +0000

Hello,
Thanks for pointing this out; it’s true that currently there’s no way to tokenize CJK text character by character. We’ll add this to the request list.

-Han