First Release of Cantonese Text Pre-processing App

This describes a professional (and experimental) tool included with the Lab Pass.

Cantonese Fonts 2 pushed far beyond what was deemed possible with a font. But it remains a font, and is still constraint inside the font renderer. Clear constraint is a good thing, however, this means operations needing database access or (expensive) computations are out of bounds.

One of the linguistic difficulties in working with written Chinese is that it’s a sequence of characters with no breaks to mark where one word ends and another begins. It may surprise you, but this is contributes about 60% of Jyutping errors from the Cantonese Font. (Another 30% comes from missing/incorrect context, and 10% is from line-breaks.)

Take as an example the sentence 林平之支點讀筆露出生銹鋼筋. When rendered with the Cantonese Font, it shows with two errors (highlighted in red):

The first is an example of a missing context: 林平之 is a proper noun (name of a fictional character) and we could not squeeze millions of possible names into a font. The second is from incorrect segmentation. The font is unable to distinguish between 露 出生 銹 鋼筋 (where it would be sang1) and 露出 生銹 鋼筋 (where it would be saang1). It applied the rules corresponding to the first interpretation, and missed. The writer intended the second meaning.

I anticipated this issue, and built in a mechanism to break up the character sequences. Adding a | marker (which is itself invisible) helps the Font “know” there is a word boundary. So if we give the font 林平之支點讀筆露出|生銹鋼筋, the rendering will be correct:

If you do this consistently, then you should be able to improve the accuracy from 99.7% to 99.9%. More dogs more happiness. (好爛呀 🙈)

However, the effort required to “do this consistently” is considerable. To put in real numbers, a typical 5,000 characters article may have 15 Jyutping errors of which 10 stems from incorrect segmentation. Segmenting 5,000 characters is no small task; you might as well just go find and correct all 15 errors!

What if there is a one-click solution? 🙃

The Text Preprocessor app, found at this page 🔗, is this one-click solution. It’s simple but mighty:¹

You simply paste the text into the Text to pre-process box, and hit the segment button. Behind the scenes, we apply state-of-the-art Hong Kong Cantonese / Traditional Chinese-specific segmentation. What if you are segmenting a story with a character you invented, and thus her name cannot be recognized? Just add the entry to the custom words and the app handles the rest.

Remember the | symbols are invisible? They don’t show up, and the Copy button (shows up when hovering over the text box) gives you exactly what you need to paste. The outcome is that it all just works at better accuracy, no matter how long your text is.

In fact, the longer the text, the more valuable this becomes. Not just because it saves you more work, but because the app also tallies the frequencies of the segments for you in the background. This is just not displayed yet (as I work out how it should work while keeping an uncluttered interface). What this means for you, as a user, is that when you pre-process a whole story that you wrote, you get the glossary (thus Anki decks) for free.

This is the first iteration of the first instance of supporting tools for doing professional work with Cantonese. In the future, this pre-processor is expected to grow to:

1. help you automatically annotate proper names that are not in the font. Canto Font was built to work with 鄧小平, but not 林平之 or 鄧月平. The tool will fill in these gaps.
2. apply pronunciation styles across the whole text. Canto Font is tuned to be expressive towards vernacular Cantonese 白話; this means a 97% accuracy when working with, say, Tang dynasty poems. Many of these changes are rule-based, for example, that 名 is pronounced as ming4 instead of meng2. Ask for this style, and let the pre-process tool do all the annotations for you.

This may, in time, integrate with the “copy-as-PNG” app, the self-serve typesetting pipeline, and the CJK-to-Jyutping tools (all of which are in early stages). You can let me know in the Forums of your comments or future wish-lists after trying it out.

1. Cantonese segmentation is non-trivial. If you think harder about the problem, it’s often not even clear what a word is. How many words should 此地無銀 be? The tool is far from perfect, but at about 80% accuracy, it approximates what a native speaker would do without specialist training. ↩︎