imago-mixed-eng-chi-tokenizer
Lemmize and tokenize string which contains Chinese and English words
Last updated a year ago by imago.ai .
Unlicense · Original npm · Tarball · package.json
$ cnpm install imago-mixed-eng-chi-tokenizer 
SYNC missed versions from official npm registry.

Mixed English and Chinese Tokenizer

Tokenizes string that contains Chinese and English words.

Optionally, lemmatizes English words. English contractions such as "don't" are always changed to full forms such as "do not".

Supports Cantonese / Taiwanese / Mandarin. Defaults to producing output in Traditional Chinese (see options below).

Example

const MEACT = require('imago-mixed-eng-chi-tokenizer');

let m = new MEACT();

(async() => {
	console.log(
		/* Produces ['hello', 'what', 'is', 'up', '边度', '有', '柜员机', '呀']: */
		await m.tokenize(`hello, what's up, 邊度有櫃員機呀?`),

		/* Produces ['hello', 'what', 'be', 'up', '边度', '有', '柜员机', '呀']: */
		await m.tokenize(`hello, what's up, 邊度有櫃員機呀?`),
	);

	m = null; // free up memory occupied by CEDICT
})();

Methods

async tokenize(text)

Returns an array of tokens from string. Punctuation is excluded.


let m = new MEACT();

let tokens = await m.tokenize('I am here'); // ['I', 'am', 'here'];

async lemmatize(text)

The same as tokenize(), except that English words are converted into lemmas.

For example, 'doing' will be changed to 'do'.


let m = new MEACT();

let tokens = await m.lemmatize('I am here'); // ['I', 'be', 'here'];

Options

The constructor takes an optional options object.

  • options.simplified - Boolean. Whether to use Simplified Chinese. Default is false and all output is forced to Traditional Chinese, even if it is in Simplified Chinese.
  • options.lemmaCache - Object. An object storing the cache of all English lemmas. Lemmatizing an English word is computationally expensive (50-400 ms). For that reason, you may want to use an object that will cache the lemmas. You could store that object globally or save it to disk. The code would look roughly like this:
let lemmaCache = readLemmaCacheFromDisk(); // assume you stored it somewhere

const m = new MEACT({ lemmaCache });
let lemmatized = m.lemmatize(text);

// lemmaCache has been updated with new lemmas, save it to disk. Next time the
// same sentence will be lemmatized much faster.
saveLemmaCacheToDisk(lemmaCache);

Current Tags

  • 1.0.0                                ...           latest (a year ago)

1 Versions

  • 1.0.0                                ...           a year ago
Maintainers (1)
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 0
Last Month 1
Dependencies (5)
Dev Dependencies (10)
Dependents (0)
None

Copyright 2014 - 2016 © taobao.org |