hx-tokenizer
Split text into tokens.
Last updated 4 years ago by pomber .
MIT · Repository · Bugs · Original npm · Tarball · package.json
$ cnpm install hx-tokenizer 
SYNC missed versions from official npm registry.

tokenizer Build Status npm version XO code style

Split text into tokens.

Install

$ npm install --save hx-tokenizer

Usage

const tokenizer = require("hx-tokenizer");
const tokens = tokenizer.tokenize("Lorem ipsum, something.");
// tokens == ["Lorem", "ipsum", ",", "something", "."]
const text = tokenizer.join(tokens);
// text == "Lorem ipsum, something."

API

tokenizer.tokenize(text)

tokenizer.join(tokens)

License

MIT © Hexacta

Current Tags

  • 1.0.0                                ...           latest (4 years ago)

2 Versions

  • 1.0.0                                ...           4 years ago
  • 0.1.0                                ...           4 years ago
Maintainers (1)
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 0
Last Month 0
Dependencies (0)
None
Dev Dependencies (1)
Dependents (0)
None

Copyright 2014 - 2016 © taobao.org |