node-normalizer
Normalize and clean text
Last updated 4 years ago by silentrob .
MIT · Original npm · Tarball · package.json
$ cnpm install node-normalizer 
SYNC missed versions from official npm registry.

Normalize, clean and fix text

npm install node-normalizer

The simple app processes input and tries to make it consumable for a bot.

The order in which the processing happes is important.

  • <xxx means sentence start then xxx
    1. spelling corrections for common spelling errors
    1. idiom conversions
    1. junk word removal from sentence
    1. special sentence effects (question, exclamation, revert question)
    1. abbreviation expansion and canonization
  • for abbreviations, do not use _ before the .
  • for apostrophied left side, must follow tokenizing conventions
  • for apostrophied right side, it means do not spell check the word, the apostrophe will disappear
  • Format is left phrase separated by _ yields right phrase separated by +
  • if right side is %value means set that bit on the sentence (%EXCLAMATIONMARK %QUESTIONMARK)
  • if right side is a ~word its an interjection
  • only proper names should have capital letters
  • Right phrase missing means delete left phrase
  • Substitutions files include:
  • we use + because we dont want the resulting phrase recognized by the idiom processor and thus cause the processor to delete the phrase
  • xxx> means sentence then end stop
  • if you want to have the result NOT tokenized, put it in quotes

Current Tags

  • 1.0.0-alpha4                                ...           alpha (4 years ago)
  • 0.2.0                                ...           latest (4 years ago)

21 Versions

  • 1.0.0-alpha4                                ...           4 years ago
  • 1.0.0-alpha3                                ...           4 years ago
  • 1.0.0-alpha2                                ...           4 years ago
  • 1.0.0-alpha1                                ...           4 years ago
  • 0.2.0                                ...           4 years ago
  • 0.1.4                                ...           5 years ago
  • 0.1.3                                ...           5 years ago
  • 0.1.2                                ...           5 years ago
  • 0.1.1                                ...           5 years ago
  • 0.1.0                                ...           5 years ago
  • 0.0.11                                ...           6 years ago
  • 0.0.10                                ...           6 years ago
  • 0.0.9                                ...           6 years ago
  • 0.0.8                                ...           6 years ago
  • 0.0.7                                ...           6 years ago
  • 0.0.6                                ...           6 years ago
  • 0.0.5                                ...           6 years ago
  • 0.0.4                                ...           6 years ago
  • 0.0.3                                ...           6 years ago
  • 0.0.2                                ...           6 years ago
  • 0.0.1                                ...           6 years ago
Maintainers (2)
Downloads
Today 0
This Week 0
This Month 0
Last Day 0
Last Week 0
Last Month 21
Dependencies (4)
Dev Dependencies (2)

Copyright 2014 - 2016 © taobao.org |