Tokenizer

String tokenizer component

usage

var Tokenizer = require('tokenizer');

var tokenizer = new Tokenizer();

tokenizer.tokenize('viva la pepa    ');
  //['viva', 'la', 'pepa']

Methods

Tokenizer#trim

Tokenizer#tokenize