Language Version : | ActionScript 3.0 |
Runtime Versions : | AIR 1.0, Flash Player 9.x |
The TextTokenizer
class locates the boundaries of words in a
block of text.
Word boundary locations are found according to these general principles:
- Be able to tokenize a block of text specified by start and end positions
- Default separator is Unicode white space character. Also break on newlines
- Tokens consist of either words or numbers in which case it may include commas, etc..
- Apostrophes or hyphens within a word are kept with the word
- Punctuation, spaces and other characters that are not part of a token, are broken out separately
In the future versions, this class would also provide a way for the developers to customize the separators used by the tokenizer.
ignoredSeparators:Vector.<int>
Language Version : | ActionScript 3.0 |
Runtime Versions : | AIR 1.0, Flash Player 10 |
Get all of ignored separators used by this tokenizer class.
A vector of int containing all of ignored separators code point which are used by this class.
Implementation public function get ignoredSeparators():Vector.<int>
public function set ignoredSeparators(value:Vector.<int>):void
public function TextTokenizer(textHolder:String, startIndex:int = 0, endIndex:int)
Language Version : | ActionScript 3.0 |
Runtime Versions : | AIR 1.0, Flash Player 10 |
The tokenizer for a String object.
This class implements the ITokenizer interface.
Constructs a new TextTokenizer object to break String to words by creating with a new piece of text.
Parameters | textHolder:String — A String object to hold the text which will be processed by this tokenizer.
|
|
| startIndex:int (default = 0 ) — A int type input to hold the starting index of input text should be scanned.
|
|
| endIndex:int (default = NaN ) — A int type input to hold the ending index of input text should be scanned.
|
public function getFirstToken():Token
Language Version : | ActionScript 3.0 |
Runtime Versions : | AIR 1.0, Flash Player 10 |
Return the first word in the text being scanned.
NOTE: In a special case when there are no valid tokens in text, it returns a pseudo token having first and last index set to int.MAX_VALUE. As a result firstToken().first
equals int.MAX_VALUE and firstToken().last
equals int.MAX_VALUE.
Returns public function getNextToken(token:Token):Token
Language Version : | ActionScript 3.0 |
Runtime Versions : | AIR 1.0, Flash Player 10 |
Determine the next word following the current token.
Returns the token of the next word.
NOTE: When there are no more valid tokens, it returns a pseudo token having first and last index set to int.MAX_VALUE. As a result getNextToken().first
equals int.MAX_VALUE and getNextToken().last
equals int.MAX_VALUE.
Parameters
| token:Token — A Token object to be used for determining next word.
|
Returns public function getPreviousToken(token:Token):Token
Language Version : | ActionScript 3.0 |
Runtime Versions : | AIR 1.0, Flash Player 10 |
Determine the word preceding the current token.
Returns the token of the previous word or getFirstToken
object if there is no preceding word.
Parameters
| token:Token — A Token object to be used for determining previous word.
|
Returns
Fri Oct 24 2014, 03:33 PM +11:00