PYTHON token_get_all

(11)

Python replacement for PHP's token_get_all [ edit ]

Do you know a Python replacement for PHP's token_get_all ? Write it!

PHP token_get_all

PHP original manual for token_get_all [ show | php.net ]

token_get_all

(PHP 4 >= 4.2.0, PHP 5)

token_get_all — Split given source into PHP tokens

Description

array token_get_all ( string $source )

token_get_all() parses the given source string into PHP language tokens using the Zend engine's lexical scanner.

For a list of parser tokens, see List of Parser Tokens, or use token_name() to translate a token value into its string representation.

Parameters

source: The PHP source to parse.

Return Values

An array of token identifiers. Each individual token identifier is either a single character (i.e.: ;, ., >, !, etc...), or a three element array containing the token index in element 0, the string content of the original token in element 1 and the line number in element 2.

Examples

Example #1 token_get_all() examples


<?php
$tokens = token_get_all('<?php echo; ?>'); /* => array(
                                                  array(T_OPEN_TAG, '<?php'), 
                                                  array(T_ECHO, 'echo'),
                                                  ';',
                                                  array(T_CLOSE_TAG, '?>') ); */

/* Note in the following example that the string is parsed as T_INLINE_HTML
   rather than the otherwise expected T_COMMENT (T_ML_COMMENT in PHP <5).
   This is because no open/close tags were used in the "code" provided.
   This would be equivalent to putting a comment outside of <?php ?> tags in a normal file. */
$tokens = token_get_all('/* comment */'); // => array(array(T_INLINE_HTML, '/* comment */'));
?>