Doctrine Lexer is a PHP library that provides a base class for creating lexical scanners. It's primarily used for tokenizing strings into smaller parts, which is useful in parsing and analyzing text or code. The lexer is designed to be extended for specific use cases.

Pros

• Simple and lightweight implementation
• Easily extendable for custom lexing needs
• Well-integrated with other Doctrine projects
• Provides a solid foundation for building more complex parsers

Cons

• Limited built-in token types
• Requires additional implementation for specific use cases
• May be overkill for very simple tokenization tasks
• Documentation could be more comprehensive

Code Examples

1. Basic usage of the Lexer class:

use Doctrine\Common\Lexer\AbstractLexer;

class MyLexer extends AbstractLexer
{
    protected function getCatchablePatterns()
    {
        return [
            '[a-z]+',
            '\d+',
            '\s+',
        ];
    }

    protected function getNonCatchablePatterns()
    {
        return [];
    }

    protected function getType(&$value)
    {
        if (is_numeric($value)) {
            return 'NUMBER';
        }
        if (ctype_alpha($value)) {
            return 'WORD';
        }
        if (ctype_space($value)) {
            return 'WHITESPACE';
        }
        return 'UNKNOWN';
    }
}

$lexer = new MyLexer();
$lexer->setInput('hello 123 world');
foreach ($lexer as $token) {
    echo $token['type'] . ': ' . $token['value'] . PHP_EOL;
}

2. Using the lexer to tokenize a simple mathematical expression:

class MathLexer extends AbstractLexer
{
    const T_NUMBER = 1;
    const T_OPERATOR = 2;

    protected function getCatchablePatterns()
    {
        return [
            '\d+',
            '[+\-*/]',
        ];
    }

    protected function getNonCatchablePatterns()
    {
        return ['\s+'];
    }

    protected function getType(&$value)
    {
        if (is_numeric($value)) {
            return self::T_NUMBER;
        }
        if (in_array($value, ['+', '-', '*', '/'])) {
            return self::T_OPERATOR;
        }
        return null;
    }
}

$lexer = new MathLexer();
$lexer->setInput('5 + 3 * 2');
foreach ($lexer as $token) {
    echo $token['type'] === MathLexer::T_NUMBER ? 'NUMBER' : 'OPERATOR';
    echo ': ' . $token['value'] . PHP_EOL;
}

Getting Started

To use Doctrine Lexer in your project, first install it via Composer:

composer require doctrine/lexer

Then, create a custom lexer class by extending Doctrine\Common\Lexer\AbstractLexer:

use Doctrine\Common\Lexer\AbstractLexer;

class CustomLexer extends AbstractLexer
{
    protected function getCatchablePatterns()
    {
        return [
            // Define your patterns here
        ];
    }

    protected function getNonCatchablePatterns()
    {
        return [
            // Define patterns to ignore here
        ];
    }

    protected function getType(&$value)
    {
        // Define your token types here
    }
}

Finally, instantiate your custom lexer and use it to tokenize input:

$lexer = new CustomLexer();
$lexer->setInput('your input string');
foreach ($lexer as $token) {
    // Process tokens
}