How to enable multi-line matching in custom tokens provider

Viewed 285

I have a custom Monarch tokenizer for function names in Monaco that works well when the open paren appears on the same line:

But the tokenizer fails to correctly identify the function name if the open paren appears on a separate line:

Note that the foo function on line 1 is correctly highlighted, but the tokenizer gets lost when it comes to the foo function on line 5, so it highlights it as "invalid" (the default). (For color-blind readers, sorry about the green/red coloring... it's the theme default.)

The tokenizer uses regexes, but it seems like nothing I can do will force the regex to span more than the current line. Is there some way to instruct Monaco's tokenizer to use multi-line matching, or is this just a known limitation? I can't find any mention of it in the documentation, one way or the other.

Here's a simple repro you can copy/paste into the Monaco Editor Playground:

// Register a new language
monaco.languages.register({ id: 'mySpecialLanguage' });

// Register a tokens provider for the language
monaco.languages.setMonarchTokensProvider('mySpecialLanguage', {
    defaultToken: "invalid",
    tokenizer: {
        root: [
            [
                /(foo)([\s\n\r]*)([(])/,
                ["type.function", "", "@brackets"],
            ],

            [
                /"/,
                {
                    token: "string.quote",
                    bracket: "@open",
                    next: "@string",
                },
            ],

            [
                /[()\[\]]/,
                "@brackets",
            ],
        ],

        string: [
            [
                /[^"]+/,
                "string",
            ],
            [
                /"/,
                {
                    token: "string.quote",
                    bracket: "@close",
                    next: "@pop",
                },
            ],
        ],
    }
});

monaco.editor.create(document.getElementById("container"), {
    theme: 'vs',
    value: 'foo\t(\n\t\t"bar"\n\t)\n\nfoo\n\t(\n\t\t"bar"\n\t)\n',
    language: 'mySpecialLanguage'
});
0 Answers
Related