How to update semantic tokens as the visible range of the editor changes? (VS Code Extension)

user_13659134 0
Viewed 122

I have written a Semantic Tokens Provider for my VS Code Extension, and it is causing issues when large files are opened (e.g. files with one million lines will crash the Extension Host).

This is happening because, to parse the active editor's text into tokens, every time there's a change in the text, the provider loops through each line one by one. This means that the smaller the number of lines of text, the faster the extension will provide semantic highlighting. Thus, one million lines is too much for the extension to handle, and, as a result, the Extension Host gets terminated:

Extension Host Error

To solve this issue, my idea was to check the editor's visible range as it changes when the user scrolls up and down, and update the document semantic tokens. This way, on each change of the visible range, there would be a different range of lines parsed, and thus, different tokens provided, in order to make this operation more efficient.

So, I know I can do something like this to get the top and bottom lines of the visible range in the active editor:

const window = vscode.window
const textEditor = window.activeTextEditor;

if(textEditor) {
    window.onDidChangeTextEditorVisibleRanges(() => {
        const visibleRange = textEditor.visibleRanges[0];
        const topLineIndex = visibleRange.start.line;
        const bottomLineIndex = visibleRange.end.line;
        console.log(topLineIndex, bottomLineIndex);
    });
}

However, I do not know how to update tokens dynamically as the visible range changes, because I can't return a SemanticTokens instance on an event.

Is there a way to update the semantic tokens on an event? If not, does there exist a better and more efficient solution than just looping through each line to parse the text into tokens?

vscode-extensions visual-studio-code typescript
1 Answers

You can look into VSCode's built-in commands and specifically see "vscode.provideDocumentSemanticTokens" which looks like what you are looking for.

Now, is your parser the problem or the token provider? What you could do to optimize things is to only parse the lines which were edited instead of the whole file everytime the user changes the file.

If I am not wrong, semantic tokens are only updated when necessary, i.e when a portion of text inside of a token's range changes, so your parser might be to blame for the crashes instead.

user_102317000
Related