VSCode: Tests won't proceed after using executeCommand to open a sample project

user_7361479 0
Viewed 475

I'm trying to add some unit tests to a Visual Studio Code extension I'm developing. I've followed the recipe for setting up extension testing described here: https://code.visualstudio.com/api/working-with-extensions/testing-extension

This recipe, however, doesn't show anything useful being done to actually test an extension, just the skeleton for getting set up to do so.

For the testing I want to do, one of the first things I want to do is open a sample project. And that's where I get stuck. Here is just one of many variations I've tried for opening a project folder:

import assert from 'assert';
import { after, before, it } from 'mocha';
import path from 'path';
import { commands, Extension, extensions, Uri, window } from 'vscode';

suite('Extension Tests', () => {
  let extension: Extension<any>;
  const projectFolder = Uri.file(path.join(__dirname, '../../../test/suite/sample-project'));

  window.showInformationMessage('Start all tests.');

  before(() => {
    extension = extensions.getExtension('kshetline.ligatures-limited');
    const cmd = commands.executeCommand('vscode.openFolder', projectFolder).then(
      () => console.log('opened'),
      () => console.log('didn\'t open'));
    console.log('before');
    return cmd;
  });

  after(() => {
    console.log('after');
  });

  it('should load and activate extension', () => {
    assert.ok(extension);
    assert.ok(extension.isActive);
  });

  it('second test', () => {
    assert.ok(true);
  });
});

If I take out the executeCommand, both tests run and the test suite properly terminates, with test window closing.

If I leave the executeCommand in, sometimes neither test is executed, sometimes just the first test. The test suite doesn't terminate -- the test window is left open, and I have to manually stop the test.

Variations I've tried:

  • Making the before function async, and awaiting executeCommand.
  • Adding a done parameter, for both async and non-async, and calling done() at the end of the before function, or in the then clause of the executeCommand promise.
  • Returning or not returning the executeCommand promise.

Examples of how to do this correctly are hard to come by. I looked at the repositories for one extension after another, and it seems doing tests on extensions isn't very popular. VSCode extension writers in general don't apparently bother with tests very often. (And maybe the trouble I'm having is the reason why?)

I can see that the sample project does indeed open, and I see no console errors to indicate why the testing gets stuck and fails to proceed any further.

vscode-extensions visual-studio-code typescript mocha.js unit-testing
2 Answers

Unfortunately, vscode reloads the window when you open or close a folder. This means the extension host is restarted and looses connection to both the debugger as well as the currently running test script. This is why your before function never returns and your tests are aborted mid-way.

There is an open issue on this behaviour, although not connected to testing.

The reason why the window reloads is, that switching workspaces may change the vscode environment. Settings may be overwritten, workspace trust changes, the entire state and cache of running extensions is invalidated and different extensions may be activated. Unfortunately that does not reload the tests passed into vscode by --extensionTestsPath, most likely to prevent infinite loops when your test code triggers a window or extension host reload.

Your approach of opening the folder on startup is the correct one until the issue is addressed. Although, if you are just doing Unit Tests you can just open single files and invoke the functions used by your commands / events directly. If you are doing end-to-end or user-like tests and have workspace-related activationEvents, you might not get around opening a folder.

I can't find any documentation confirming it, but in my experience your extension is enabled by default and you can invoke any commands directly, so there is usually no need to open a workspace.

Regarding the documentation for tests, I highly recommend following the guides on the vscode website. When you bootstrap your extension using yeoman as suggested there, you already have a test runner set up, including some sample code. It also generates a launch.json that can be used to debug your tests.

For more examples have a look into the built-in extensions for vscode (e.g. TypeScript Language Features) or some of microsofts own extensions. Most of them include tests with varying complexity. You'll probably find one that matches your setup best.

Example: Using runTest from vscode-test

The runTest function from vscode accepts multiple options, including the arguments you are using in your launch.json:

# From the typings of vscode-test

interface TestOptions {
    ...,
    /**
     * Absolute path to the extension root. Passed to `--extensionDevelopmentPath`.
     * Must include a `package.json` Extension Manifest.
     */
    extensionDevelopmentPath: string;
    /**
     * Absolute path to the extension tests runner. Passed to `--extensionTestsPath`.
     * Can be either a file path or a directory path that contains an `index.js`.
     * Must export a `run` function of the following signature:
     *
     * ```ts
     * function run(): Promise<void>;
     * ```
     *
     * When running the extension test, the Extension Development Host will call this function
     * that runs the test suite. This function should throws an error if any test fails.
     *
     */
    extensionTestsPath: string;
    /**
     * A list of launch arguments passed to VS Code executable, in addition to `--extensionDevelopmentPath`
     * and `--extensionTestsPath` which are provided by `extensionDevelopmentPath` and `extensionTestsPath`
     * options.
     *
     * If the first argument is a path to a file/folder/workspace, the launched VS Code instance
     * will open it.
     *
     * See `code --help` for possible arguments.
     */
    launchArgs?: string[];
}

It should be fairly easy to edit the src/test/runTest.ts scaffolded using yeoman to load multiple workspaces and run specific tests in them.

const workspaceTests = [
    { testDir: '../your/tests', workspaceDir: '../your/workspace' },
    ...
]

async function main() {
  try {
    const extensionDevelopmentPath = path.resolve(__dirname, '../../');

    for (const { testDir, workspaceDir } of workspaceTests) {  
        const extensionTestsPath = path.resolve(__dirname, testDir);
        const workspacePath = path.resolve(__dirname, workspaceDir);

        // Run tests in the specified workspace
        await runTests({
            extensionDevelopmentPath,
            extensionTestsPath,
            launchArgs: [ workspacePath ]
        });
    }
  } catch (err) {
    console.error('Failed to run tests');
    process.exit(1);
  }
}
user_52494110

For reasons that aren't fully clear to me, using the vscode.openFolder command while a unit test is running is simply A Bad Thing to Do. I guess it somehow disrupts the connections between the test window and the test environment.

Fortunately in this particular situation I can get by with opening just one folder, and that can be done in my `launch.json' config before the test code is launched.

{
  "version": "0.1.0",
  "configurations": [
    {
      ...
    },
    {
      "name": "Test Extension",
      "type": "extensionHost",
      "request": "launch",
      "runtimeExecutable": "${execPath}",
      "args": [
        "--extensionDevelopmentPath=${workspaceFolder}",
        "--extensionTestsPath=${workspaceFolder}/out/test/suite",
        "${workspaceFolder}/test/suite/sample-project"  // <-- project to open goes here
      ],
      "outFiles": [
        "${workspaceFolder}/out/test/**/*.js"
      ],
      "preLaunchTask": "npm"
    },
  ]
}

Once I've got my testing environment bootstrapped with this one sample project open to begin with, I can safely use the vscode.open command to open specific documents from within the sample project folder, and then monitor and test what my extension does with those documents.

user_73614790
Related