The starter template for VS Code extensions gives you a lot of flexibility when it comes to testing. By default, the extension under test is loaded in a special development mode, but this can be changed. For our own extension, Symflower for VS Code, we wanted to run tests on already released versions to smoke-test full end-to-end flows. Here’s how we did it.

Installing another extension in VS Code extension test setups

Before we can talk about replacing the extension loaded in development mode, we should figure out how to install an extension into VS Code in our test setup. VS Code extension tests (if you run them using the @vscode/test-electron library, which is how it is usually done) execute in a separate installation of VS Code that is wiped before a test run. If you’re using a setup that’s close to the starter template, there should be a file called runTest.ts which looks something like this:

async function main() {
  try {
    // The folder containing the Extension Manifest package.json
    // Passed to `--extensionDevelopmentPath`
    const extensionDevelopmentPath = path.resolve(__dirname, '../../');

    // The path to test runner
    // Passed to --extensionTestsPath
    const extensionTestsPath = path.resolve(__dirname, './suite/index');

    // Download VS Code, unzip it and run the integration test
    await runTests({ extensionDevelopmentPath, extensionTestsPath });
  } catch (err) {
    console.error('Failed to run tests');
    process.exit(1);
  }
}

We would like to be able to install an extension before the call to runTests. VS Code extensions can be automatically installed using the CLI, so it would be useful to have a path to the binary that provides the CLI, then pass that to the runTests function.

If you look through the @vscode/test-electron library you will discover a function called resolveCliArgsFromVSCodeExecutablePath which sounds promising: it expects an “executable path” which is not the CLI but a separate binary that the CLI uses internally and returns the path to the CLI combined with command line arguments to include in all invocations. And how can we get at the executable path? There’s another function simply called download which does whatever it needs to do to ensure a testing instance of VS Code is installed and returns its executable path. Additionally, we also need to pass this executable path to runTests since otherwise, it would call download on its own.

async function main() {
  try {
    // vscodeExecutablePath holds the path of the VS Code executable used for tests.
    const vscodeExecutablePath = await download();

    // cliPath holds the path of the wrapper script used to start VS Code in a non-blocking way or to install extensions.
    // This is the binary you normally interact with when you type "code" in your shell.
    // cliCommonArguments holds arguments that should be included in invocations of the "cliPath" script.
    const [cliPath, ...cliCommonArguments] = resolveCliArgsFromVSCodeExecutablePath(vscodeExecutablePath);

    // TODO: Do something with the CLI.

    // The folder containing the Extension Manifest package.json
    // Passed to `--extensionDevelopmentPath`
    const extensionDevelopmentPath = path.resolve(__dirname, '../../');

    // The path to test runner
    // Passed to --extensionTestsPath
    const extensionTestsPath = path.resolve(__dirname, './suite/index');

    // Run the integration test
    await runTests({
      extensionDevelopmentPath,
      extensionTestsPath,
      vscodeExecutablePath,
    });
  } catch (err) {
    console.error('Failed to run tests');
    process.exit(1);
  }
}

Now we just need to use the CLI to install our desired extension. Let’s replace that TODO comment.

// Install whatever extension you want, optionally also with a version (e.g. "symflower.symflower@0.0.26838").
const result = child_process.spawnSync(cliPath, [...cliCommonArguments, '--install-extension', 'symflower.symflower', '--force'], {
  encoding: 'utf-8',
  stdio: 'inherit',
});

if (result.status !== 0) {
  throw new Error('Failed to install extension');
}

Disabling extensionDevelopmentPath

We mentioned at the beginning of the article that our goal was to test released versions of our extension, downloaded from the Visual Studio Marketplace. We’ve figured out the download part. But if we ran the tests now, they would still load our extension from our local source directory in development mode.

We can disable that by changing the path we pass as extensionDevelopmentPath to runTests. Passing an invalid file path (such as an empty string) prevents the tests from running but passing a path to an empty directory works. It’s also possible to pass a null device (for example /dev/null on Linux).

await runTests({
  extensionDevelopmentPath: path.resolve(__dirname, 'empty-directory'),
  extensionTestsPath,
  vscodeExecutablePath,
});

If we run some tests now (maybe with a timeout to suspend the test so that we can click around) we’ll see that the extension passed to the CLI is installed, while the extension from our project is absent.

Feedback

Found something useful in this article? Great! Perhaps you would also enjoy Symflower for IntelliJ IDEA, GoLand, Android Studio, and of course VS Code to help with writing and maintaining software tests. You’re also invited to join our newsletter where we post insights into software development and testing.

If you have any questions or feedback for our articles or our plugin, we’d love to hear from you! You can send us an hello@symflower.com or find us on social media.