A more robust way to install VS Code for extension testing

Through testing our VS Code extension – Symflower for VS Code – we’ve stumbled over some interesting issues and edge cases. One of them was that the copy of VS Code, which was automatically installed by the test framework, sometimes got corrupted during a test run. This broke all test jobs that started afterwards, since they are using that same cached installation. We’re not sure why exactly that happens – it might be something the tests do or an issue in VS Code or something in the whole setup just can’t handle abrupt exits when the CI server cancels a job.

In any case, we found a workaround: we’ve improved the logic that installs VS Code to be more stringent about checking the integrity of installations and not re-using installations between different test runs while still caching downloads. Here’s how we did it.

How VS Code is installed normally

We’ve touched on this in a previous post but let’s go over it again shortly. In a default VS Code extension project (initialized by the Yeoman generator) there’s a file called runTest.ts. This file handles the installation and execution of the VS Code instance for tests, with all that being implemented by the @vscode/test-electron library. There’s also @vscode/test-web for testing the web version of VS Code but test-electron is more common.

A runTest file might look something like this:

async function main() {
  try {
    // The folder containing the extension manifest package.json
    // Passed to `--extensionDevelopmentPath`
    const extensionDevelopmentPath = path.resolve(__dirname, '../../');

    // The path to test runner
    // Passed to --extensionTestsPath
    const extensionTestsPath = path.resolve(__dirname, './suite/index');

    // Download VS Code, unzip it and run the integration test
    await runTests({ extensionDevelopmentPath, extensionTestsPath });
  } catch (err) {
    console.error('Failed to run tests');
    process.exit(1);
  }
}

By default, the VS Code download is abstracted away from you and happens in the runTests function. But we can move it out of there to perform actions on the downloaded installation before any tests are run. There’s a function called download in the test-electron library that downloads and unpacks a release archive of VS Code and returns a path to the VS Code executable.

const vscodeExecutablePath = await download();

await runTests({
  extensionDevelopmentPath,
  extensionTestsPath,
  vscodeExecutablePath,
});

If we want to customize this download behavior, we have to implement our own download function that obtains VS Code somehow from somewhere and returns a path to the executable.

Implementing our own download function

What do we want? We want a download function that does not re-use installations between test runs. But we also don’t want to download VS Code again every time. In other words: we want a download function that caches VS Code releases and installs them anew every time we start our tests. The@vscode/test-electron library installs a VS Code version once and re-uses that installation over multiple test runs, which breaks our first requirement, so we have to manage the download ourselves.

VS Code releases are provided as archives (ZIP on Windows, gzipped TAR on Linux and macOS) that we can just download and keep around. We can unpack the archives before test runs. Additionally, the VS Code update API provides checksums for those archives, so we can verify the integrity of archives before unpacking. The archives themselves should not get corrupted anyways but we get that for free with update checks and it can’t hurt to make the installation logic more robust.

Let’s define the general structure of this. We’re going to go through some function signatures that we’ve defined in our internal codebase at Symflower and find out what each function does and how it all comes together to create our custom download implementation. Explaining the implementation of each function in detail would take too long but you can find the full code on our GitHub.

At the top level, we have our equivalent to the download function in @vscode/test-electron. Although we’ve actually called it installVSCode since we felt that’s more descriptive of what it does.

function installVSCode(cachePath: string, installationPath: string): Promise<string>;

This function combines everything from figuring out which version of VS Code to install, downloading, and installing that version. It returns the path of the VS Code executable just like the download function in test-electron. Our function takes two parameters: a path where archives should be cached (cachePath) and a path that holds the unpacked VS Code installation files during test runs (installationPath).

The functionality of the top-level function is split into multiple single-purpose functions to make the code easier to maintain. The first of these functions fetches information about the latest VS Code release.

function fetchLatestVSCodeVersion(): Promise<VSCodeVersionInfo>;

The next function takes that version info and makes sure the release archive for the current platform for that version is available and is not damaged.

function downloadVSCode(versionInfo: VSCodeVersionInfo, cachePath: string): Promise<VSCodeArchiveInfo>;

It does so by checking if the archive already exists in the cache with the right checksum (obtained from versionInfo) and if not, attempts a download. Downloads are retried a limited amount of times on failure.

Last in the chain there’s a function that unpacks the downloaded archive.

function extractVSCode(archiveFile: VSCodeArchiveInfo, installationPath: string): Promise<void>;

However, we still need the path of the VS Code executable to have a drop-in replacement for the default download function. The @vscode/test-electron library has a function for that called downloadDirToExecutablePath that we can use. It takes a path of a VS Code installation (installationPath in our case) and returns the executable path. We also have to specify what platform we’re dealing with. For the current platform we can just use the systemDefaultPlatform export from the test library.

All of this is connected together in the installVSCode function:

async function installVSCode(cachePath: string, installationPath: string): Promise<string> {
  const latestVersion = await fetchLatestVSCodeVersion();
  const archive = await downloadVSCode(latestVersion, cachePath);
  await extractVSCode(archive, installationPath);
  return downloadDirToExecutablePath(installationPath, systemDefaultPlatform);
}

Now that we’ve gone over all that, we can replace the call to download with our own installVSCode function:

const cachePath = '.vscode-test-archives';
const installationPath = path.join(os.tmpdir(), 'vscode-test-installation');

const vscodeExecutablePath = await installVSCode(cachePath, installationPath);

await runTests({
  extensionDevelopmentPath,
  extensionTestsPath,
  vscodeExecutablePath,
});

And that’s it! If we start our extension tests now, a VS Code release archive will be downloaded to the cache path, extracted and run from the installation path.

One thing to note: The download function from @vscode/test-electron by default uses streams to download and extract release archives concurrently, so our solution is a little bit slower. On Linux we haven’t noticed any slowdowns (we suspect it might be around one or two seconds per test run – not per individual test). On Windows, at least when we’ve tested it, extraction took a considerable amount of time, so slowdowns might be more noticeable there. (Although given how much longer extraction takes on Windows, there might be something wrong with the extraction logic in general.)

We’ve also proposed upstreaming this to the authors of @vscode/test-electron but it was rejected for the performance reasons above, which is fair. It should be possible to modify our implementation to take advantage of streaming as well but that’s left as an exercise for the reader. 🙂

Thanks to the VS Code team for providing such an extensible testing library! Also, the code of @vscode/test-electron is pretty readable, we’ve re-used some bits for our own archive extraction function.

If you want to stay up-to-date on our insights in VS Code extension development and software testing, consider subscribing to our newsletter. Also check out Symflower for VS Code or IntelliJ to automatically generate unit tests if you haven’t already.