Testing IntelliJ plugins on Windows in your CI using GitLab Runner

At Symflower, we subject all of our software components to mandatory automatic test suites before release, which is implemented using a GitLab-based CI/CD workflow. One of those components is our plugin for IntelliJ IDEA, GoLand and Android Studio. One of the ways that plugin is tested is using end-to-end tests with UI automation on multiple operating systems including Windows. Getting that set up, was a bit tricky.

A quick primer on UI tests

“UI testing” in the context of an IntelliJ plugin refers to automated tests where the plugin runs in a full-blown instance of IntelliJ that is operated automatically by the controlling test case. The test has the ability to interact with the IDE using both UI automation and JavaScript snippets that are evaluated in the context of the IDE and have access to the entire IntelliJ platform API. Usually, windows opened by the IDE are also recorded by a screen capture tool to provide developers with a better understanding of test failures.

Setting up a Continuous Integration environment on Linux, that can execute UI tests is relatively easy. All that’s required (in slightly simplified terms) is an X server, which is a component on Linux that provides graphical user interface support, and those come in several shapes and sizes. A CI job would most likely use Xvfb which provides a GUI environment without rendering to an actual screen.

Windows, however, doesn’t use the X window system or X servers, nor does it have an equivalent to Xvfb. We have to set up a Windows installation with a GUI (that means any Windows version except Nano Server) and install GitLab Runner on that.

Windows problems: Services

Our first approach was to execute a GitLab Runner instance in its default configuration on a Windows Server machine which consists of a Windows service. Our thinking was: since Windows Server comes with a GUI, UI automation should just work. That’s when we ran into our first issue: turns out Windows services cannot create or interact with any GUI. We also tried setting a flag titled “Allow service to interact with desktop” in the service’s configuration but no dice.

The only way for a program to interact with a GUI seems to be to start it in a user session but at least Windows has a tool for that: autostart scripts. To find the directory that contains autostart scripts for the logged in user, press Win+R and enter shell:startup.

Then, create a file called gitlab-runner.cmd in the folder that should have been opened with the following content:

rem Replace the following path with the intended working directory for GitLab Runner.
cd C:\Gitlab-Runner
start .\gitlab-runner.exe run

At this point, you should also disable the GitLab Runner service in case that has been set up by an automated installation.

However, that’s not a complete solution. We still need to log the user in on startup for the session to be created and autostart scripts to run. Windows has that built in via specific registry options that you can set. Alternatively, you can use a tool like SysInternals' Autologon to do those registry modifications for you and provide you with a nice UI.

HOWEVER, we have tried that and it did not work. The automatic login did, as did our autostart script. GitLab Runner was executing jobs just fine, but the tests were constantly failing. Checking the screen recordings, all we got was a completely black screen. What’s going on?

Windows problems: Keeping the session alive

It turns out that Windows somehow deactivates the graphical session when no interactive console is connected (either in the form of a screen that’s plugged in, or an open RDP session). But we need to keep that session running for our IntelliJ instance to have a GUI environment to run in.

We’re not going to bore you with all the details (this post is long enough as-is) but we’ve already tried disabling the lock screen and fiddling with RDP session settings, to no avail. In the end, we did find a solution to this problem, though it’s a bit unorthodox.

There is a proprietary tool called “LogonExpert” that handles automatic logins and optionally keeps the screen unlocked with the graphical session still alive even when no console is connected. That’s it, it’s that simple. We don’t know how that tool works exactly or if there are any open source alternatives, but at this point we just wanted to get done with this setup, so we bought some licenses.

First-time setup via the installer is pretty straightforward and out of the box you get a ten-day trial period. Once installed, launch the “LogonExpert Administrator” application. In the tab titled “Credentials” enter the credentials of the user that will be executing GitLab Runner. (In our case that’s the “Administrator” user though to be fair, a user without administrator privileges would be more secure.)

Next, switch to the “Options” tab and check the “Keep computer unlocked under” checkbox. Select your automatically logged in user from the dropdown on the right and click apply.

Close the app and reboot. The user should have been logged in automatically and GitLab Runner should be running. Complete any GitLab Runner setup steps in case you haven’t yet.

If you now start a UI test on that runner, it should run just fine.

Bonus tip: Use labels to restrict where your jobs can run. We have a label called “windows-ui” for CI jobs that need to run on a Windows runner with a GUI setup. That runner is is set up with the configuration described in this article. Jobs tagged with that will only be picked up by those runners.

Got some use out of this article? Great! Perhaps you would also enjoy Symflower for IntelliJ IDEA, GoLand, Android Studio or VS Code to help with writing and maintaining software tests. You’re also most welcome to join our newsletter where we post insights into software development and testing.

If you have any questions or feedback for our articles or our extension, we’d love to hear from you! You can send us an hello@symflower.com or find us on social media.