When testing an IntelliJ plugin, you’ll likely reach for UI tests to verify that both the user interface as well as the code behind it functions correctly. In UI tests, your plugin runs in a real instance of IntelliJ that is similar to a production environment on a user’s machine. That increases your end-to-end coverage but sometimes it’s harder to verify success conditions for a test in that environment. So as a last resort, you might want to call into a service that’s part of the plugin to verify some internal state directly.

For example, we found for our Symflower plugin, we needed this to check for progress indicators displayed by our plugin. We could not verify our plugin’s progress indicators purely from the UI because they were too fast for the UI automation to pick them up. So we had to build a service into our plugin that, when run under test, keeps references to shown progress indicators to allow tests to verify them. But how would one go about doing that?

JavaScript in UI tests

IntelliJ test code for UI tests normally runs in a separate process, isolated from the plugin under test which runs installed in a full-blown IntelliJ instance (started using the runIdeForUiTest Gradle task). Tests can communicate with that instance using UI automation and JavaScript code that’s sent to IntelliJ and can optionally send a result back.

To run a script without getting a result back, use the runJs method on the RemoteRobot class:

remoteRobot.runJs("System.out.println('I run in the IntelliJ test instance!')");

The callJs method behaves similarly but returns the result of the last evaluated JavaScript expression from the Java method call:

boolean oneIsTwo = remoteRobot.callJs("System.out.println('Doing something…'); 1 === 2");

The JavaScript code has access to an environment that contains the entire IntelliJ platform API available to plugins. The API is automatically translated from Java to JavaScript, so you can write JavaScript code that looks similar to Java code found in an IntelliJ plugin.

The problem with plugin classes

Let’s try calling some code in a class that’s part of a plugin from our tests. We’ll start with a simple IntelliJ plugin that we can edit. (Don’t have a plugin to edit? Check out the previous part of this series where we develop a very simple plugin step-by-step.)

First, let’s define a service class that we’re going to call from our tests.

// DummyService.java
package com.example.demo;

import com.intellij.openapi.components.Service;

@Service
public final class DummyService {
    public String getMessage() {
        return "Hello from the other side!";
    }
}

Then define a test that simply loads the service, calls the getMessage method, and compares the result.

// In some test class
@Test
void dummy() {
    var remoteRobot = new RemoteRobot("http://127.0.0.1:8082");
    String message = remoteRobot.callJs("com.intellij.openapi.application.ApplicationManager.getApplication().getService(com.example.demo.DummyService.class).getMessage()");
    assertEquals("Hello from the other side!", message);
}

Alright, let’s see if this works. To execute the test we’ve just defined, start the runIdeForUiTest Gradle task and then the actual test. Aaaand we get an error:

Can't find method com.intellij.openapi.components.ComponentManager.getService(object).
com.intellij.remoterobot.client.IdeaSideException: Can't find method com.intellij.openapi.components.ComponentManager.getService(object).
	at com.intellij.remoterobot.client.IdeRobotClient.throwIdeaSideError(IdeRobotClient.kt:142)
	at com.intellij.remoterobot.client.IdeRobotClient.processRetrieveResponse(IdeRobotClient.kt:118)
	at com.intellij.remoterobot.client.IdeRobotClient.retrieve(IdeRobotClient.kt:75)
	at com.intellij.remoterobot.JavaScriptApi$DefaultImpls.callJs(JavaScriptApi.kt:78)
	at com.intellij.remoterobot.RemoteRobot.callJs(RemoteRobot.kt:32)
	at com.intellij.remoterobot.JavaScriptApi$DefaultImpls.callJs(JavaScriptApi.kt:67)
	at com.intellij.remoterobot.RemoteRobot.callJs(RemoteRobot.kt:32)
	at com.example.demo.DummyTest.dummy(Test.java:11)

“Can’t find method getService”? That sounds strange. Perhaps there’s something wrong with using a class like that in JavaScript snippets? Let’s try another way to get a reference to the class:

String message = remoteRobot.callJs("com.intellij.openapi.application.ApplicationManager.getApplication().getService(Class.forName('com.example.demo.DummyService')).getMessage()");

Now we get a much clearer error message:

Wrapped java.lang.ClassNotFoundException: com.example.demo.DummyService PluginClassLoader(plugin=PluginDescriptor(name=Robot server, id=com.jetbrains.test.robot-server-plugin, descriptorPath=plugin.xml, path=[...], version=0.11.16, package=null, isBundled=false), packagePrefix=null, instanceId=18, state=active)
com.intellij.remoterobot.client.IdeaSideException: Wrapped java.lang.ClassNotFoundException: com.example.demo.DummyService PluginClassLoader(plugin=PluginDescriptor(name=Robot server, id=com.jetbrains.test.robot-server-plugin, descriptorPath=plugin.xml, path=[...], version=0.11.16, package=null, isBundled=false), packagePrefix=null, instanceId=18, state=active)
	at com.intellij.remoterobot.client.IdeRobotClient.throwIdeaSideError(IdeRobotClient.kt:142)
	at com.intellij.remoterobot.client.IdeRobotClient.processRetrieveResponse(IdeRobotClient.kt:118)
	at com.intellij.remoterobot.client.IdeRobotClient.retrieve(IdeRobotClient.kt:75)
	at com.intellij.remoterobot.JavaScriptApi$DefaultImpls.callJs(JavaScriptApi.kt:78)
	at com.intellij.remoterobot.RemoteRobot.callJs(RemoteRobot.kt:32)
	at com.intellij.remoterobot.JavaScriptApi$DefaultImpls.callJs(JavaScriptApi.kt:67)
	at com.intellij.remoterobot.RemoteRobot.callJs(RemoteRobot.kt:32)
	at com.example.demo.DummyTest.dummy(DummyTest.java:11)

You might be able to guess what’s happening here already, but let’s break it down: Class references (both direct via DummyService.class and using Class.forName) are resolved using a so-called class loader. Class loaders in Java are responsible for locating and reading class files (or loading classes from elsewhere) and parsing the class bytecode. One Java program can use multiple class loaders where each class loader is loaded by a parent class loader, all the way up to the bootstrap class loader which is provided by the JVM.

Why is that relevant? IntelliJ uses separate class loaders for each plugin. Those class loaders only provide classes from the plugin they have been initialized for and classes available to all plugins to avoid name clashes when two plugins use the same fully qualified name for a class. But for our test, this poses a problem: JavaScript in the test is evaluated in the context of a different plugin (called “Robot server”) that is also installed in the testing environment and handles all sorts of automatic interactions between the tests and the testing instance of IntelliJ. Since each plugin has its own namespaced class loader and the plugin under test is different from the plugin where the test script is executed, the script effectively runs in a different namespace.

Solution

Fortunately, there’s an easy solution for our problem. While we cannot use Class.forName to load our desired plugin class, since it uses a different plugin’s class loader, we can get at the class loader of the plugin under test and use that directly to load the right class. Let’s modify our test:

@Test
void dummy() {
    var remoteRobot = new RemoteRobot("http://127.0.0.1:8082");
    String message = remoteRobot.callJs("const pluginManager = com.intellij.ide.plugins.PluginManager.getInstance();" +
            "const pluginID = com.intellij.openapi.extensions.PluginId.findId('com.example.demo');" +
            "const plugin = pluginManager.findEnabledPlugin(pluginID);" +
            "const clazz = plugin.getPluginClassLoader().loadClass('com.example.demo.DummyService');" +
            "com.intellij.openapi.application.ApplicationManager.getApplication().getService(clazz).getMessage()");
    Assertions.assertEquals("Hello from the other side!", message);
}

And that finally passes. Nice!

Wrapping it up (into helper functions)

The code above still looks a bit rough and perhaps we want to do something similar in other test cases of our plugin. Let’s define some re-usable functions for what we’ve done above that we can call in JavaScript snippets.

We’re going to use a global hook that runs before all tests to define those helper functions, so that they’re available everywhere. In case your plugin is built using the same structure as the starter template for IntelliJ plugins you can implement the org.junit.jupiter.api.extension.BeforeTestExecution interface in the RemoteRobotExtension class and add a runJs block in the beforeTestExecution method where you define your helpers.

override fun beforeTestExecution(context: ExtensionContext?) {
    remoteRobot.runJs(
        """
        global.put('loadPlugin', function () {
            const pluginManager = com.intellij.ide.plugins.PluginManager.getInstance();
            const pluginID = com.intellij.openapi.extensions.PluginId.findId("com.example.demo");
            return pluginManager.findEnabledPlugin(pluginID);
        });

        global.put('loadPluginClass', function (className) {
            return global.get('loadPlugin')().getPluginClassLoader().loadClass(className);
        });

        global.put('loadPluginService', function (className) {
            return com.intellij.openapi.application.ApplicationManager.getApplication().getService(global.get("loadPluginClass")(className));
        });
        """.trimIndent()
    )
}

We have to put our helpers in the global map. If we just defined them as regular JavaScript functions in the global scope, they wouldn’t be available in the next callJs or runJs call.

The helpers can then be used like the following in UI tests:

@Test
void dummy() {
    var remoteRobot = new RemoteRobot("http://127.0.0.1:8082");
    String message = remoteRobot.callJs("global.get('loadPluginService')('com.example.demo.DummyService').getMessage()");
    Assertions.assertEquals("Hello from the other side!", message);
}

What do you think? Do you have any feedback or questions? We want to hear from you! Write us on social media or at hello@symflower.com . You can also subscribe to our newsletter to learn more about IntelliJ plugin development and testing.