Automated UI testing is an integral part of our continuous integration process with Miradore Online. Today, we have some 140 individual UI tests that run every time our test environment is updated. Each test is composed of multiple small operations, i.e. opening forms and views, going through action wizards, editing fields, testing field validators, etc. We’ve selected Selenium as our UI test library. It is commonly used, available for a wide range of platforms and it’s easy to integrate. Selenium supports basically all major browsers via components called WebDrivers, mobiles included. There’s also a JavaScript-based dummy browser, PhantomJS, which we use on our test server. We use PhantomJS mainly because it’s fast and doesn’t open any actual windows, which would be pointless since the tests are run by a background service.

A single test takes anywhere from a few seconds to a few minutes to run, and in total the 140 tests take a little over an hour to complete. Because the test environment is constantly updated, an hour is a tad too much and the execution time is slowly becoming a problem. The reason for such a long execution time is that the tests are run in sequence, causing a single long-running test to block every other test from running until it finishes. A seemingly simple solution would be to run multiple tests simultaneously. Since Miradore Online is .NET based, we use Visual Studio and MSTest to execute tests. However, this setup poses some limitations to parallel test execution. One of the limitations is that parallel execution should not be utilized with UI tests (see MSDN article). Microsoft doesn’t really specify why, so one can only guess. Anyway, we felt that ignoring this particular limitation was ok.

Running multiple unit tests simultaneously can be achieved by using a test settings file instead of allowing the framework to run with the defaults. A test settings file can be created in Visual Studio by selecting the solution level in solution explorer and pressing Ctrl-Shift-A (Add new item). From this dialog, you can create the test settings by selecting the right item type (Test Settings) and clicking Add. This should create two new files under Solution Items in your solution. Next, use a text editor outside Visual Studio to edit the file ending in .testsettings extension. You need to manually add the following under the root node:

<Execution parallelTestCount="5"/>

This setting is probably self-explanatory but as a side note, Microsoft states that 5 is the maximum. Since this was enough for us, we haven’t tested if this is actually true.

To use the newly created test settings in Visual Studio, open Test menu and use Test Settings > Select Test Settings File to pick the settings. This is enough to make the tests run simultaneously. As noted earlier, Microsoft states that you should not run UI tests in parallel. However, from the test framework’s perspective, there’s no difference between UI tests and standard unit tests. To me it seems that the limitation is purely a statement that you shouldn’t do it. As long as we can properly isolate individual tests to tackle concurrency issues, we should be fine.

Before going further into details, I feel like I should describe our test setup a bit. All of our UI tests start by resetting the database of a test site to ensure the test conditions don’t change, and then logging in to the site. When the tests are run in sequence, we are good with just one test site configuration. There simply can’t be any concurrency related problems. However, to achieve isolation when multiple tests are running at the same time, we have to use a different test site configuration for each simultaneously running test. The way we did it was by pooling a set of test site configurations. The tests queue for a configuration until one becomes available.

Let’s dig more in to the technical details. At the start of a test, the test framework always creates a new instance of the test class, even when running tests from the same class. This is a good thing. It means that we can store the test settings to class members during test class initialization and don’t need to worry about manually passing the correct configurations to different test methods. Since every test requires us to reset the database and log in, a common base class felt like a good solution. The base class also takes care of queuing and storing the settings at the beginning of the test, and releasing them at the end of the test. Shown below are the contents of said base class containing the test initialization and cleanup methods. The initialization method reserves test settings, creates a new browser window, resets the database and finally logs in. The test cleanup method logs out, closes the browser window and releases test settings for the next queued test.

namespace Miradore.Server.Tests
{
  [TestClass]
  public abstract class ServerTestBase : TestBase
  {
    protected LoginConfiguration Configuration { get; private set; }
    protected IWebDriver WebDriver { get; private set; }

    [TestInitialize]
    public void TestInitialize()
    {
      // Reserve a login configuration for this test
      Configuration = Settings.ReserveConfiguration();

      WebDriver = Settings.GetDriver();

      TestUtils.ResetDatabase(Configuration);

      WebDriver.Navigate().GoToUrl(Configuration.GetInstanceURL());
      FillTextBoxByName("ctl00$MainContent$LoginControl$UserName", Configuration.LoginUser);
      FillTextBoxByName("ctl00$MainContent$LoginControl$Password", Configuration.LoginPassword);
      ClickElementByName("ctl00$MainContent$LoginControl$ctl05");
    }

    [TestCleanup]
    public void TestCleanup()
    {
      WebDriver.Navigate().GoToUrl(Configuration.GetLogoutURI());
      WebDriver.Quit();
      WebDriver = null;

      // Release the configuration for other tests
      Settings.ReleaseConfiguration(Configuration);
      Configuration = null;
    }
  }
}

Next shown is the handler that contains the logic for queuing test settings. When ReserveConfiguration() is run for the first time, it loads all test settings from the configuration file and adds them to a queue. When a test run tries to reserve a configuration, one is picked from the queue and stored locally by the test class. If the queue is empty, the test will wait until a configuration becomes available. ReleaseConfiguration() adds the configuration back to the queue, allowing the next test to pick it up. Concurrency issues are tackled with an object lock.

namespace Miradore.Server.Tests
{
  public class Settings
  {
    private static bool _initialized = false;
    private static readonly Object _lock = new Object();
    private static readonly Queue<LoginConfiguration> _configQueue = new Queue<LoginConfiguration>();

    public static LoginConfiguration ReserveConfiguration()
    {
      Monitor.Enter(_lock);

      if (!_initialized)
      {
        List<LoginConfiguration> configs = InitTestSettings();

        foreach (LoginConfiguration config : configs)
          {
            _configQueue.Enqueue(config);
          }

        _initialized = true;
      }

      while (_configQueue.Count == 0)
      {
        Monitor.Wait(_lock);
      }

      LoginConfiguration config = _configQueue.Dequeue();

      Monitor.Exit(_lock);

      return config;
    }

    public static void ReleaseConfiguration(LoginConfiguration config)
    {
      Monitor.Enter(_lock);
      _configQueue.Enqueue(config);
      Monitor.PulseAll(_lock);
      Monitor.Exit(_lock);
    }
  }
}

With these rather small changes we’ve achieved parallel test execution and managed to drop the execution time of our UI tests drastically. Previously, running the full test set took a little less than 70 minutes. After the changes, the execution takes less than 15. Talk about improvement!

Build results before

Build results before the changes. Notice the execution time, 1 hour and 13 minutes.

Build results after

Build results after the changes. A solid 56 minute improvement. Environment update takes about 10 minutes of the total time, so the test run actually took less than 10 minutes here.

A word of advice though. Do not use real browsers with Selenium when running multiple tests simultaneously. Real browsers share the login state between browser windows so you will not achieve proper isolation between the tests. Concurrency issues will follow. As stated earlier, we use PhantomJS with our continuous integration system. PhantomJS doesn’t share the same problem.

Miradore Ltd
Follow us!

Miradore Ltd

Miradore is the European pioneer in managing diverse IT environments and supporting Bring Your Own Device (BYOD) policies.
Miradore Ltd
Follow us!