Selenium Testing and Reproducible Environments

2017-11-09

Lately I've been faced with some of the peculiarities of UI testing, in my case, using Selenium. Not only are the tests slower than other kinds of tests, they are often tied to specific enviornments. In an effort to combat this coupling of "special" environments I've been working on reliably creating new environments.

Provisioning Virtual Machines

I've been re-reading portions of Re-Engineering Legacy Software specifically those chapters dealing with automating the development workflow. Consequently I've been working to automate the provisioning of new virtual machines using Ansible and Vagrant.

If I have one complaint with Ansible, it is in the suggested structure of a project. I keep it, grudgingly, to maintain consistency but the directory structure always seems needlessly deep and repetitive.

├── roles
│   ├── browsers
│   │   └── tasks
│   │       └── main.yml
│   ├── drivers
│   │   └── tasks
│   │       └── main.yml
│   ├── selenium
│   │   └── tasks
│   │       └── main.yml
│   ├── twisted
│   │   ├── files
│   │   │   └── report-server.service
│   │   └── tasks
│   │       └── main.yml
│   └── xvfb
│       └── tasks
│           └── main.yml
└── selenium.yml

At the top level (selenium.yml), each role is simply enumerated:

- name: Setup Selenium test environment
  hosts: all
  become: true

  roles:
    - drivers
    - xvfb
    - selenium
    - browsers
    - twisted

Each role consists of a main.yml file, which I've outlined below:

# Selenium
- name: Install pip
  apt:
    name: python-pip

- name: Install pytest-selenium
  pip:
    name: pytest-selenium

# Drivers
- name: Download geckodriver
  unarchive:
    src: https://github.com/mozilla/geckodriver/releases/download/v0.19.1/geckodriver-v0.19.1-linux64.tar.gz
    dest: /usr/local/bin
    remote_src: yes

- name: Install unzip (required for chromedriver extract)
  apt:
    name: unzip

- name: Extract chromedriver
  unarchive:
    src: https://chromedriver.storage.googleapis.com/2.33/chromedriver_linux64.zip
    dest: /usr/local/bin
    remote_src: yes

# Browsers
- name: Install Firefox
  apt:
    name: firefox

# xvfb
- name: Install xvfb
  apt:
    name: xvfb

# Twisted
- name: Install Twisted
  pip:
    name: twisted[tls]

- name: Install report service
  copy:
    src: report-server.service
    dest: /etc/systemd/system/report-server.service
    mode: 755

It is in the final role, that of Twisted, that we see something a bit different, I also install a systemd service to run a report server (a basic static file server) within the same virtual machine. The service is pretty plain and mirrors others that I've described previously

[Unit]
Description=Report Web Server

[Service]
ExecStart=/usr/local/bin/twistd --nodaemon --pidfile= web --port tcp:8080 --path .

WorkingDirectory=/home/vagrant
Restart=always

[Install]
WantedBy=multi-user.target

That's actually all it takes to setup an entirely new virtual machine for Selenium testing. In my case, I can build a new VM from scratch in about 2 minutes.

Running Headless

One key piece of software in the virtual machine setup I've described is xvfb, which allows X window applications to be run without the need of a real display. This means no annoying application pop-up windows to manage during a long-running test suite. I achieve this by prefixing any test invocation (using pytest) with xvfb-run which handles the mundane parts of setting up the virtual display. I haven't yet needed to dig into specifying aspect ratios or resolutions, but I understand it is possible if necessary.

Why Not Leverage Browsers?

It turns out that while both Firefox and Chrome purport to support headless operation, neither reaches parity with the standard display operation of either browser. This surfaced in one obvious failure, where with Chrome, the browser simply doesn't start using webdriver. The second, more subtle problem was with Firefox, where form data was submitted incompletely even after verifying fields were populated at the time of form submission. I have been unable to entirely track this problem down, and it may stem from a problem in the front-end of the application being tested. Either way, it prevents testing under the browser-supported headless operation for the time being.

Pytest Niceties

There are a few nice additions that pytest provides, out of the box and through the use of the pytest-selenium package. These include automatic screenshots on failure, an HTML report, and automatically injecting Selenium webdriver instances into tests. One of the most productive bits I've found though is included as part of pytest, which is the --lastfailed flag. With UI tests, which are unavoidably slow, taking minutes to run each, the last-failed flag allows the test runner to pick out on a second run only those tests that failed in the first run.

After the two minutes it takes to provision the virtual machine, and with the tests directory synced from the host machine, the whole setup gives me a pretty good environment for developing Selenium tests on my local machine. More importantly, it gives others on the team the ability to develop and run tests without the need for shared environments.