indexpost archiveatom feed syndication feed icon

Selenium Testing and Reproducible Environments


Lately I've been faced with some of the peculiarities of UI testing, in my case, using Selenium. Not only are the tests slower than other kinds of tests, they are often tied to specific enviornments. In an effort to combat this coupling of "special" environments I've been working on reliably creating new environments.

Provisioning Virtual Machines

I've been re-reading portions of Re-Engineering Legacy Software specifically those chapters dealing with automating the development workflow. Consequently I've been working to automate the provisioning of new virtual machines using Ansible and Vagrant.

If I have one complaint with Ansible, it is in the suggested structure of a project. I keep it, grudgingly, to maintain consistency but the directory structure always seems needlessly deep and repetitive.

├── roles
│   ├── browsers
│   │   └── tasks
│   │       └── main.yml
│   ├── drivers
│   │   └── tasks
│   │       └── main.yml
│   ├── selenium
│   │   └── tasks
│   │       └── main.yml
│   ├── twisted
│   │   ├── files
│   │   │   └── report-server.service
│   │   └── tasks
│   │       └── main.yml
│   └── xvfb
│       └── tasks
│           └── main.yml
└── selenium.yml

At the top level (selenium.yml), each role is simply enumerated:

- name: Setup Selenium test environment
  hosts: all
  become: true

    - drivers
    - xvfb
    - selenium
    - browsers
    - twisted

Each role consists of a main.yml file, which I've outlined below:

# Selenium
- name: Install pip
    name: python-pip

- name: Install pytest-selenium
    name: pytest-selenium

# Drivers
- name: Download geckodriver
    dest: /usr/local/bin
    remote_src: yes

- name: Install unzip (required for chromedriver extract)
    name: unzip

- name: Extract chromedriver
    dest: /usr/local/bin
    remote_src: yes

# Browsers
- name: Install Firefox
    name: firefox

# xvfb
- name: Install xvfb
    name: xvfb

# Twisted
- name: Install Twisted
    name: twisted[tls]

- name: Install report service
    src: report-server.service
    dest: /etc/systemd/system/report-server.service
    mode: 755

It is in the final role, that of Twisted, that we see something a bit different, I also install a systemd service to run a report server (a basic static file server) within the same virtual machine. The service is pretty plain and mirrors others that I've described previously

Description=Report Web Server

ExecStart=/usr/local/bin/twistd --nodaemon --pidfile= web --port tcp:8080 --path .



That's actually all it takes to setup an entirely new virtual machine for Selenium testing. In my case, I can build a new VM from scratch in about 2 minutes.

Running Headless

One key piece of software in the virtual machine setup I've described is xvfb, which allows X window applications to be run without the need of a real display. This means no annoying application pop-up windows to manage during a long-running test suite. I achieve this by prefixing any test invocation (using pytest) with xvfb-run which handles the mundane parts of setting up the virtual display. I haven't yet needed to dig into specifying aspect ratios or resolutions, but I understand it is possible if necessary.

Why Not Leverage Browsers?

It turns out that while both Firefox and Chrome purport to support headless operation, neither reaches parity with the standard display operation of either browser. This surfaced in one obvious failure, where with Chrome, the browser simply doesn't start using webdriver. The second, more subtle problem was with Firefox, where form data was submitted incompletely even after verifying fields were populated at the time of form submission. I have been unable to entirely track this problem down, and it may stem from a problem in the front-end of the application being tested. Either way, it prevents testing under the browser-supported headless operation for the time being.

Pytest Niceties

There are a few nice additions that pytest provides, out of the box and through the use of the pytest-selenium package. These include automatic screenshots on failure, an HTML report, and automatically injecting Selenium webdriver instances into tests. One of the most productive bits I've found though is included as part of pytest, which is the --lastfailed flag. With UI tests, which are unavoidably slow, taking minutes to run each, the last-failed flag allows the test runner to pick out on a second run only those tests that failed in the first run.

After the two minutes it takes to provision the virtual machine, and with the tests directory synced from the host machine, the whole setup gives me a pretty good environment for developing Selenium tests on my local machine. More importantly, it gives others on the team the ability to develop and run tests without the need for shared environments.