Selenium Testing and Reproducible Environments2017-11-09
Lately I've been faced with some of the peculiarities of UI testing, in my case, using Selenium. Not only are the tests slower than other kinds of tests, they are often tied to specific enviornments. In an effort to combat this coupling of "special" environments I've been working on reliably creating new environments.
Provisioning Virtual Machines
I've been re-reading portions of Re-Engineering Legacy Software specifically those chapters dealing with automating the development workflow. Consequently I've been working to automate the provisioning of new virtual machines using Ansible and Vagrant.
If I have one complaint with Ansible, it is in the suggested structure of a project. I keep it, grudgingly, to maintain consistency but the directory structure always seems needlessly deep and repetitive.
├── roles │ ├── browsers │ │ └── tasks │ │ └── main.yml │ ├── drivers │ │ └── tasks │ │ └── main.yml │ ├── selenium │ │ └── tasks │ │ └── main.yml │ ├── twisted │ │ ├── files │ │ │ └── report-server.service │ │ └── tasks │ │ └── main.yml │ └── xvfb │ └── tasks │ └── main.yml └── selenium.yml
At the top level (
selenium.yml), each role is simply enumerated:
- name: Setup Selenium test environment hosts: all become: true roles: - drivers - xvfb - selenium - browsers - twisted
Each role consists of a
main.yml file, which I've outlined below:
# Selenium - name: Install pip apt: name: python-pip - name: Install pytest-selenium pip: name: pytest-selenium # Drivers - name: Download geckodriver unarchive: src: https://github.com/mozilla/geckodriver/releases/download/v0.19.1/geckodriver-v0.19.1-linux64.tar.gz dest: /usr/local/bin remote_src: yes - name: Install unzip (required for chromedriver extract) apt: name: unzip - name: Extract chromedriver unarchive: src: https://chromedriver.storage.googleapis.com/2.33/chromedriver_linux64.zip dest: /usr/local/bin remote_src: yes # Browsers - name: Install Firefox apt: name: firefox # xvfb - name: Install xvfb apt: name: xvfb # Twisted - name: Install Twisted pip: name: twisted[tls] - name: Install report service copy: src: report-server.service dest: /etc/systemd/system/report-server.service mode: 755
It is in the final role, that of Twisted, that we see something a bit different, I also install a systemd service to run a report server (a basic static file server) within the same virtual machine. The service is pretty plain and mirrors others that I've described previously
[Unit] Description=Report Web Server [Service] ExecStart=/usr/local/bin/twistd --nodaemon --pidfile= web --port tcp:8080 --path . WorkingDirectory=/home/vagrant Restart=always [Install] WantedBy=multi-user.target
That's actually all it takes to setup an entirely new virtual machine for Selenium testing. In my case, I can build a new VM from scratch in about 2 minutes.
One key piece of software in the virtual machine setup I've described is
allows X window applications to be run without the need of a real display. This
means no annoying application pop-up windows to manage during a long-running
test suite. I achieve this by prefixing any test invocation (using pytest) with
xvfb-run which handles the mundane parts of setting up the virtual display. I
haven't yet needed to dig into specifying aspect ratios or resolutions, but I
understand it is possible if necessary.
Why Not Leverage Browsers?
It turns out that while both Firefox and Chrome purport to support headless operation, neither reaches parity with the standard display operation of either browser. This surfaced in one obvious failure, where with Chrome, the browser simply doesn't start using webdriver. The second, more subtle problem was with Firefox, where form data was submitted incompletely even after verifying fields were populated at the time of form submission. I have been unable to entirely track this problem down, and it may stem from a problem in the front-end of the application being tested. Either way, it prevents testing under the browser-supported headless operation for the time being.
There are a few nice additions that pytest provides, out of the box and through
the use of the pytest-selenium
package. These include automatic screenshots on failure, an HTML report, and
automatically injecting Selenium webdriver instances into tests. One of the
most productive bits I've found though is included as part of pytest, which is
--lastfailed flag. With UI tests, which are unavoidably slow, taking
minutes to run each, the last-failed flag allows the test runner to pick out on
a second run only those tests that failed in the first run.
After the two minutes it takes to provision the virtual machine, and with the tests directory synced from the host machine, the whole setup gives me a pretty good environment for developing Selenium tests on my local machine. More importantly, it gives others on the team the ability to develop and run tests without the need for shared environments.