Cross Browser Testing Protocol

Here’s an overview of the way I defined some kind of protocol for recurring cross-browser testing.

The goal of recurring cross-browser testing

So far, most of the cross-browser testing I had been doing was on website I had made, while doing only the frontend dev, on static pages. I was rather easy to know what had been tested and what wasn’t as my process was rather linear : open a psd, code that page, make sure it works in other browsers, fix what doesn’t.

All this has changed since I’ve joined a team to work as a frontend developer on an existing website. A lot of code already exists, I’m adding a lot of code to that (more on this in another blog post), it’s becoming a huge pile of CSS, parts I don’t know.

To fix that, I decided to get into some kind of routine : testing the whole website in each browser (our compatibility table being IE7~9, FF and Chrome).

The spreadsheet approach

First step is to create a spreadsheet with three information : date of the test, page tested and the browser/version.

It doesn’t really look good but at least it’s simpler, I added some dynamic styling to it to make empty rows (not tested), tested with bugs and tested without bugs look different, but overal it’s rather simple to understand.

When I’m ready to test, I just start a Virtual Machine, go through each of these pages (about 40 different kinds of pages), look at everything on the page, test the JS components and note how many bugs I’ve found. Most of them I fix right away, some other I leave for later, assigning myself a ticket to fix it.

Spreadsheet Generation

Being a developer, the idea of going, by hand, through all the pages simply wasn’t an option. So here’s how I did to automate this process, I’ll talk at the end of this blog post about addition I made to this script to make it even more awesome !

Wget

First, I use wget to get all the pages of the website. That’s quite a lot of pages, but I couldn’t find any reliable way to crawl a website, some methods would fail because of too many connections at the same time, coding it by hand seemed too long, so I went with this easy way.

wget -r -k -m -p -E --reject=*logout*,*redirect* --header='Accept-Language: fr' --user='u' --password='p' --load-cookies='cookies.txt' http://mypreprod.server.com/

I won’t try to replace wget’s documentation, but basically, -r for recursive ensures all the pages are downloaded, -k transforms all the links to point to their local copy instead of the online version, -m and -p and there to ensure all the assets required for each page are downloaded. With --reject I list the pages I don’t want crawled, here some specific pages that would make the crawl loop forever.

The interesting bit is the cookies part. Here, I open a cookie that contains my session as an admin on the website, tell wget to use that to browse the website. This way all the pages (including admin pages) are downloaded, but also user profile editing forms and all that.

Php to sort all that

Once all my pages are downloaded (which takes roughly 10 minutes), using php I get all the *.html pages. This is done using this wonderful snippet :

if ( ! function_exists('glob_recursive'))
{
    // Does not support flag GLOB_BRACE

    function glob_recursive($pattern, $flags = 0)
    {
        $files = glob($pattern, $flags);

        foreach (glob(dirname($pattern).'/*', GLOB_ONLYDIR|GLOB_NOSORT) as $dir)
        {
            $files = array_merge($files, glob_recursive($dir.'/'.basename($pattern), $flags));
        }

        return $files;
    }
}

Using a recursive version of glob, I can get all the html files of my website and loop through them.

The html template class

On each page, the html tag has a class that contains a slug of the controller and action that generated the page.

This is something I especially enjoyed working with drupal : elements provide a lot of information through classes. And it comes in handy here as well.

With a simple regex, while looping through my html files in Php, I check the class on the html tag, if it’s the first time this class is seen during the loop, it’s stored in an array along with the path to the file.

A list of files and classes, now what ?

Well now I can already start testing using this list, if everything went right, it should contain an example of each controller-action of my website. Of course there are special cases that need to be taken care of, but that’s still much closer to exhaustive than clicking around looking for bugs to show up.

Taking that a step further

Since I’m jumping in a huge pile of code I don’t really know, I wanted to take screenshots of all the pages of the website for reference, to give me some kind of bird-eye view of the overal project.

No problem, I plugged some phantomjs script in my php to make it generate a screenshot of each html-class unique page.

Improvements

So far, the script does :

  • Empty folders (to remove previously generated pages and screenshots)
  • exec() the wget command
  • Do the path/html-class pairing and sorting
  • Take the screenshots
  • Copy all that in dropbox for coworkers to enjoy the screenshots as well

I plan on adding short scripts to make it run tests (such as presence of metadata description, alt attributes on all images, that kind of stuff), I’m trying to figure out a better way than wget to crawl the website, something faster. I need to manually export the cookies from Firefox before I run the script, preventing me from running that as a cron. Shouldn’t be too difficult to do using save-cookies with wget, but it’s still not done.

Bonus Points

Most of my time is spent fixing bugs spotted by coworkers and doing improvements my bosses ask for. If I’m not doing one or the other, they can’t really know what I’m doing and why. Using simple metrics, such as the number of bugs nobody had found that I fixed today (14, my spreadsheet says so) is a simple way to explain that that time wasn’t spent doing nothing, it was spent ensuring that previous bug fixes and improvements weren’t ruining the experience of other visitors.

A few of my coworkers found the list of pages to test really useful. Not all are developers, but it looks like it can help anyone to have some kind of list of all the different pages in the website, and the number of pages is unexpectedly big, I was surprised.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>