Nov 16, 2013

TDD for Responsive Design. Or how to automate testing of website layout for different devices

Actually its quite hard to tell what Galen Framework is. It all began with the need for testing website on different browsers. I had to check whether my website is acting weird in IE, Chrome, Firefox, Opera etc. Then the Responsive Design trend came in and now suddenly I get more work. I have to stretch my browser to check if the layout is broken or not for mobile and tablet versions. And even having WebDriver and Selenium Grid at that time did not really help to solve my problem with testing. One of the ideas I tried to implement was like this: tests were making screenshots of the same page in different browsers and then later these screenshots were added to one big report. Later one of the QA engineers had to quickly go through the report and look for something suspicious. This idea didn’t live for a long time. At some point we had to throw it away. Especially it became useless when Responsive Design kicked in. I needed a special tool that would allow me to express how I see my website on different devices so later I could use it as a document as well as executable test. Thats how I came up with Galen Framework. The idea was quite simple – just take a location and dimensions of element and verify it relatively to other elements on page.

In short – Galen Framework is a special language and a tool for testing website layout in a real browser. Or even more short – Galen Framework allows to test Responsive Design and Cross-Browser Html Layout.

Galen Framework (or simply Galen) uses Selenium for opening of a test page in selected browser, changing browser window size and getting the information about the elements on page (absolute location and size).

Also with Galen you could introduce another interesting idea – TDD (Test Driven Development) for frontend. You do it in 4 simple steps: write a test code using Galen language, implement website, test your website, get the feedback and refactor. You don’t have to follow exactly the same approach as it is just a concept. But what if it would work? Lets take a look how we could do it with Galen so I can introduce you to this new tool.
Lets imagine we are sitting in the cafe thinking on a design for our website.

1. Lets make a sketch on a sheet of paper

We have to think through a design for at least 3 devices: desktop, tablet and mobile

As you see on our sketch we have defined a bit different behaviour for a mobile device.

2. Writing Galen Specs

For the sake of clarity we try to do everything in one file called homepage.spec. The first thing we are going to do is – define all objects on page for our spec.
As I already can image how I am going to build by html code I am going to put the locators for each object. Galen handles 3 types of locators: id, css and xpath. I suggest to avoid xpath expressions and use them only in rare cases when css is simply not enough.

=========================================== header id header header-logo css #header .logo header-caption css #header h1 menu id menu menu-item-* css #menu li a content id content side-panel id side-panel side-panel-caption css #side-panel h2 side-panel-links-* css #side-panel li a article id article article-caption css #article h1 article-text id article-text comments id comments footer id footer ==========================================

As you see for some objects we used * in the name. That is needed in a situation when you have a lot of similar items on page. Galen in this case will first fetch all the objects on page using the specified locator and then will construct the object references in a spec with the specified name. The * symbol will be replaced by an index of object as it appears on page (starts with 1). As an example lets take our menu items. We defined menu-item-* and this means that in the end we will get objects: menu-item-1, menu-item-2, menu-item-3, menu-item-4

Now since we are done with object definitions lets do some actual testing. I suggest to start with the simplest – test the skeleton of our website without menu items. Lets try to explain Galen how it should test our objects depending on the device type.

We are going to use 4 tags: mobile, tablet, desktop, all. The last tag means that the check should be done for any type of device. It is only for your own convenience.
Next piece of code we shall add to our file homepage.spec after the object declaration, however it is not important for Galen if it is before or after.

@ all ------------------------------------ header width: 100% of screen/width height: 100px above: menu 0px menu width: 100% of screen/width below: header 0px above: content 0px footer width: 100% of screen/width height: > 100px content inside: screen 0px left below: menu 0px @ desktop, tablet ----------------------------------- side-panel width: 300px below: menu 0px inside: screen 0px right near: content 0px right menu height: 50px footer below: content 0px @ mobile ----------------------------------- side-panel width: 100% of screen/width below: content 5px content width: 100% of screen/width above: side-panel 5px footer below: side-panel 0px

As you see for desktop and table tags the “content” object is the only one thats doesn’t have a width defined. That is because at the moment Galen has no arithmetic expressions and we cannot simple say “100% of screen/width - 300px”. But from the other side we don’t need this as we are checking that the “side-panel” is located near object “content” by 0 pixels to the right. Also we check that “content” is laying on the left side of screen and “side-panel” - on the right. In case if the width of “content” object is not enough or too much – we shall see an error right away.

The code above can also be optimized but I didn’t do so in the begging for easier understanding. But here is how we could optimize it. Just remove the redundant checks for above or below. As if we check that object A is above object B – we don’t have to check that object B is below object A. Also we can combine all objects that stretch to screen width in one single spec.

@ all ------------------------------------ header, menu, footer width: 100% of screen/width header height: 100px above: menu 0px menu height: 50px above: content 0px footer height: > 100px content inside: screen 0px left @ desktop, tablet ----------------------------------- side-panel width: 300px below: menu 0px inside: screen 0px right near: content 10px right @ mobile ----------------------------------- side-panel, content width: 100% of screen/width side-panel below: content 5px

3. Writing Html/CSS/Javascript

So as we are finished with our basic test lets do some html coding.
Here is a basic website which should behave same as we wanted in our original sketch. Try it out with this link

4. Running tests with Galen Framework

Now as we are finished with our version 1 of home page we could test it with Galen. By default Galen will use Firefox.

Important! As Galen uses Selenium there might be a situation that the last version of Firefox is not yet supported. This is one of the issues for Selenium. So if at the moment your version is not supported – you could temporarily downgrade to older version of Firefox. Once the new Selenium version is out there – Galen will be updated right away.

galen check homepage.spec --url "http://samples.galenframework.com/tutorial1/tutorial1.html" --size 1024x768 --include "all,desktop" --htmlreport desktop-reports galen check homepage.spec --url "http://samples.galenframework.com/tutorial1/tutorial1.html" --size 600x800 --include "all,tablet" --htmlreport tablet-reports galen check homepage.spec --url "http://samples.galenframework.com/tutorial1/tutorial1.html" --size 400x600 --include "all,mobile" --htmlreport mobile-reports

In the end Galen will create html reports with all checks and screenshots for all failures. Here how it looks like:

If you click on a highlighted failed spec you will get a popup with screenshot. As you see on a screenshot Galen also highlights the page elements so it is easier for you to understand whats going on the page.

As you see we have an error on the website. In the error message Galen says “side-panel” is 0px below “content” instead of 5px. Well that is because we defined the 5px margin between content and side-panel objects. We can fix that in our spec or fix the html code if we really want that margin there.
Once we are done with fixing we should run the tests one more time. But the way we did it previously doesn’t feel nice. Its not a good idea to become swamped in all kind of separate commands and reports. We need to have a way to do it simple. Galen Framework has a solution for that: Galen Test Suites. Lets make a file called galen-tutorial.test and write down the following:

@@ Parameterized | device | size | tags | | desktop | 1024x768 | all,desktop | | tablet | 600x800 | all,desktop | | mobile | 400x600 | all,mobile | Tutorial 1 Home page in ${device} device http://samples.galenframework.com/tutorial1/tutorial1.html ${size} check homepage.spec --include "${tags}"

Done. Now we can run the test with the single command and in the end we will get a single report overview. The resulting reports you can check here

galen test galen-tutorial.test --htmlreport reports

More complicated checks

As we are finished with our website skeleton and we are sure it has been tested we can proceed with more tests on a detailed level. I will try to show you more capabilities of Galen syntax.

@ all ----------------------------------- header-logo inside: header 10px left centered vertically inside: header height: 60px header-caption inside: header 10 to 20px top left near: header-logo ~ 10 px text is: My Awesome Website!

As you see we got new checks like “centered vertically inside” and “text is”. The first one is used in order to check that the logo is located vertically in the middle and inside object “header”, and the second spec is used for checking the visible text within object "header-caption"
Also you noticed the new interesting instruction “10 to 20 px”. As you might guessed – it is a range from 10 to 20 pixels within which the location of element is allowed. Also there is another symbol “~” in a range syntax. That means that we take it approximately which in the end is treated by Galen as plus-minus 2 pixels. Actually you can configure that if you want to have a bigger approximation.

Now lets go for a bit more complicated example just to show-off. Lets tell Galen that on desktop and tablet devices the menu is expected to be horizontally aligned and for mobile devices it should become a table with two columns and two rows. I will try to explain every spec in comments.

@ all -------------------------------------- # Checking that all menu items # are actually located completely inside menu menu contains: menu-item-* # Each menu item should have height of 50 pixels menu-item-* height: 50px @ desktop, tablet -------------------------------------- # Checking that every item of menu # is located next to following # item with 0 pixel margin # # Also we check that both items # are aligned horizontally # by both sides: top and left [ 1 - 3 ] menu-item-@ near: menu-item-@{+1} 0px left aligned horizontally: menu-item-@{+1} @ mobile -------------------------------------- # Each menu item should take # approximately half of the menu width menu-item-* width: 45 to 50 % of menu/width # Checking that the first column # of menu is located next to # the second column with 0 pixel margin [ 1, 3 ] menu-item-@ near: menu-item-@{+1} 0px left aligned horizontally: menu-item-@{+1} # Checking that first row # of menu (table view) is # located above the second # row with 0 pixel margin [1, 2] menu-item-@ above: menu-item-@{+2} 0px aligned vertically: menu-item-@{+2}

The construction [ 1 – 9 ] tells Galen that it should parameterize the first following object specs with the sequence of parameters: 1,2,3,4,5,6,7,8,9. And whenether it finds the “@” symbol in spec – it will put the parameter value instead. This makes it easier to control all your test objects. In the parameterization line you can also define a comma separated list of values or even mix it with ranges like this: [1, 3, 5, 10-15, 20, 21].
Also as you see we used construction @{+1}. This tells Galen to perform the simple arithmetic operation and increase the parameter value by one. This way we were able to compare previous menu item with following in a list.

So I think this is enough for the first tutorial as I don’t want to make a dump of documentation here. As you see we can test almost any element on page. However it is important to pick proper specs for your specific case so you make your tests cleaner. If you are curious what Galen is capable of please visit the official website galenframework.com. There are a lot of examples with sketches and complete explanations.

In short will try to summarize what Galen is capable of and what it can’t do yet (the missing functionaly is in the roadmap).

What Galen is capable of:

  1. Checking location of elements relatively to each other
  2. Checking visible text
  3. Run in Selenium Grid
  4. Inject custom javascript in the test page (e.g. you need to test a drop-down menu)
  5. Interact with browser with Selenium so you could get to the area of the website which is not directly accessible from a single url.
  6. Component testing. Galen can launch separate spec test per specific objects. This way you get the opportunity to perform same checks for complex object with just one line of code (e.g. comments, search results, etc).
  7. Perform corrections on fetched locations and dimensions of page objects. This might be helpful in two cases: either Selenium gives incorrect information about the object, either you want to make some kind of “guides” to which you will stick to when verifying other objects.
  8. Condition blocks. Might be helpful when you don’t know which exact object to expect on page (e.g. banners)
  9. Parameterizations of test in test-suites. You can go even further and parameterize the parameterized.

What Galen is not yet capable of:

  1. Checking the location of visible text. At the moment Galen is checking the location of the whole block and if it is an <h1> element – its width would be bigger than the width of actual text. This is still something to consider.
  2. Run in real mobile browsers. At the moment this is a problem for Galen as it relies on Selenium. And in Selenium currently there is an issue of getting locations and dimensions of elements in mobile browsers.
  3. Check the color and compare parts of images on screenshots. This is in a roadmap for future improvements.
  4. Complex arithmetic operations in parameterized checks. At the moment you can only do simple things like: @{+1}, @{*2}, @{-1}, @{/10}. But it would be nice to have something more complex.
  5. Arithmetic operation in regular checks. For instance it would be nice to have something like this: width 100% of screen/width – 10px + 10% of comments/width
  6. There is no proper installer. Galen requires Java 1.6+ but it also would be nice to make it standalone with installers for different operating systems.

The project is on the GitHub and is distributed under Apache License, Version 2.0.
Galen Framework is being maintained by one person in a free time. So it would take time to fix all the bugs you find but if you do find them please post the in GitHub Issues. You can download the Galen binary here

comments powered by Disqus