Why must we automate visual testing
Now, visual testing is probably a new concept for the majority of us.
I know that when I first started out, I have never really done automated Visual Validation simply because all the ways of doing it, doing something like pixel to pixel comparison was not robust enough, it was very flaky and you need a tolerance and it didn’t provide any good results.
Now, with the addition of Applitools, visual validation is something that is critical and is extremely important. Once you start utilizing it, you start finding out how many visual bugs there can actually be in different kinds of web applications. I wanted to show you some examples of real visual bugs that have happened to real web applications.
Examples of visual bugs
Check out this bug from UPS website on a mobile device, you can see how you actually can’t read anything that’s going on here. These two areas have, kind of, called us together. Now, this is unreadable, that is a visual bug and it’s very easy to catch this bug using Applitools.
Here is another visual bug from an online newspaper. You can see that their text actually wraps into the next and covers up the very first line of this text right here.
It’s really funny because as I’m working through teaching this tutorial, I’m actually discovering a ton of visual bugs on my own website which is great because I can show those examples. But, it kind of sucks because I realized that I need to do a lot more work to make my website more visually and user-friendly.
Take a look at this, see how in this specific resolution, which is not a bad resolution, but it’s smaller than I’m used to. I’m used to looking at in 1080p, the website actually starts to put all of the menu options on top of my logo. You can’t even read it.
This icon here as well, I think, I don’t think that’s a good place for it because there’s a search icon and it’s kind of in the middle of a random place. If anything, maybe, it should be like somewhere over here, but again a real visual bug that I’m actually having problems with.
Why do visual bugs happen?
So, why do visual bugs actually happen?
It’s just simply a matter of the test traceability matrix, a single web page can have hundreds of different combinations. You can have multiple different operating systems Android, iOS, Windows. You can have different kinds of operating system versions, right? Lollipop, iOS 10, 11 and so on.
Now, you have to start testing your web application on all of these different devices, in all of the different operating system versions. You also have to take into consideration the different screen resolutions. What I showed you with my website was actually just a different screen resolution for the same exact website. Even though it’s always on the same exact browser, the same exact operating system, the different screen resolution caused a bug on UltimateQA.com.
You also have to take into consideration all of the different font sizes and how they will be reflected on different combinations of environments.
There are even different languages such as English, German, French, that you have to take into consideration. How all of that will look for just a single web page.
My Ultimate QA homepage, as you’re going to see in the future, we’re going to test it on multiple different resolutions. On each resolution, it looks different because my website is responsive. We have to analyze whether on each of those resolutions the website is actually looking good and if it has any bugs.
It just becomes simply too much to test all of this. Even in an automated functional testing perspective, it is way too hard and way too much. That’s why visual bugs happen because we don’t have the coverage for all of these different devices and matrices.
Automating Visual Checks
That is why you actually must automate visual checks.
Think about how you would actually automate all of the functional checks. Using any tool, right? Selenium WebDriver, coded UI or whatever tool you’re using. How would you actually check that all of the elements and the entire webpage looks as it should?
For example, this element over here, you can see that it’s not very readable.
First of all, that’s a bug that I need to fix on my end because I didn’t test the application and this resolution. I’m always looking it on a giant 1080p screen. To me, the webpage looks absolutely fine.
However, in this other resolution, you can see that you can’t even read the logo at the bottom. It looks kind of pixelated, that isn’t good.
How would we actually test it using Selenium, right?
You have to get the logo, you have to check its resolution, but even if you get its resolution. I guess at some point you have to figure out what is the expected resolution and what is the expected size of the logo. Then, once you know that, you can compare it.
Also, how about this for example?
Is this the right text? Is it the right text size? Is it the right text color?
There are so many things to check about each of these elements, even this button, right?
Is it the right color? Does it have the right text? Is it in the right location?
This is the same exact page over here, in a smaller resolution. Take a look at all of the problems that you can see.
Look at the logo in the upper left-hand corner, for example. It is now smaller and totally unreadable. Absolutely needs to be fixed.
How about that white space in the center over here? Can you see that that is definitely a bug? I had no idea that it even existed until I opened up this page in a smaller resolution.
Now, the question is, how would you? Think about it! How would you even automate this?
If you can’t figure out a way to test for this gap between the two different sections, then does that mean that you need to automate this for testing every single gap between every single section? Okay, that’s pretty complicated, right? So many lines of code.
There’s also stuff like this, where you have to validate that, this is the right icon that appears here and it is it the right color.
How about like this gap for example? Did you know that gaps are extremely visually important? I had no idea until I’ve been talking to my user experience buddies and gaps like this are very important.
In fact, what’s even important is spacing between the letters, right? Because imagine, if the letters are too closely spaced, things like this, the “P” will start overlapping with the “H”, so things will become unreadable.
In fact, something with the appropriate space is much easier to read, than something with an incorrect space. I actually worked with my UX buddies to design something like this, to make it look better and easier to read.
But, how do you actually automate and test all of that using functional testing and Selenium-WebDriver? Can you imagine how many thousands of lines of code it will take to check every single element for every single CSS property?
It would be insane!
That’s why Applitools comes in and helps by making it extremely easy with a single method that can second check an entire webpage.
Benefits of Applitools
Now, you may be asking, why do we want to use Applitools?
There are so many other tools out there, why should we even do Applitools and use Visual Validation?
Applitools VV is very stable
Well, for me, one of the very main benefits, and if you worked with test automation for any period of time, even couple weeks, you’re going to know this, especially the UI automation, that it is extremely flaky, right? It is extremely hard to get an automated stable functional UI test but with Applitools and visual validation, it is very stable.
It is the most stable type of UI test automation that I have ever done in my career.
In fact, anytime that I run about 9,000 automated Visual Validations, every two weeks, none of those ever come back as a false positive or a false negative.
We get failures and it’s either a real bug or it’s a requirements update that just needs to be made. Then we change a baseline, but I’ve never had a false positive where something fails. It really shouldn’t be looked at, it was just a matter of a problem and the test.
That’s benefit number 1! VV will stand for Visual Validation. Very stable!
Why does this occur? Well, it occurs because if you look at this HTML page over here, my beautiful HTML page that I drew. What can happen a lot of times on different browsers and operating systems? Elements can shift, right? Maybe this element can shift over.
Maybe the text in here could get messed up. For whatever reason, maybe this image over here can totally disappear. Any kind of that stuff or even maybe style can change, now instead of a pink box it becomes a black box. I made a very weird like that so you can see that it overlaps.
If you wanted to actually automate the testing of this single HTML page. What would you need to do?
Well, you would probably need to do something like, step one, open the page, right?
Using Selenium. Step two, you need to find an element. Step three, what would you do? You could get the CSS style of that element.
You know you could get attribute and then maybe it’s CSS. Then you can assert on that style and, maybe, that is referring to the font of this item right here. This test is, for me, the font of this item. You can assert that it matches what is expected, but the problem with that is that it’s so extensive and it would require so many automated tests in order to make all of this happen.
The Applitools has a very smart algorithm that is able to do all of this. With a single method, it can validate the entire page to make sure that, whatever is on the page, didn’t change compared to the baseline. It has a very smart algorithm that only detects what is visible to the human author. If something shifts by a pixel or two, it’s not going to detect it, if it is not visible to the human eye.
However, if this text, for example, changes from Times New Roman to Sans-Serif, for example. Then, it will detect it and it will let you know. That it’s up to you to decide what you do with it. Because of that, the Visual Validation is really stable, because you don’t really worried about locating any elements. You’re simply looking at the entire page and checking the state of the entire page.
Easy to Use
With this, what that means, it leads to the second benefit of using Applitools. That is, it’s very easy to insert into your current automated visual, into your automated functional test.
First, it’s easy to use, because it’s usually a single method that you need to plug it into your context. For example, if we were working on this page here, you would open the page and you wouldn’t even need to find an element, you wouldn’t even need to do any of this.
Actually, if you were just using Applitools, what you would do is, you would insert another method here, called CheckWindow. There are several other methods that you can insert, but in this case, we can do CheckWindow.
You would just insert that in this step and so now, really, your test is only two lines of code. Instead of all of these four lines of code, your test will only be two lines of code. You open the page and you visually validate.
Then you’re done!
Not only that, it is it drastically reduces how much code you would have to write. Because, remember, imagine if you have to visually validate, if you had to automate all of these elements, for their different attributes, right?
You can have stuff like font, color, border, and you get the picture, right? So many elements to validate on just one page and Applitools can do it all with one single method.
That’s why it’s so easy to use and so easily plugged into your current automated functional tests.
Applitools has a smart algorithm
Another benefit, I briefly mentioned it but I want to cover it more in depth now. Applitools actually has a very smart algorithm for identifying whether the current page matches to the baseline.
A lot of people will be here, Visual Validation right away. They think, oh pixel to pixel comparison. It’s so flaky or I guess you have to set a tolerance and, maybe, sometimes you’ll get false positives, sometimes you’ll get false negatives. No, but it doesn’t work like that, it’s much smarter than that.
There is no tolerance, there are different levels of the Visual Validations you can do. You don’t have to worry about any tolerance, you know? 75%, 90% tolerance, pixel to pixel problems. No, it’s really smart.
I just forgot how to spell algorithm. I think that’s right! For some reason, doesn’t seem right. Anyway, that’s supposed to be “algorithm”.
How does the algorithm tell you if the current page has failed compared to the baseline?
Well, if a shift on the page has occurred and it’s visible to the human eye, Applitools will spot it for you.
For example, if this has shifted in any way that is visible, Applitools will tell you. However, if this is moved by like a pixel, which may happen from device to device. Different devices may, you know, cut off a pixel here or add an extra pixel there.
Maybe there’s different coloring, right? A 4k display versus a 1080p display might be a little bit different. Because of that, those kinds of changes would not be spotted to you by Applitols, as a result, avoiding the false positives.
But, if it is visible to the human eye, it will spot it for you and let you know “hey, there’s a failure!”. Figure out whether that’s a bug or you need to update your baseline.
Easy integrations
The final benefit of Applitools, that I really like, is it plugs into so many different tools and environments that you’ve already used.
It plugs into all flavors of Selenium. Well, I’m gonna say everywhere. It may not be 100% everywhere but it’s pretty much anything that you really want to use, you know?
For example, it does all flavors of Selenium WebDriver. All bindings, you know, Java, C#, Ruby, Python, whatever. It plugs into even stuff like UFT, not sure if you heard of it, leanFT, that’s the new version of UFT.
It plugs into stuff like coded UI, you can use it in technologies like Appium. You can even use it in continuous integration environments like TeamCity, that`s not how you spell “Team”, Jenkins.
It basically plugs in everywhere that you need it.
Therefore, it is extremely easy to use and their API is easily reusable across all the different platforms.
When you understand it in one, you basically understand it in all of them.
That’s why I really enjoy Applitools and I’m very impressed by everything they’ve been doing.
Your Turn
Have you tried manually testing your website for visual bugs and errors? How do you find it? Or have you tried automating visual validations using Applitools?
Let me know in the comments.