What will you learn in this lecture?
In this video, we will go through the disadvantages (yes there are some) of running automated tests in Sauce Labs. In the previous tutorial, we discussed the different advantages of using Sauce Labs to run automated tests with Selenium Webdriver.
Disadvantages of Sauce Labs
So let me tell you about the disadvantages of Sauce Labs.
There are many, but just with everything, there are disadvantages and I do want to be honest and straightforward with you so you understand them upfront.
So the very first disadvantage and this one is the most important to know in my point of view is that that tests that you run on your local environment will actually take a longer time to execute than you do in your local environment in Sauce Labs.
So Sauce tests run slower than local tests.
In my experience, I’ve found that they can be slower anywhere between 50 to 100%. Now, that’s for tests that are running about for about a minute. For a minute long test, they’ll be running between 50 to 100%.
Now, obviously that’s not the case if your tests run longer than 30 minutes and I’ll explain to you why is that.
But what does this mean to you?
This means for you that for example if you have a suite of automated tests, let’s say you have 3 test cases here in the suite, this box represents the suite.
And you’re going to execute them locally on your machine. If you would like to run these tests locally on your machine, what you have to do is run one test at a time.
So if you’re running this test locally, what’s actually going to happen is you gonna run for 1 test and that’s gonna be 1 minute for we assume that it’s gonna take 1 minute. Then we run test #2 which will be 1 minute once this first test finishes. Then this will be test #3 this gonna run as soon as test #2 finishes. You can’t start a test until one test finishes because you can only run a single test at a time.
I’m not going to parallelization turned on but I am assuming it for a second. So at the end of the day, will that mean that your entire test suite runtime will be 3 minutes long locally. Now in Sauce Labs, however, if you’re running the same sequence, each of these tests is going to take two minutes up to 100%. So let’s say they’re taking the maximum amount of time, two minutes.
However, because Sauce Labs allow you to run in parallel. You can turn on parallelization and so rather than executing each of this tests in sequence, you can execute them all at the same time. So this is when the test is running. So at the end of the day, these 3 tests will actually run in two minutes versus the 3 minutes that you needed here for your local environment.
So what does this mean for you? It means for you that because of this disadvantage, you gonna require at least 3 virtual machines if you are logging to speed up your suite execution time. If your suite has hundreds or thousands of tests, for example, we have thousands of tests, functional automated tests that are running. Thousands of minutes of execution time is not acceptable at all. And so we want to speed things up. And whenever you want to speed things up, it’s some point you get to the social level when the only way to speed things up is to begin running things in parallel.
Right parallelization is so cheap now, you just throw more hardware, add it and you can run as many tests in parallel.and decrease that time. However, you need at least 3 virtual machines in order for you to make more of it. Right so that we can run 3 test at the same time versus what we can do if you got 2 virtual machines, this one port because you gonna execute these 3 tests and they gonna rotate in 3 minutes.
But if you got 2 virtual machines, and you run these tests in parallel you run two tests in 2 minutes but after that you still need to execute another test which will take another minute and so then it is a total of 3 minutes and your way because you’re going to execute 2 batches versus a single batch that you can handle in your entire test. So that’s the main disadvantage.
So you may be asking now why is that? Why is it that the tests run slower in the Sauce Labs hub than they do in our local environment? Well, let me explain this to you and I think I know you will definitely understand why that is the case.
So locally if you’re running your test, let’s pretend that this right here is your CI environment. So this is your CI server which you’ve been running your test on. If you want to run those tests, it’s very easy. All you need to do is run it on CI server.
They pull up a browser, let’s make this browser call it Firefox and then they run your test. And then they communicate back with that local resource. There’s very little time here required for this communication is very low. If you think the way Sauce Labs works is you got your CI server. The CI server through the remote web driver now has to talk to the hub.
The hub may not be located where you are. You may live in Australia, the Sauce Labs hub maybe in California, just that internet travel time just like the law of physics it takes some time you can’t GPS that time. So regardless of that in the speed of the internet, there’s a period of time required for that. So there’s some time here.
Then once you even get to the hub, there’s more processing that happens. So whenever guess with hubs, Sauce Labs has read one of your config files or even in the source code, and then it has to figure out which virtual machine you would like to spin up.
And so Sauce Labs goes out, spins up a virtual machine for you a brand new clean virtual machine. And so now you can run your test.
This process is pretty fast. Maybe it takes 15-20 seconds I’m not 100% sure on the time its fast, but again it takes some more time. Let’s say it takes 15 seconds, so now you just added 15 seconds to your test time.
Now once all these happens, afterward the test still has to communicate to the hub. And these may not even be located on the same date. I don’t know I’ve been setup Sauce Labs but they might not be. And so that will take time. And then talking back to your CI server will also take more time.
So these whole transitions add about 50-100% more execution time in order for your test to execute. And so that’s how that works and that’s why that happens and that why the test is slower.
So just keep that in mind if you’re looking to improve your test suite execution time, you wanna cut it by 50%, you wanna cut it with 100%, you need many more VMs than you think. It’s not a one to one. It’s 50-100% slower for a minute test.
Now, before I mentioned it’s for a minute long test. It’s not for something like 30 mins long test Why is that? Well because if your test already is taking 30 mins, then these amount of time that required is gonna be much smaller percentage than your test. These maybe lets say its a maximum of 1 min if we’re assuming its a 100%, then if these takes 1 min and your test takes 30 mins to run you don’t need to do any extra cycles here. So it’s gonna be 31 mins long test vs. 30 mins and so that time is negotiable but you should never be having a 30 mins test run. That’s a future mistake if your doing that you do something wrong and not doing something right which has to automation especially keep it to a maximum of 1 min long, but that’s a totally different subject.
Anyways, so the second disadvantage of Sauce Labs is that it’s not free. Unfortunately, such as fantastic service cannot be free, but maybe in the future but not right now, and it does cost money.
In my point of view, this is not that big of the problem. The amount of money that it cost for you to run your own Sauce Labs Grid is miniscule to the amount of time that you would waste maintaining your own Selenium Grid. There are many managers, many bosses above you if you wanna set up your Sauce Labs Grid. So you should understand the benefits.
Let me explain to you the benefits of using something like Sauce Labs. If you wanna maintain your own grid, let’s say you can maintain 10 nodes your one engineer trying to maintain 10 nodes that have different operating systems and virtual machines and your own hub locally.
Maintaining those 10 nodes will cost you at least 15% of your time. You’ve got to maintain the browser, you’ve got maintain the operating systems, you’ve got update the operating systems, you’ve got the update the browsers, you’ve got the make sure certain browsers don’t get updated, you’ve got the right scripts to clean stuff up, it’s gonna be at least 15% of your time.
If that engineer is making about 100,000.00 a year which is again on the law and keeping everything on the law and just to see that how even the law and the scenario will workout, that means that its costing the company $15,000.00 per year to maintain you upgrade. But that’s not the worst part.
The worst part is that, there’s opportunity cost. And the opportunity cost that the engineers make missing out on, is that they can’t automate more test. They can’t improve application coverage for testing. So it’s gonna decrease test coverage, because rather than doing more test automation, helping to decrease the gap of coverage, they’re working on maintaining the Selenium Grid.
And so that again will cost a lot of money and depending on your application and how many users you have this could be worth a lot of money hundreds of thousand of dollars or maybe millions per year.
So, Sauce Labs they do have their pricing online at the enterprise pricing they ask you to call but I can say that it’s definitely less than these per year to maintain a grid of about a 10 virtual machines and you get enough minutes to educate you automated test you don’t have to worry on anything and they will maintain the whole grid for you.
They have probably hundreds of browser Os combinations you can run on mobile like iOs and Android, you can run on mobile emulators, and you can run on any operating systems and browsers that you want, and they maintain that all for you, they store all those records and you get all that logging and the video capture and the screenshot capture out of the box.
So these is not really disadvantage for me, these totally negotiable I would always get Sauce Labs over maintaining my own selenium grid, but I just want you to make aware.
See you on the next video..
In the next tutorial, you will learn how to sign up for a Sauce Labs free trial.
What will you learn in this course?
This course is an introduction to Sauce Labs with Selenium webdriver.
Taught by Nikolay Advolodin, CEO of www.ultimateqa.com. He had been teaching test automation for 5 years and is the author of top rated online courses to 50,000 students in over 100 countries around the globe.
In this course from Ultimate QA, you will learn:
– Why Sauce Labs is an amazing test automation tool.
– Overview of Sauce Labs.
– Sauce Labs advantages and disadvantages.
– How to analyze results.
– How to perform live debugging.
At the end of the course, you will take a final quiz along with the answers to the quiz questions for better understanding.