WebDriver and BrowserMob Proxy


The execution of web pages generally involves three tasks: fetching resources, page layout and rendering, and JavaScript execution. Apart from these, sometimes websites need to track user behaviors so they need many third party integrations and this may affect page load time. Also, we can have different problems like javascript, instability, etc. We can filter all these things using BroserMob Proxy.

In this post, we will go over with how can we get performance related data from selenium test scenarios & integrate them with browsermob proxy.

Outline

  • Page Load & Bandwith & Latency Relation
  • What is BrowserMob ?
  • BrowserMob Features
  • What is HAR ?
  • Creating Profile
  • Creating a Selenium project using BrowserMob
  • Chrome Developer Tools

Page Load & Bandwidth & Latency Relation

2016-04-01_00-12-09

(Picture from High Performance Browser Networking)

Latency:  The time from the source sending a packet to the destination receiving it.

Bandwidth:  Maximum throughput of a logical or physical communication path.

What is BrowserMob?

BrowserMob proxy is a free tool that supports monitoring and manipulating the network traffic from web applications. You can download from https://github.com/lightbody/browsermob-proxy

Some of features

The proxy is programmatically controlled via a REST interface or by being embedded directly inside Java-based programs and unit tests. It captures performance data in the HAR format. In addition, it can actually control HTTP traffic, such as:

  • black-listing and white-listing certain URL patterns
  • simulating various bandwidth and latency
  • remapping DNS lookups
  • flushing DNS caching
  • automatic BASIC authorization

What is HAR?

The HAR format is based on JSON, as described in RFC 4627.
Har is an open format for exporting HTTP tracing information called HTTP Archive (HAR).
It is possible to generate Har files using Firefox, Firebug and NetExport together.
​The data stored as a JSON document.

What kind of information we can get from har file:

  • DNS information load time
  • ​Each object requested (load time)
  • Connection time to server
  • From server to browser (load time)
  • Object is blocked or not

Some of HAR object types log, creator, browser, pages, page timings, etc.

We can check har file results using different tools:

HAR viewer is a free and open-source (PHP + JavaScript) application, which you can host yourself.

2016-03-31_22-45-37

2016-03-31_22-47-47

The gem will start a local server and automatically open a browser tab to view the HAR file.

If you go to harviewer website, copy and past your .har file inside & you will get an interactive timeline.

2016-03-27_22-34-29

2016-03-31_22-56-50

 

2016-03-31_21-55-41

Part of my HAR file created by WebDriver.

You can format the file using Notepad++ json viewer plugin, you can get more information about har specification from here.

Installing BrowserMob Proxy

First, we need to download BrowserMob proxy from here or github & extract the content.

If you are running from the command line, make sure to run the browsermob-proxy executable with the --use-littleproxy true

Create a project using Eclipse:

Click to add external JARS, add BroswerMob Proxy and Webdriver jars.

If you’re using Maven, you can add this to your pom.xml:

check location of har file:

2016-04-02_21-09-42

Testng report result:

2016-04-02_21-10-29

You can check new browsermob proxy interface and its methods from this link 

Chrome Developer Tools: 

2016-03-21_21-11-35

  • Copy All as HAR: Copies all resources to the system clipboard as HAR data. A HAR file contains a JSON data structure.
  • Save as HAR with Content: Saves all network data to an HAR file along with each page resource.

2016-03-30_11-51-19

Chrome Developer Tools, go to Network tab, and make sure you’ve selected the Preserve log checkbox, otherwise the log is reset when you change page.

javafx

Database Operations in JavaFX

By Onur Baskirt / Apr 1, 2016 / 56 Comments
Before started this section, please check the first article and learn How to Start JAVAFX! http://www.swtestacademy.com/getting-started-with-javafx/ At first, part of JavaFX tutorial series, we created a sample JavaFX project, designed the draft version of the UI and set up an...
rest assured

REST API Testing with Rest Assured

By Onur Baskirt / Mar 8, 2016 / 32 Comments
Outline In this post, I will explain what is API and API testing, what is the difference between SOAP and REST services, and how to test REST APIs with Rest Assured Library. What is API? API stands for Application Programming...
extentreports

How to Write Effective CSS Locators

By Onur Baskirt / Oct 1, 2017 / 0 Comments
Hi all, in this tutorial, I will describe you how to write effective CSS locators to interrogate web elements for your automation projects. As a rule of thumb, your interrogation strategy should be in below order: First try to use...
extentreports

Selenium-11: Execute JavaScript with JavascriptExecutor

By Onur Baskirt / Jan 27, 2016 / 15 Comments
Outline Sometimes we cannot handle some conditions or problems with Webdriver, web controls don't react well against selenium commands. In this kind of situations, we use Javascript. It is useful for custom synchronizations, hide or show the web elements, change...
javafx

Getting Started with JavaFX

By Onur Baskirt / Mar 25, 2016 / 0 Comments
When I started to work in my current position, one of my task is to do manual operations for campaign products  every week. After the second week, I thought that I have to automate this task using a GUI based...
extentreports

How to Write Smart XPath Locators

By Onur Baskirt / Sep 24, 2017 / 4 Comments
Hi all, in this tutorial, I will describe you how to write smart and non-brittle XPath locators. When we write our test scripts, we generally prefer to use id, name, class, etc. these kinds of locators. However, sometimes we could not...
extentreports

Selenium-1: Quick Start to Automation with Selenium WebDriver & JAVA & JUnit & Maven & IntelliJ

By Onur Baskirt / Sep 8, 2015 / 26 Comments
Outline Selenium Webdriver is the most popular open source web test automation framework across wide range of browsers and platforms. In this tutorial you will learn how to do web test automation with Selenium Webdriver and the related tools. Audience...
page object model

Page Object Model with C#

By Ege Aksoz / Jun 18, 2017 / 8 Comments
In the previous tutorial, we’ve taken the initial steps and entered the world of automated testing. We also wrote our first automated test. From this point on, since we are not just going to write one test, we need to...
extentreports

How to Select a Date From DatePicker Using Selenium

By Onur Baskirt / Aug 13, 2016 / 6 Comments
When you need to automate a airway, hotel, or similar websites you need to deal with Datepickers and some times it is a little bit cumbersome to select a specific date on the Datepicker or calendar.  In this post, I...
extentreports

Selenium Webdriver Performance testing with Jmeter and Selenium Grid

By Ozgur Kaya / Aug 9, 2016 / 2 Comments
In this post, we will complete Selenium Webdriver Performance testing scenario using Jmeter and Selenium Grid. 1- Install Java 7 or later If necessary https://java.com/tr/download/ 2- Download latest Jmeter version 3.0 or higher. http://jmeter.apache.org/download_jmeter.cgi 3- Download Jmeter PluginsManager JAR file and...
By | 2017-01-13T21:33:42+00:00 April 4th, 2016|Performance Testing, Selenium|19 Comments

About the Author:

Onur Yazir
Onur YAZIR is a Senior Software Testing & Deployment Engineer 7+ years of experience. He has worked at Huawei, Ericsson, Teknosa as Software Test Engineer. After that, he worked as DevOps Engineer at London-based fintech company EFT Software. Now, he is a DevOps Engineer at Gulf News Dubai Office. You can find detailed information about him on his linked-in page.

19 Comments

  1. keshav gaur December 27, 2016 at 11:30 am - Reply

    I am using browser-mob with Nightwatch js, Is there any way by which I can start my server directly from my tests?

  2. Harish Mudaliar March 16, 2017 at 3:40 pm - Reply

    Yes I got the HAR file but how will I parse it in java??

  3. Ankit Jain April 24, 2017 at 12:14 pm - Reply

    I got the har file from browsermob but unable to convert that har file to JMX.
    Can you please provide me solution for the same.

  4. Ankit Jain April 25, 2017 at 7:45 am - Reply

    When I tried convert Har to Jmx through blazemeter then it’s showing me “Fatal error during conversion:
    Failed to validate json file SeleniumEasy.har: check file format”.
    Also getting below error logs:

    2017/04/25 04:40:08 INFO – com.blazemeter.utils.Configuration: Configuration was loaded…
    2017/04/25 04:40:08 INFO – com.blazemeter.Main: Log file=/var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.log
    2017/04/25 04:40:08 INFO – com.blazemeter.Main: Input file=/var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.har
    2017/04/25 04:40:08 INFO – com.blazemeter.Main: Output file=/var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.jmx
    2017/04/25 04:40:08 INFO – com.blazemeter.utils.Utils: Trying to detect input file type vie mime_type….
    2017/04/25 04:40:08 INFO – com.blazemeter.utils.Utils: Detected input file type=text/html
    2017/04/25 04:40:08 ERROR – com.blazemeter.Main: Failed to start converting: com.github.fge.jsonschema.core.exceptions.ProcessingException: fatal: Failed to find jsonSchema for /var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.har
    level: “fatal”

    at com.blazemeter.utils.JsonFormatDetector.converterType(JsonFormatDetector.java:187)
    at com.blazemeter.utils.Utils.converterType(Utils.java:78)
    at com.blazemeter.utils.Utils.converter(Utils.java:187)
    at com.blazemeter.Main.main(Main.java:83)
    at com.blazemeter.Conversion$ConversionTask.call(Conversion.java:56)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

    Can you please help me on the same.

    • Onur Baskirt
      Onur Baskirt April 25, 2017 at 10:15 pm - Reply

      It is better to ask Blazemeter support. 🙁 This part has a problem. It can not find json schema.

      com.github.fge.jsonschema.core.exceptions.ProcessingException: fatal: Failed to find jsonSchema for /var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.har
      level: “fatal”

  5. ABQAR June 2, 2017 at 2:50 pm - Reply

    Thank you for your helpful Tutorial, I did as you explain but I get this it fail up in the first Test,

    AILED CONFIGURATION: @BeforeClass setup
    java.lang.NoSuchMethodError: com.google.common.io.Closeables.closeQuietly(Ljava/io/Closeable;)V
    at org.openqa.selenium.firefox.internal.ClasspathExtension.writeTo(ClasspathExtension.java:62)
    at org.openqa.selenium.firefox.FirefoxProfile.installExtensions(FirefoxProfile.java:527)
    at org.openqa.selenium.firefox.FirefoxProfile.layoutOnDisk(FirefoxProfile.java:505)
    at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.start(NewProfileExtensionConnection.java:75)
    at org.openqa.selenium.firefox.FirefoxDriver.startClient(FirefoxDriver.java:149)
    at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.java:72)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:128)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:87)
    at com.proxy.BrowserMobProxy.MobProxy.setup(MobProxy.java:37)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
    at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:551)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138)
    at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107)
    at org.testng.TestRunner.privateRun(TestRunner.java:768)
    at org.testng.TestRunner.run(TestRunner.java:617)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
    at org.testng.SuiteRunner.run(SuiteRunner.java:240)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:87)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1188)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1113)
    at org.testng.TestNG.run(TestNG.java:1025)
    at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:230)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:76)

    I need your help please ! Thank you

  6. ABQAR June 6, 2017 at 2:03 pm - Reply

    Thank your for yout answer I fixed this problem then I get exception on @Before test, The problem is surely in the seleniumCapabilities, it can’t accept this constructor driver = new FirefoxDriver(seleniumCapabilities);

    [TestNG] Running:
    C:\Users\Hamza Abkar\AppData\Local\Temp\testng-eclipse–1727457014\testng-customsuite.xml

    FAILED CONFIGURATION: @BeforeClass setup
    org.openqa.selenium.SessionNotCreatedException: InvalidArgumentError: Expected [object Undefined] undefined to be an integer
    Build info: version: ‘3.4.0’, revision: ‘unknown’, time: ‘unknown’
    System info: host: ‘LAPTOP-CE1IQ7E0’, ip: ‘192.168.1.95’, os.name: ‘Windows 10’, os.arch: ‘amd64’, os.version: ‘10.0’, java.version: ‘1.8.0_121’
    Driver info: driver.version: FirefoxDriver
    remote stacktrace: stack backtrace:
    0: 0x48916f –
    1: 0x489f59 –
    2: 0x439ced –
    3: 0x4473ea –
    4: 0x445128 –
    5: 0x41dbb1 –
    6: 0x407997 –
    7: 0x6bbb39 –
    8: 0x415b39 –
    9: 0x6b6043 –
    10: 0x7ffcdc038364 – BaseThreadInitThunk
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.openqa.selenium.remote.W3CHandshakeResponse.lambda$new$0(W3CHandshakeResponse.java:57)
    at org.openqa.selenium.remote.W3CHandshakeResponse.lambda$getResponseFunction$2(W3CHandshakeResponse.java:104)
    at org.openqa.selenium.remote.ProtocolHandshake.lambda$createSession$22(ProtocolHandshake.java:365)
    at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
    at java.util.Spliterators$ArraySpliterator.tryAdvance(Unknown Source)
    at java.util.stream.ReferencePipeline.forEachWithCancel(Unknown Source)
    at java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown Source)
    at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
    at java.util.stream.FindOps$FindOp.evaluateSequential(Unknown Source)
    at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    at java.util.stream.ReferencePipeline.findFirst(Unknown Source)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:368)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:159)
    at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:142)
    at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:82)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:637)
    at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:250)
    at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:236)
    at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.java:137)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:191)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:108)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:137)
    at com.proxy.BrowserMobProxy.MobProxy.setup(MobProxy.java:43)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
    at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:551)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138)
    at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107)
    at org.testng.TestRunner.privateRun(TestRunner.java:768)
    at org.testng.TestRunner.run(TestRunner.java:617)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
    at org.testng.SuiteRunner.run(SuiteRunner.java:240)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:87)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1188)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1113)
    at org.testng.TestNG.run(TestNG.java:1025)
    at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:230)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:76)

    SKIPPED CONFIGURATION: @AfterClass shutdown
    SKIPPED: teknosa_test1

    ===============================================
    Default test
    Tests run: 1, Failures: 0, Skips: 1
    Configuration Failures: 1, Skips: 1
    ===============================================

    ===============================================
    Default suite
    Total tests run: 1, Failures: 0, Skips: 1
    Configuration Failures: 1, Skips: 1
    ===============================================

    [TestNG] Time taken by [TestListenerAdapter] Passed:0 Failed:0 Skipped:0]: 22 ms
    [TestNG] Time taken by org.testng.reporters.XMLReporter@4909b8da: 13 ms
    [TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@60addb54: 87 ms
    [TestNG] Time taken by org.testng.reporters.jq.Main@614c5515: 58 ms
    [TestNG] Time taken by org.testng.reporters.JUnitReportReporter@6df97b55: 6 ms
    [TestNG] Time taken by org.testng.reporters.EmailableReporter@6537cf78: 5 ms

    • Marina July 20, 2017 at 12:49 pm - Reply

      I faced the same problem, did you find the solution finally?

  7. Paresh Kalinani June 10, 2017 at 6:11 pm - Reply

    This worked flawlessly for me!

    package makemyhar;
    import java.io.FileOutputStream;
    import java.io.IOException;
    import java.util.ArrayList;
    import net.lightbody.bmp.BrowserMobProxy;
    import net.lightbody.bmp.BrowserMobProxyServer;
    import net.lightbody.bmp.core.har.Har;
    import net.lightbody.bmp.proxy.CaptureType;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.phantomjs.PhantomJSDriver;
    import org.openqa.selenium.phantomjs.PhantomJSDriverService;
    import org.openqa.selenium.remote.CapabilityType;
    import org.openqa.selenium.remote.DesiredCapabilities;

    public class MakeMyHAR {
    public static void main(String[] args) throws IOException, InterruptedException {

    //BrowserMobProxy
    BrowserMobProxy server = new BrowserMobProxyServer();
    server.start(0);
    server.setHarCaptureTypes(CaptureType.getAllContentCaptureTypes());
    server.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
    server.newHar(“Google”);

    //PHANTOMJS_CLI_ARGS
    ArrayList cliArgsCap = new ArrayList();
    cliArgsCap.add(“–proxy=localhost:”+server.getPort());
    cliArgsCap.add(“–ignore-ssl-errors=yes”);

    //DesiredCapabilities
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
    capabilities.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);
    capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap);
    capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,”C:\\phantomjs.exe”);

    //WebDriver
    WebDriver driver = new PhantomJSDriver(capabilities);
    driver.get(“https://www.google.co.in/”);

    //HAR
    Har har = server.getHar();
    FileOutputStream fos = new FileOutputStream(“C:\\HAR-Information.har”);
    har.writeTo(fos);
    server.stop();
    }
    }

  8. bob john September 18, 2017 at 7:11 pm - Reply

    I want to you browsermobproxy in TESTNG such that in @BeforeSuite annotation/method , it will get started and n @AfterSuite annotation/ method, it will get closed.

    How can I meet my requirement ?

  9. Ravi October 15, 2017 at 9:10 am - Reply

    I want to run Browsermobproxy tool in windows 7 and Mac with selenium cucumber. please some one help how to setup Browsemobproxy and integrate with selenium cucumber

    • Onur Baskirt
      Onur Baskirt October 15, 2017 at 9:29 pm - Reply

      Didn’t you find any articles on google about it?

      • Ravi October 16, 2017 at 8:27 am - Reply

        Hi Onur Baskit,

        I found some articles on google but am facing some issues.

Leave A Comment