WebDriver and BrowserMob Proxy

The execution of web pages generally involves three tasks: fetching resources, page layout and rendering, and JavaScript execution. Apart from these, sometimes websites need to track user behaviors so they need many third party integrations and this may affect page load time. Also, we can have different problems like javascript, instability, etc. We can filter all these things using BroserMob Proxy.

In this post, we will go over with how can we get performance related data from selenium test scenarios & integrate them with browsermob proxy.

Outline

  • Page Load & Bandwith & Latency Relation
  • What is BrowserMob ?
  • BrowserMob Features
  • What is HAR ?
  • Creating Profile
  • Creating a Selenium project using BrowserMob
  • Chrome Developer Tools

Page Load & Bandwidth & Latency Relation

2016-04-01_00-12-09

(Picture from High Performance Browser Networking)

Latency:  The time from the source sending a packet to the destination receiving it.

Bandwidth:  Maximum throughput of a logical or physical communication path.

What is BrowserMob?

BrowserMob proxy is a free tool that supports monitoring and manipulating the network traffic from web applications. You can download from https://github.com/lightbody/browsermob-proxy

Some of features

The proxy is programmatically controlled via a REST interface or by being embedded directly inside Java-based programs and unit tests. It captures performance data in the HAR format. In addition, it can actually control HTTP traffic, such as:

  • black-listing and white-listing certain URL patterns
  • simulating various bandwidth and latency
  • remapping DNS lookups
  • flushing DNS caching
  • automatic BASIC authorization

What is HAR?

The HAR format is based on JSON, as described in RFC 4627.
Har is an open format for exporting HTTP tracing information called HTTP Archive (HAR).
It is possible to generate Har files using Firefox, Firebug and NetExport together.
​The data stored as a JSON document.

What kind of information we can get from har file:

  • DNS information load time
  • ​Each object requested (load time)
  • Connection time to server
  • From server to browser (load time)
  • Object is blocked or not

Some of HAR object types log, creator, browser, pages, page timings, etc.

We can check har file results using different tools:

HAR viewer is a free and open-source (PHP + JavaScript) application, which you can host yourself.

2016-03-31_22-45-37

2016-03-31_22-47-47

The gem will start a local server and automatically open a browser tab to view the HAR file.

If you go to harviewer website, copy and past your .har file inside & you will get an interactive timeline.

2016-03-27_22-34-29

2016-03-31_22-56-50

 

2016-03-31_21-55-41

Part of my HAR file created by WebDriver.

You can format the file using Notepad++ json viewer plugin, you can get more information about har specification from here.

Create a project using Eclipse:

Click to add external JARS, add BroswerMob Proxy and Webdriver jars.

If you’re using Maven, you can add this to your pom.xml:

check location of har file:

2016-04-02_21-09-42

Testng report result:

2016-04-02_21-10-29

You can check new browsermob proxy interface and its methods from this link 

Chrome Developer Tools: 

2016-03-21_21-11-35

  • Copy All as HAR: Copies all resources to the system clipboard as HAR data. A HAR file contains a JSON data structure.
  • Save as HAR with Content: Saves all network data to an HAR file along with each page resource.

2016-03-30_11-51-19

Chrome Developer Tools, go to Network tab, and make sure you’ve selected the Preserve log checkbox, otherwise the log is reset when you change page.

By |2018-12-04T22:57:27+00:00April 4th, 2016|Selenium Tutorials|24 Comments

About the Author:

Onur YAZIR is a Senior Software Testing & Deployment Engineer 7+ years of experience. He has worked at Huawei, Ericsson, Teknosa as Software Test Engineer. After that, he worked as DevOps Engineer at London-based fintech company EFT Software. Now, he is a DevOps Engineer at Gulf News Dubai Office. You can find detailed information about him on his linked-in page.

24 Comments

  1. keshav gaur December 27, 2016 at 11:30 am - Reply

    I am using browser-mob with Nightwatch js, Is there any way by which I can start my server directly from my tests?

  2. Harish Mudaliar March 16, 2017 at 3:40 pm - Reply

    Yes I got the HAR file but how will I parse it in java??

  3. Ankit Jain April 24, 2017 at 12:14 pm - Reply

    I got the har file from browsermob but unable to convert that har file to JMX.
    Can you please provide me solution for the same.

  4. Ankit Jain April 25, 2017 at 7:45 am - Reply

    When I tried convert Har to Jmx through blazemeter then it’s showing me “Fatal error during conversion:
    Failed to validate json file SeleniumEasy.har: check file format”.
    Also getting below error logs:

    2017/04/25 04:40:08 INFO – com.blazemeter.utils.Configuration: Configuration was loaded…
    2017/04/25 04:40:08 INFO – com.blazemeter.Main: Log file=/var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.log
    2017/04/25 04:40:08 INFO – com.blazemeter.Main: Input file=/var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.har
    2017/04/25 04:40:08 INFO – com.blazemeter.Main: Output file=/var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.jmx
    2017/04/25 04:40:08 INFO – com.blazemeter.utils.Utils: Trying to detect input file type vie mime_type….
    2017/04/25 04:40:08 INFO – com.blazemeter.utils.Utils: Detected input file type=text/html
    2017/04/25 04:40:08 ERROR – com.blazemeter.Main: Failed to start converting: com.github.fge.jsonschema.core.exceptions.ProcessingException: fatal: Failed to find jsonSchema for /var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.har
    level: “fatal”

    at com.blazemeter.utils.JsonFormatDetector.converterType(JsonFormatDetector.java:187)
    at com.blazemeter.utils.Utils.converterType(Utils.java:78)
    at com.blazemeter.utils.Utils.converter(Utils.java:187)
    at com.blazemeter.Main.main(Main.java:83)
    at com.blazemeter.Conversion$ConversionTask.call(Conversion.java:56)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

    Can you please help me on the same.

    • Onur Baskirt April 25, 2017 at 10:15 pm - Reply

      It is better to ask Blazemeter support. 🙁 This part has a problem. It can not find json schema.

      com.github.fge.jsonschema.core.exceptions.ProcessingException: fatal: Failed to find jsonSchema for /var/cache/tomcat8/temp/7nhbj8kaarnlk3jemcnttusson9067326786491721299/SeleniumEasy.har
      level: “fatal”

  5. ABQAR June 2, 2017 at 2:50 pm - Reply

    Thank you for your helpful Tutorial, I did as you explain but I get this it fail up in the first Test,

    AILED CONFIGURATION: @BeforeClass setup
    java.lang.NoSuchMethodError: com.google.common.io.Closeables.closeQuietly(Ljava/io/Closeable;)V
    at org.openqa.selenium.firefox.internal.ClasspathExtension.writeTo(ClasspathExtension.java:62)
    at org.openqa.selenium.firefox.FirefoxProfile.installExtensions(FirefoxProfile.java:527)
    at org.openqa.selenium.firefox.FirefoxProfile.layoutOnDisk(FirefoxProfile.java:505)
    at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.start(NewProfileExtensionConnection.java:75)
    at org.openqa.selenium.firefox.FirefoxDriver.startClient(FirefoxDriver.java:149)
    at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.java:72)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:128)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:87)
    at com.proxy.BrowserMobProxy.MobProxy.setup(MobProxy.java:37)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
    at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:551)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138)
    at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107)
    at org.testng.TestRunner.privateRun(TestRunner.java:768)
    at org.testng.TestRunner.run(TestRunner.java:617)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
    at org.testng.SuiteRunner.run(SuiteRunner.java:240)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:87)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1188)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1113)
    at org.testng.TestNG.run(TestNG.java:1025)
    at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:230)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:76)

    I need your help please ! Thank you

  6. ABQAR June 6, 2017 at 2:03 pm - Reply

    Thank your for yout answer I fixed this problem then I get exception on @Before test, The problem is surely in the seleniumCapabilities, it can’t accept this constructor driver = new FirefoxDriver(seleniumCapabilities);

    [TestNG] Running:
    C:\Users\Hamza Abkar\AppData\Local\Temp\testng-eclipse–1727457014\testng-customsuite.xml

    FAILED CONFIGURATION: @BeforeClass setup
    org.openqa.selenium.SessionNotCreatedException: InvalidArgumentError: Expected [object Undefined] undefined to be an integer
    Build info: version: ‘3.4.0’, revision: ‘unknown’, time: ‘unknown’
    System info: host: ‘LAPTOP-CE1IQ7E0’, ip: ‘192.168.1.95’, os.name: ‘Windows 10’, os.arch: ‘amd64’, os.version: ‘10.0’, java.version: ‘1.8.0_121’
    Driver info: driver.version: FirefoxDriver
    remote stacktrace: stack backtrace:
    0: 0x48916f –
    1: 0x489f59 –
    2: 0x439ced –
    3: 0x4473ea –
    4: 0x445128 –
    5: 0x41dbb1 –
    6: 0x407997 –
    7: 0x6bbb39 –
    8: 0x415b39 –
    9: 0x6b6043 –
    10: 0x7ffcdc038364 – BaseThreadInitThunk
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.openqa.selenium.remote.W3CHandshakeResponse.lambda$new$0(W3CHandshakeResponse.java:57)
    at org.openqa.selenium.remote.W3CHandshakeResponse.lambda$getResponseFunction$2(W3CHandshakeResponse.java:104)
    at org.openqa.selenium.remote.ProtocolHandshake.lambda$createSession$22(ProtocolHandshake.java:365)
    at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
    at java.util.Spliterators$ArraySpliterator.tryAdvance(Unknown Source)
    at java.util.stream.ReferencePipeline.forEachWithCancel(Unknown Source)
    at java.util.stream.AbstractPipeline.copyIntoWithCancel(Unknown Source)
    at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
    at java.util.stream.FindOps$FindOp.evaluateSequential(Unknown Source)
    at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    at java.util.stream.ReferencePipeline.findFirst(Unknown Source)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:368)
    at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:159)
    at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:142)
    at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:82)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:637)
    at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:250)
    at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:236)
    at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.java:137)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:191)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:108)
    at org.openqa.selenium.firefox.FirefoxDriver.(FirefoxDriver.java:137)
    at com.proxy.BrowserMobProxy.MobProxy.setup(MobProxy.java:43)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
    at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:551)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
    at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:138)
    at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:175)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:107)
    at org.testng.TestRunner.privateRun(TestRunner.java:768)
    at org.testng.TestRunner.run(TestRunner.java:617)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
    at org.testng.SuiteRunner.run(SuiteRunner.java:240)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:87)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1188)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1113)
    at org.testng.TestNG.run(TestNG.java:1025)
    at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:132)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:230)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:76)

    SKIPPED CONFIGURATION: @AfterClass shutdown
    SKIPPED: teknosa_test1

    ===============================================
    Default test
    Tests run: 1, Failures: 0, Skips: 1
    Configuration Failures: 1, Skips: 1
    ===============================================

    ===============================================
    Default suite
    Total tests run: 1, Failures: 0, Skips: 1
    Configuration Failures: 1, Skips: 1
    ===============================================

    [TestNG] Time taken by [TestListenerAdapter] Passed:0 Failed:0 Skipped:0]: 22 ms
    [TestNG] Time taken by [email protected]: 13 ms
    [TestNG] Time taken by [email protected]: 87 ms
    [TestNG] Time taken by [email protected]: 58 ms
    [TestNG] Time taken by [email protected]: 6 ms
    [TestNG] Time taken by [email protected]: 5 ms

    • Marina July 20, 2017 at 12:49 pm - Reply

      I faced the same problem, did you find the solution finally?

  7. Paresh Kalinani June 10, 2017 at 6:11 pm - Reply

    This worked flawlessly for me!

    package makemyhar;
    import java.io.FileOutputStream;
    import java.io.IOException;
    import java.util.ArrayList;
    import net.lightbody.bmp.BrowserMobProxy;
    import net.lightbody.bmp.BrowserMobProxyServer;
    import net.lightbody.bmp.core.har.Har;
    import net.lightbody.bmp.proxy.CaptureType;
    import org.openqa.selenium.WebDriver;
    import org.openqa.selenium.phantomjs.PhantomJSDriver;
    import org.openqa.selenium.phantomjs.PhantomJSDriverService;
    import org.openqa.selenium.remote.CapabilityType;
    import org.openqa.selenium.remote.DesiredCapabilities;

    public class MakeMyHAR {
    public static void main(String[] args) throws IOException, InterruptedException {

    //BrowserMobProxy
    BrowserMobProxy server = new BrowserMobProxyServer();
    server.start(0);
    server.setHarCaptureTypes(CaptureType.getAllContentCaptureTypes());
    server.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
    server.newHar(“Google”);

    //PHANTOMJS_CLI_ARGS
    ArrayList cliArgsCap = new ArrayList();
    cliArgsCap.add(“–proxy=localhost:”+server.getPort());
    cliArgsCap.add(“–ignore-ssl-errors=yes”);

    //DesiredCapabilities
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
    capabilities.setCapability(CapabilityType.SUPPORTS_JAVASCRIPT, true);
    capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, cliArgsCap);
    capabilities.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,”C:\\phantomjs.exe”);

    //WebDriver
    WebDriver driver = new PhantomJSDriver(capabilities);
    driver.get(“https://www.google.co.in/”);

    //HAR
    Har har = server.getHar();
    FileOutputStream fos = new FileOutputStream(“C:\\HAR-Information.har”);
    har.writeTo(fos);
    server.stop();
    }
    }

  8. bob john September 18, 2017 at 7:11 pm - Reply

    I want to you browsermobproxy in TESTNG such that in @BeforeSuite annotation/method , it will get started and n @AfterSuite annotation/ method, it will get closed.

    How can I meet my requirement ?

  9. Ravi October 15, 2017 at 9:10 am - Reply

    I want to run Browsermobproxy tool in windows 7 and Mac with selenium cucumber. please some one help how to setup Browsemobproxy and integrate with selenium cucumber

    • Onur Baskirt October 15, 2017 at 9:29 pm - Reply

      Didn’t you find any articles on google about it?

      • Ravi October 16, 2017 at 8:27 am - Reply

        Hi Onur Baskit,

        I found some articles on google but am facing some issues.

  10. kenneth villaruel December 6, 2017 at 9:02 am - Reply

    Hi guys, i’m currently using the browsermob, and i’m tracking the mixed content URL but the browsermob did not return any mixed content. mixed content are url that was is http. Right now, our website was migrated to https. Please response, thanks

    • Onur Baskirt December 6, 2017 at 8:06 pm - Reply

      I haven’t got any experience on it but I hope SW Test Academy community will help.

  11. nani April 5, 2018 at 1:14 pm - Reply

    hi onur baskirt
    can u please write an code that asserts two values in network tap using java in selenium .
    i have an requirement like ,
    consider two strings(json format) that are in different links , we need to compare them whether they are equal or not.

    • Onur Baskirt April 5, 2018 at 11:40 pm - Reply

      Nani, I really would like to help but nowadays I am super busy. If I will have time, I will help. But I will also search this on the internet. But when I read your request, I think it is better to do this with Rest-Assured. As I understand, you are trying to compare an API result and that result format is JSON. With rest-assured you can do this much easier. Please check here: http://www.swtestacademy.com/api-testing-with-rest-assured/

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.