How to Crawl a Website and Validate Links by Apache JMeter

Hi,

Today i am going to tell you a solution to overcome a challenge.

We have a test case where we should validate all product categories’ links in our customer’s website. The reason behind this requests is that the client changed the CMS systems and some tags were corrupted so nobody know which link is corrupt or not.

But there are 168 main product categories and 50 subcategories in each product category. That means there are at least 8.000 links to check. Implementing this test case via any UI based framework is a pain in the ass. There’s no way you can implement a proper solution. Even if you do, performance would suffer.

So we need to make a brainstorm session to think outside the box. During that meeting we realised that corrupted links have HTTP 404 response, so we decided to switch to another approach to solve this problem.

We brought Apache JMeter into the battlefield.

And we have created script below:

Firstly, we created a CSV file containing all Main Category links. Secondly, one Thread Group will read the category link and make a get request to it. Then by using data extraction, we’ll fetch subcategory links on that page. In that web site, there are two types of products. So we create two different CSS extractor for different products. At last, by using and IF-ELSE desicion point, we navigate to the product and check HTTP Status Code. Finally we collect all the output into a “View Results in Table”

Extract Value From The Page

My CSS Extractor has the below configuration. It fetches the links by using [data-name=’brand’] selector and reads data-href attribute as the links are stored in that attribute. In case there are no links, it return  noBrand value.

At the end, CSS Extractor will save all results into brandLink variable.

Decide If There’s a Link or Not

If Controller validates if brandLink variable contains a valid link or not.

Loop Thought the Links

Then by executing a ForEach Controller, we are able to loop through all the links stored in brandLink variable. During the iteration, every value in the array list will be assigned to brandOutput variable so we can use it in an HTTP request.

Check the Results

Finally, we make a GET request to the category link.

In case JMeter gets 404, View Result Tree will mark this result with RED color. Inc ase it’s a 200, it will mark the result with GREEN color.

You can use JMeter as a functional test tool. Don’t forget, you need to choose the right tool for different situations. Always think out of the box.

[fusion_widget_area name=”avada-custom-sidebar-performancetestingwidget” title_size=”” title_color=”” background_color=”” padding_top=”” padding_right=”” padding_bottom=”” padding_left=”” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” class=”” id=”” /]

Thanks.
Canberk