Evolution of LinkCrawler

The beginning

LinkCrawler born on May 2012, it was the result of the need for a tool that properly analyzes a complete site and detect broken links. The idea was to create a basic NetBeans Swing Application capable to perform such duty.

LinkCrawler v1.0

Click here to expand







While not perfect, the application was capable to crawl an entire site easily, but with errors and slow performance. The Crawl Engine was based on Selenium, so it was depending of an installed browser to work properly.

Anecdote: The usage of plain HTTP connection was considered but, it was replaced in a later decision with Selenium and eventually with HTMLUnit.

Moving to version 2.3.0

After testing the application in the field, I did add new features based on certain scenarios like “Detection of broken images”, “Depth Level selection” and a little XML Sitemap Validator. Personally those feature were a big challenge to the engine, so I did change the Selenium approach and adopted HTMLUnit.

HTMLUnit was clearly a better approach, it does not depends on local browsers, less java dependencies and quite faster than selenium.

LinkCrawler v2.3

Click here to expand







But there was a problem with v2.3.0, the size of package: 30.2Mb. A real problem for such small application.

Maven to the rescue, v3.0.0 rises!!

After an inactivity period, I was investigating several techniques and strategies for software development, until a partner came up with the Maven idea. The Maven itself was brilliant, it comes features like  dependency management, easy to compile, build and deploy, free, and finally, anyone can download the source and continue the work on any Development GUI. The path was clear, so I started to remove all netbeans dependencies (Swing application libs and a few proprietary stuff) and migrate the source into a standard Java Swing application. The result: a 11.8Mb package.

3.0.0 went with several bug fixes and with just the required dependencies, maven did handle the job very well, here is the pom.xml of the LinkCrawler 3.0.0.


The pom includes goals for cobertura, test and packaging.

The new UI is almost ready

After reading some comments from some of the users of LinkCrawler, I took the liberty to start the creation of a new UI, with the objective to make it easy to read, dynamic and fully compatible with other operating systems like Mac OS and other unix-based systems. So far this is new UI:

Click to Expand

Click to Expand











While keeping the controls in the same place, I did add some controls provided by the JavaFX, one is the “Statistics” sections, which provides a Pie Chart, the chart will improve the perception of what’s going on, and a dynamic table that will be populated whenever a link is analyzed.

The new version will be available soon, but in case that you want to try it out, you can use a nightly build from here.



Web Developer, QA Engineer and Gamer Enthusiast, Developer of Linkcrawler and Father of a future computer ninja. Follow me on twitter @darkshadowx07



Pin It on Pinterest

Share This
Optimization WordPress Plugins & Solutions by W3 EDGE