It’s Only Natural to Compare – Comparison Website Technology

One of the biggest success stories of internet business in the past 5 years is the growth of shopping comparison websites in their various forms. Whether you’re buying a new TV, an insurance policy or a hotel there’s likely to be a comparison site out there that can help you find the lowest price from a number of different providers.

These sites, which are mostly financed on a CPA basis, can be hugely lucrative although research has shown that most consumers don’t seem to know or care how they are financed. In the UK, comparison sites for financial products like insurance, credit cards and loans have become such big business the largest players have become bigger and more profitable than many of the companies they advertise, leading to some high profile acquisitions by the cash rich comparison sites like’s buyout of MoneySavingExpert for £87 million and which has be sold several times (most recently at a substantial loss).

The really juice thing about comparison sites from a marketers perspective is that when a comparison site works well it provides an invaluable service to consumers, potentially saving them a huge amount of time and money. This means very high conversion rates on affiliate referrals and that most precious of assets — positive word of mouth.

So comparing is not only natural, but also very good business — lets look at how you can launch your own comparison service to profit from this fast moving marketing machine.


Although on the surface comparison sites seem very simple to use, the technology behind them is often complex. Particularly in financial services where there may be hundreds of suppliers to return pricing from, complex quote systems and compliance hoops to jump through.  The biggest comparison sites have large in-house technical teams creating a barrier to entry for smaller operators. Bear this in mind if you’re thinking of building your own comparison site.

There are essentially 4 types of systems you can use to display product data and pricing on your comparison site.

Data feeds

Data feeds provided by merchants (i.e. online shops) are the simplest way of populating your comparison site with products and prices. Data feeds are usually formatted as XML or CSV files and you can import the feed into your on database in order to return pricing on your website. Getting hold of affiliate data feeds is usually as simple as signing up to the different merchants affiliate programs and some affiliate networks like Affiliate Window let you create a single feed with products from multiple merchants. The issue with using data feeds to power product comparison sites usually comes when trying to match up fields provided by different merchants to present data side by side. What you’ll often find is that 2 merchants will call the same product by a different name – for example Amazon might call a product “Tron: The Movie”, HMV might call it “Tron DVD” and might call it “Tron – The Movie DVD” so your system needs to map these 3 products and recognise that they’re the same thing and should be displayed on the ame page with prices from the different merchants. Sound simple? Well try doing it with 1,000,000 products from 1,000 different suppliers each with different datafeed formats and it soon gets more complicated!

Screen scraping

Before the days of XML screen scraping was the most common way of gathering data for your comparison site. Screen scraping involves sending an automated request (a scraper) to the merchants website in order to fetch information like pricing before returning it to your on website where its shown to your visitor. The problem with screen scraping is it can be a hugely intensive process if you want to compare prices from tens or hundreds of sites that a huge number of screen scraping programs you’ll need to maintain and run. Screen scraping is still used by some comparison sites, especially in travel and financial services where data feeds often aren’t available. Screen Scraping tool Mozenda is worth a look if you’re planning on going down the screen scraping route.

Merchant managed

If your comparison site is comparing products in a very niche area, or if you’re able to get a strong foothold on the market you may be able to get advertisers (merchants) to manage their own accounts and update prices directly on your system. For example Skip and Bin, a hyper niche comparison site for UK skip hire operators has a merchant management system where skip hire companies have their own login where they can go in and update their prices manually. The downside of this system is that it requires suppliers to be co-operative with keeping their prices up to date — this sort of system is unlikely to work with large sets of products.

Web services — API’s and Whitelabelling

Perhaps the most appealing option for new entrants to the comparison business is to use preformatted data provided by another comparison site in order to power your website. For example HotelsCombined run an affiliate program which gives affiliates access to their pricing information from over 30 different hotel booking sites — this means as an affiliate you only need to work with one technology (i.e. the HotelsCombined system) and your visitors can benefit from the technology which is managed by a larger team. The big drawback of this kind of implementation is that commission is likely to be lower- because the web service provider (in the example HotelsCombined) is going to take a share of the revenue. You’re also not going to build as valuable a business if you’re not developing your own technology which would make its potential sales price lower further down the line. However this sort of arrangement can be great for newbie’s who want to develop a site quickly.

For internet marketers, comparison websites undoubtedly present a huge opportunity — if you’re able to leverage the right technology to develop a site which helps consumers find the products or services they want for less then there’s a lot of potential to make money from affiliate referrals but the task of integrating data from multiple sources shouldn’t be undertaken lightly.

John McElborough specialises in affiliate technology and internet marketing, as well as writing regularly about trends in web technology he consults for a number of affiliate marketing companies in the UK.