Diaz Research UK and Ireland login button Global Best login button

The trusted hub for IT people and organization best practice  

Pay Data and its limits: precisely inaccurate and maybe inadequate?

March 2015

More articles

Pay data can be confusing at the best of times. It is amazing how often even experienced survey analysts and pay consultants inject spurious precision into what is emphatically NOT a precise science. ("You are paying these job-holders 9.45% below the market rate of 47,527".)

Matching errors, sampling errors, and more matching errors

Taking the most common type of pay survey - the job-matched survey - how can errors creep in?

These surveys start with the pay data providers producing a Job Profile, to which reward managers match their companies’ jobs. It is here that the first major source of error appears: job matching error. No real job matches the survey company's Job Profile precisely. Inevitably, reward mangers have to choose the Job Profile which best fits the role they want to benchmark – not an exact science! So the data that is input to survey databases is never entirely appropriate to the Jobs as described in the Profiles.

Then, of course, we need to remember the obvious: not all companies participate in all pay surveys. Pay data will always only be based on a small subset of companies that have any particular role. This leads to the second major source of error: sampling error. This is less of a problem, however, as it is better understood and can be estimated.

The third major source of error creeps in with the application of pay data. Here, more matching error is likely to occur: companies are applying best-fit survey Job Profiles to the jobs they actually have and inaccuracies are sure to abound!

These errors are amplified when it comes to IT. The nature of many IT jobs requires specialist knowledge on the part of reward managers. This is a major issue in specialist areas. Surprisingly it can also be acute in common-place jobs: even standard terms such as ‘Business Analyst’ and ‘Project Manager’ can convey staggeringly different realities in different companies. Here, much will depend on the clarity of the Job Profiles.

What does this mean?

Taken as a whole, these factors can very seriously undermine confidence in specific pay surveys, or in pay surveys in general. There is no doubt that they underlie many of the ‘discrepancies’ between data from different pay surveys, or even from the same pay survey, sampled at different times by different people.

But do these errors mean that pay data is totally meaningless? No, but they do mean that the utmost care is needed before applying pay data. There are very many potential pitfalls.

The biggest hazard lies in applying pay data in a superficial way, with no awareness of the possible inaccuracies. Sadly, too many compensation and benefits analysts are only too happy to be told a precise and - they believe! - highly accurate pay figure for a given job. After all, it makes their job easier in the short term. And many data suppliers are happy to go along with the idea that their data is both precise AND accurate. This creates problems that emerge only in the longer term.

What can you do about it?

Like explosives, fast cars, and small babies - pay surveys can be deadly in the wrong hands! So how do you ensure your pay data is being interpreted and used safely? There are two simple actions you can take to text your pay data provider:

1. Ask your pay data provider about errors or sources of error in their data - How aware are they of this as an issue?

2. Get them to estimate the level of error (even very approximately) in any comparisons they supply, in relation to jobs you actually have - Can they talk confidently and sensibly about this?

If they pretend the problem is a minor one, or if they cannot discuss the inaccuracies of pay data, be warned. You may not be getting the quality of advice you need to make good decisions.

The bit that might be missing: context!

These observations may leave the reader wondering whether there is any point whatsoever in using pay surveys. Experienced reward professionals will say there is, but only as part of a wider picture. In fact, their very inaccuracy might cause you to participate in more than one survey! Then, if two sources of market data are consistent in terms of what they are telling you about your IT pay, you can proceed with a much greater level of assurance.

In general it is wise to regard pay surveys as contributing just one part of the picture. Other sources of data that will help you to decide where your pay practices fit in the market include exit interview data, retention statistics and issues, and recruitment successes or failures. If these are mostly telling you the same thing as the market pay comparisons, you can proceed with confidence to respond to the underlying pay issues. But pay comparisons on their own should never be allowed to drive your pay policies and practices.

More articles by Diaz Research

On our site

From startup to Spotify: can corporate IT attract young talent?

From e-people to Digital people: digital skills and the skills ecosystem

Elsewhere on the web

Is JP Morgan shooting itself in the foot with its LinkedIn rules? Published by Computerworld UK

Managing your staff after outsourcing, published by Computerworld UK

Career structures for IT professionals, published by ComputerWeekly.com

Offshoring impact analysed, published by CIO.co.uk

What offshoring is really doing to jobs, published by Computerworld UK

Competency combinations: the real skills shortage, published by Computerworld UK

IT staff and bonded labour: not that different? Published by Computerworld UK

IT staff performance: ahead of the curve? Published by Computerworlk UK

Recruitment: why history graduates are as good as techies, published by Computerworld UK

Career paths and reward for internal IT consultancy roles, published by e-reward.co.uk