Research shows that most employee rating systems are based on false data.
Have you ever wondered if employee rating systems really work? Ever been suspicious that bosses are underrating their managers’ performance?
If so, research conducted over the past fifteen years may have finally confirmed your doubts. The Harvard Business Review has highlighted a rating phenomenon that is widely discussed in academic circles, yet often goes unnoticed in the business world.
Due to a bias called the Idiosyncratic Rater Effect, as The Marcus Buckingham Company, this research has found that HR rating systems very often reveal more about the rater than about the ratee.
Who Is the System Evaluating?
The three largest studies of multi-source ratings cited by the HBR revealed that 61% of any given rating depended on the personal biases of the rater. This is a problem in a system where promotions and other important HR procedures are often completely reliant on peer and management employee ratings.
The vast majority of organisations run under the assumption that people, with training and practice, become reliable raters of other workers. This has obviously proven to be very much not the case, with the 61% bias figure impacting results no matter how “experienced” the rater is.
This prejudice has to do with whether the personal quality in question is possessed by the person rating it and how important that quality is to them. Indeed, when one closely examines the questions, there are far too many variables in play.
As HBR suggests, managers should try rating an employee’s strategic thinking on a scale of 1 to 5. But first, ask yourself what exactly “strategic thinking” is. And what is the difference between “frequently” (score of 4) and “constantly” (5)? How much weight should an HR official put on a manager’s ability to distinguish between “frequently” and “constantly,” or their ability to define “strategic thinking”? As you can see, those inherent biases just keep popping up – no matter the questions or rating system.
Why Hasn’t Anything Changed?
In fact, the study by the American Psychological Society that first revealed the 61% bias rate was conducted over ten years ago. Then in 2010, a follow-up study by Personnel Psychology confirmed the existence of the Idiosyncratic Rater Effect. And yet businesses still have yet to adapt to what the world of psychometrics accepted as necessary changes to employee evaluations long ago.
Perhaps this is understandable, though: the Effect reveals a deeply flawed structure of business based on the input of unreliable data – one that would require a complete overhaul to correct. Most businesses aren’t looking to make that kind of drastic change based on the word of a few outside sources.
What Employee Evaluations Should Look Like
If reform does eventually come to pass, even through incremental change, what’s most important is that any new system starts with more reliable data input. This adjustment could take a number of differ forms and will likely differ from company to company.
Organisations may choose to stress in-person interviews rather than peer evaluations, or if there’s a specific project that an employee would be leading, the assessment could be modeled on that project’s characteristics and results.
When delineating new evaluation methods, it’s important to focus on “what” an employee does and “how” he or she works, rather than on “when” and “where.” An older article in the Harvard Business Review reports on the ever-present bias of management by observation – in other words, judging performance by physical presence and appearance rather than by the quality of work. Progress on this issue starts with eliminating that bias.
While employee evaluation systems will always have their problems, each research report and new measure is a step towards more effective strategies. Employers and managers should never settle for a system that is not fulfilling its purpose.