Psychometrics: making waves
Professor Peter Saville talks exclusively to IRS Employment Review about his latest suite of psychometric tools. We also hear how EMTEC, Thresher Group, Kimberly-Clark and Deloitte are using psychometric testing as a key part of their selection processes.
On this page:
The devil is in the detail
Culture
The big five
Valid
Secrets and lies
Putting it into practice
The last word
EMTEC
Thresher Group
Kimberly-Clark
Deloitte
Table 1: Saville Consulting Wave Professional Styles
Table 2: Predicted culture/environment fit
Box 1: History of test development.
|
Learning point
|
"We wanted to start again with a clean sheet. That's how this all started. We brought together what we learnt from developing OPQ [the Occupational Personality Questionnaire] 25 years ago, and put a team together with a blank canvas to create something innovative, something that pushed the boundaries," says Professor Peter Saville, describing how his latest product in psychometric testing came about.
Professor Saville is probably best known in the HR world as the "Saville" in "Saville and Holdsworth", the company that he and Roger Holdsworth set up 25 years ago, but which they both left in December 2002.
Now the international chair of Saville Consulting, Professor Saville goes on to recall that: "We had a hunch - call it intuition - that we could improve on the existing measures and dimensions, and then we discovered the technology to make it work."
What has emerged, some two years later, from the Saville-led laboratory is a tool that is, potentially, set to become as much of a landmark in psychometrics as its OPQ predecessor. A complex, insightful and intelligent tool, The Saville Consulting Wave Professional Styles questionnaire shares many characteristics with that of its namesake.
Essentially, the tool uses the internet to produce valid measures of personal motive, culture and talent - all in a single summary - in greater depth and through a shorter process than it was previously possible to deliver (see table 1 for more details).
In just 35 to 40 minutes it assesses an individual's motives and talents using a combination of both ipsative (ranking) and normative (rating) measures, and then links these to workplace culture. Drawing each of these components together for the first time to predict job performance is believed to be unique in the field. The findings can be used for selection, development and overall organisational culture purposes.
The Professional Styles questionnaire is based on 36 professional styles dimensions, which are grouped under 12 section headings for each of the four major clusters: Thought; Influence; Adaptability and Delivery.
The suite of tools has been developed and validated internationally with more than 50 countries, following a research project lasting in excess of two years - one of the largest and most robust in its sector.
The devil is in the detail
Gabrielle Parry, one of the team who developed the tool, has more than 18 years' experience in the field of psychometrics. She has worked, she says, with "questionnaires that focus on personality, and questionnaires that focus on motivation. Good questionnaires. But if you do them both, it will take 90 minutes. In 35 minutes - and this is where technology that didn't exist 20 years ago makes the whole process work - you can get the data and capture so much more, faster. Previously, you would have looked at both measures, but it would've taken a long time and it would've been clunky."
Conventionally, personality questionnaires usually use either a normative or ipsative approach: to combine the two within a test is thought to be unique. The normative approach compares an individual's results against the relevant norm groups: these are identified by calibrating the scores of other individuals who have previously completed the test, and are then used as a benchmark. In practice, this means that a respondent rates their agreement or disagreement with each individual item against a scale.
In contrast, an ipsative assessment is when there is a ranking against the person's own internal standard - in athletics, for example, an individual's "personal best" is an ipsative assessment. In practice, this means that a respondent is asked to choose between, or place in rank order, two or more items.
"This [the combination of normative and ipsative] gives us massive insight. We know that the best predictor of performance is the weighted combination score, but actually hidden in there are lots of things. Overall, an individual's score may be average or typical but underneath are buried clues and prompts about how these work in practice," says Parry.
This aspect allows much more insight into individuals' responses. The data can highlight differences or discrepancies in three ways.
- The stretch in the data, where for example the range of facet scores within any one dimension is greater than two stens (standardised scoring, from one to 10).
- Motive and Talent scores indicate the alignment between interest and skill in particular job functions. Where the motive score is higher than the talent score, it indicates a development area. Where talent is greater than motive, it can suggest an area where an individual has been given a role to perform because they are good at it, but there may be some issues with the underlying motivations.
- Where ipsative scores are higher than normative scores, the person may have an overly self-critical view of themselves, and that needs to be taken into account when analysing the scores. In a situation where normative scores are higher than ipsative, it might mean that the person has been easy on themselves, or that their approach changes when pressure mounts.
Culture
Including a culture aspect to the questionnaire adds a whole new dimension to the process. It is an innovation that the team of developers feels particularly proud about. In fact, it was this dimension that made Parry, a successful consultant with a competitor organisation for 18 years, leave her previous job and join Saville Consulting. She says this was the concept that really grabbed her and convinced her that this product would be different.
It gives organisations the opportunity to look at the fit between employee attributes and workplace culture. The Predicted Culture/Environment Fit aspect gives an indication of the aspects of the culture, job and environment that are likely to enhance or inhibit a person's success (see table 2).
It was developed because the team felt that the alignment of performance to culture was a real gap in the market.
This is said to be a very powerful tool, and has been developed through extensive research done in parallel with, or as a component of, the research on developing the Professional Styles job performance predictor aspect.
Parry says: "A lot of organisations say that they do many of the things that will be in preferred culture. We've trialled with organisations where we've taken actual culture data and preferred culture data and look at the gaps. That has been very telling." The culture questionnaires explore
- actual culture;
- preferred culture;
- corporate values; and
- satisfaction and commitment.
The key is the alignment between actual culture and preferred culture. If the organisation's preferred culture is for high competitiveness and innovation, but the workforce reflects an actual culture where these attributes are absent, then the gap will represent an insurmountable obstacle to business success.
Parry pinpoints the importance of the cultural aspect to job fit in terms of ensuring that organisations select the best fit for their organisation in the first place: "You can do highly professional, well-conducted recruitment but you can find that your preferred choice will be gone within six months. They base their choice on a small number of job competencies, focusing just on what the role requires with little thought about the culture that is in place. So the individual joins the organisation, and six months later they are out. And it is very rarely because they can't do the job. That swift level of departure is almost exclusively down to the level of fit."
To see how the report presents this information, have a look at the feedback given to this researcher on the completion of her own questionnaire as it appears in her feedback report (table 2).
The big five
The professional styles model is hierarchical and aligns to the "big five", believed to be at the heart of all personalities:
- adjustments/neuroticism - single-facet traits that determine behaviour, and include anxiety, hostility, depression, self-consciousness, impulsive behaviour and vulnerability;
- agreeableness - trust, frankness, altruism, compliance, modesty and compassion;
- conscientiousness - competence, order, sense of duty, striving for achievement, self-discipline and deliberation;
- extroversion/introversion - warmth, gregariousness, assertiveness, excitement-seeking, activity and positive emotions; and
- openness to experience - fantasy, aesthetics, feelings, actions, ideas and values.
Valid
Ensuring high validity and reliability was at the core of the design of the product and its subsequent trialling in 50 countries with more than 1,100 users.
Professor Saville explains: "There has been a huge amount of research done on actually predicting a person's job performance, including work with external raters. I haven't seen anything like the ability to predict job performance in this tool. But remember we are coming 25 years after OPQ was developed; technology has moved on and this rating/ranking method is unique."
Maintaining the validity of the product is an ongoing process. Once an individual has completed their questionnaire, they are encouraged to provide details of a line manager, supervisor or colleague who can also comment on their work and work styles. This contact is then sent a questionnaire, which takes around 10 minutes to complete and which looks for information on the performance of the initial respondent. The results are then fed back into the overall research.
Secrets and lies
One of the concerns voiced in relation to the use of psychometric testing, and even more so now that many are done in an unsupervised environment, relates to the honesty of the respondent.
Professor Saville himself recognises this. "It's very difficult to know what to do with the overall lie scales; there is nothing that is unfakeable, but you do start to become inconsistent at some stage," he says.
The unique rating/ranking aspect to the questionnaire is so dynamic that the user feels like the questionnaire is interacting with them. As a result, it has been shown to be much more difficult to fake. But besides this, there are a number of inherent measures that will indicate and highlight inconsistencies in responses. These cover:
- ratings acquiescence;
- consistency of rankings;
- motive-talent agreement; and
- normative-ipsative agreement.
Each of these response summaries can give an indication of issues surrounding the honesty of responses on the part of the respondent. Besides this, "if a person has vastly higher rating compared to the ranking then we can say confidently that this is an area that has been exaggerated. But if you warn people that there are these checks in place, most people are very honest and that has been shown in research," Professor Saville tells us.
But, as nothing is "unfakeable", then this is probably the greatest demonstration of why an organisation should not use psychometric tests in isolation. A well-structured interview, conducted by skilled interviewers, is the perfect complement to psychometric testing. Professor Saville suggests that what is required is "a good interviewer, who does more listening than talking, who asks open questions. Work-sample tests are also important: if people claim to be particularly numerate, then tell them they will be tested and ensure they are."
Putting it into practice
Each respondent who completes the Professional Styles questionnaire is assured feedback. This is another important aspect for Professor Saville, who feels strongly that treating all respondents with dignity and respect is not only important, but essential. He dismisses those who do not follow the same approach as "disgusting", as he feels that not only have individuals given up their time to complete the questionnaire, but each respondent has the capacity to become, if they are not already, a client of the recruiting organisation.
Feedback takes the form of a report, and the software automatically generates two: one for the respondent, and one for the test administrator or end-user. Results are shown in graphic form, alongside verbal descriptions which provide more detailed analysis. The feedback report for the individual will not include as much detail as the end-user report, but it remains true to the findings and results.
The end-user report covers a lot more ground - much of which we have discussed above. A copy of the questionnaire and two feedback reports will cost trained users £50 for each use.
The product is designed primarily for use on the internet, although a paper version is also available. Respondents are sent an email, providing them with a link to a secure website. A log-in is created using the individual's email address, and this becomes their password if the questionnaire is not completed all in one sitting.
To administer the tests, practitioners need to be qualified to British Psychological Society Level B assessor standard. Ideally, they will have been working with the OPQ/16PF or equivalent questionnaires and will just need to go through a two-day "conversion" training course run by Saville Consulting.
The last word
Professor Saville feels that the future for this product is limitless. He believes that there is still a need for psychometric testing, now more than ever. "Incompetence is a real cancer. Incompetent people recruit equally incompetent people. You must select people who are best qualified, with the best fit, irrespective of ethnic background, gender, orientation or religion, and I am passionate that this should be done well."
Saville Consulting's new product has just been launched. However, there are a variety of psychometric tools already in existence, and below, we detail how four organisations are using some of these to enhance their recruitment and selection strategies.
EMTEC provides training and support to the global automotive industry. Its main activities include:
- apprenticeship learning: providing a range of modern apprenticeship programmes that covers student recruitment, training, assessment and qualification; and
- technical services: offering adult technical training solutions, technical field support, and health and safety services.
Each year, around 17,000 young people apply for places on apprenticeship programmes delivered by EMTEC. The company had always been using ability tests at the initial stage of its assessment process, but when it came to updating its procedure it decided to work with SHL, a worldwide psychometric testing company.
Steve Fryer, EMTEC national recruitment manager, explains: "Our recruitment process needed a more professional and modern approach. We wanted our tests to reflect the company's image of excellence."
The organisation also wanted to be able to distinguish between verbal and numerical abilities to match candidates to apprenticeships. The tests provided by SHL allow the company to effectively measure and differentiate between these abilities and also add an assessment of the applicant's mechanical-reasoning abilities - necessary for the motoring industry.
All the tests are EMTEC-branded and, so far this year, the company has invited 10,000 applicants to take a test at one of its assessment centres and more than 5,000 have been assessed. Fryer says: "Every year, we find that a significant proportion of applicants drop out before attending assessment centres. This is actually useful, as it identifies those candidates who are truly motivated and committed." He also expects the approach that his company has adopted in selection to translate into a reduction of early leavers from apprenticeships due to a better fit.
Thresher Group, the chain of drinks retailers, has introduced a new online application and selection process to recruit branch managers. As a result of working with test agency PSL, Thresher has significantly changed its recruitment and selection process to attract more suitable candidates, who then go through a rigorous selection process that results in a better fit between job and personality for successful candidates.
The process now involves prospective branch managers being invited to apply online through a new careers website. They also take an online numerical reasoning test and a customised online personality questionnaire. Thresher Group has more than 15,000 employees, so wants to ensure it has the right people for the job. It sells 62 million bottles of wine a year and enough beer to fill 50 Olympic swimming pools!
Thresher is using a version of PSL's online numerical reasoning test, which tests a candidate's understanding of basic financial information, using such elements as a profit and loss account. Around 30% of applicants are sifted out of the process at this stage.
Those who are found to be suitable from this stage then go on to take a customised version of PSL's online personality questionnaire, which measures 25 factors such as extroversion, emotionality, openness to experience, energy and structure. This screens out approximately another 30% of applicants.
The remaining candidates then attend a face-to-face assessment with a district manager, which involves competency-based interviews and work-related exercises.
The company wanted to move away from using the telephone hotline process and an audition day, because it felt they were very labour-intensive and failed to produce candidates that were best suited to work as a branch manager for Threshers.
Kimberly-Clark (KC), owner of household brands Kleenex, Andrex and Huggies, is extending its use of psychometric assessment across Europe. Testing has been used by the company as an integral part of its recruitment strategy for more than 16 years, but it now wants to roll it out in its recruitment across Europe.
KC puts the cost of poor selection decisions in the region of £30,000 a go, but sees psychometrics as key to ensuring that the company remains competitive and profitable. "We simply have to get the best people to remain competitive and the use of psychometric assessment and ability tests is crucial in this aim," says Rick Woodward, learning and development director. "In particular, the OPQ enables us to learn things about candidates who we wouldn't be able to identify from a CV or in an interview situation."
Currently, the company uses SHL's online OPQ 32 test as part of its UK graduate recruitment programme and in executive recruitment.
All managers are fully trained in its use and implementation, which also includes an online numerical test, part of SHL's Ability Screening online.
Business-advisory firm Deloitte has substantially enhanced the speed and efficiency of its graduate recruitment programme by moving to an online verbal and numerical reasoning test, provided by PSL. The company recruits in the region of 1,200 graduates and undergraduates each year - from more than 10,000 applicants.
Previously, it was using a paper-based, 90-minute verbal and numerical reasoning test, which required eligible candidates to come into an office to sit the test while being supervised. However, this was labour-intensive and time-consuming. Graduates now apply online throughout the year and those who are successful in the initial sifting process are invited to complete the psychometric tests online.
Sally Whitman, senior manager in the Graduate Recruitment team at Deloitte, says: "Our research shows that the verbal and numerical reasoning tests provide evidence of ability and likely performance in the job and performance in professional chartered accountant qualifications. We're conducting more research in this area, but already we've seen a high correlation between a candidate's score in the test and their score in professional exams. Online testing has also saved us money because we're able to screen candidates prior to inviting them in for interview."
This article was written by Noelle Murphy, researcher/writer, IRS Employment Review, noelle.murphy@irseclipse.co.uk.
Table 1: Saville Consulting Wave Professional Styles
|
Features |
Benefits |
|
Advanced scoring technology |
Shorter assessment times, greater accuracy and finely tuned measurement |
|
Dimensions selected for the best workplace validity from research on extensive groups of working people |
Enhanced measurement of job performance enabling more precise and confident decision-making |
|
New dimensions maximising the prediction of work competencies |
Relevant to a wider range of roles and sectors |
|
Configurable, "Colossus" is a Saville Consulting bank of more than 5,000 items |
Tailored versions are available, developed around in-house competency frameworks and language |
|
Dual-response style, both ipsative and normative |
Ability to home in on specific scales where a candidate may have exaggerated or been overly self-critical |
|
Multilingual versions |
Global assessment in local language |
|
Predictions of cultural enhancers and inhibitors |
Cost-effective, well-matched selection with higher retention rates |
|
Exploration of the complex interaction between talents and motives |
Identifies unfulfilled motives and latent talents; this adds insight to feedback and development planning |
|
Dynamic question format |
Homes in on the fine distinctions and subtleties of each individual |
|
Measures 36 dimensions, with each dimension comprising three sub-aspects |
Comprehensive, offering the flexibility to work at summary level or to drill down and explore specific areas |
|
Separate versions for professionals and other staff |
Greater acceptability and relevance at senior levels |
|
One assessment model |
Offers the capacity to link people and their workplace culture, with the confidence of supporting validation |
|
Two online versions are available: invited access and supervised access; a hard-copy supervised version is also available |
Flexible and unsupervised administration for screening/development but with the backup of secure versions at the final selection stage |
|
No licence fees |
Pay as you go with lower annual costs |
Source: Saville Consulting, www.savilleconsulting.com.
Table 2: Predicted culture/environment fit
The following are aspects of the culture, job and environment that are likely to enhance or inhibit Noelle Murphy's success:
|
PERFORMANCE ENHANCERS |
|
|
+ |
Where heated debate is valued and people are encouraged to challenge ideas, argue and voice disagreements openly |
|
+ |
Where there is an emphasis on analysing and solving problems, and problem-solving ability is really valued |
|
+ |
Where creativity and innovation are encouraged and radical ideas and solutions welcomed |
|
+ |
Where there are numerous opportunities for learning and extending one's range of knowledge and skills |
|
+ |
Where there is an emphasis on comprehensively researching and recording the facts and communicating them clearly in writing |
|
+ |
Where the development of theoretical ideas and concepts is encouraged |
|
+ |
Where people are encouraged to resolve conflicts quickly and a value is placed on being able to handle angry and upset people well |
|
+ |
Where people listen to others and are sensitive to their differing needs and viewpoints |
|
+ |
Where there is generally little pressure |
|
PERFORMANCE INHIBITORS |
|
|
x |
Where dissent is frowned upon and people are discouraged from challenging ideas and voicing disagreements |
|
x |
Where there are few opportunities for analysing and solving problems |
|
x |
Where conventional attitudes prevail, traditional approaches are preferred and people are discouraged from generating new ideas |
|
x |
Where opportunities for acquiring new skills and extending one's range of knowledge are severely restricted |
|
x |
Where little value is attached to exploring all the facts and communicating them well in writing |
|
x |
Where there is little interest in the application of theoretical ideas and models, and people are given little time to explore different options and possibilities |
|
x |
Where little importance is attached to the ability to manage angry or upset people |
|
x |
Where little importance is attached to understanding people and the motives behind their behaviour |
Source: Saville Consulting, www.savilleconsulting.com.
|
According to Professor Peter Saville, carrying out some kind of testing to ensure a good "job fit" was first mentioned in the Bible, in the Old Testament. Gideon was trying to put together a decent group of soldiers but had too many to choose from. As his first test, he told the assembled group of soldiers of the realities of going to war, and that there was no shame in any soldiers going home to their families. While this reduced some of the head count, he was still left with too many. So he then brought them on a long and hard journey, which ended at a lake. He then watched how some of the remaining soldiers drank; those who stayed watchful and alert were invited to join his army. Attempts at measuring the differences between the psychological characteristics of individuals have been made from the time of Hippocrates in about 400 BC. He tried to define four basic temperament types, each of which could be accounted for by a predominant body fluid or humour: blood - sanguine (optimistic); black bile - melancholic (depressed); yellow bile - choleric (irritable); and, phlegm - phlegmatic (listless and sluggish). However, the methods employed, and results obtained, were far from scientific. The first attempt to scientifically measure the differences between individual mental abilities was made in the 19th century by Sir Francis Galton, who tried to map the human mind into different dimensions so as to demonstrate differences. He looked at how people differed in terms of their ability to discriminate between stimuli and, by collating the results, he devised a system that would allow an individual's abilities to be compared with those of others - a concept still prevalent today. Another figure who undertook considerable work in this area was Alfred Binet. The work of both Galton and Binet resulted in the view - still held today - that human ability is composed of a number of specific abilities. The basic tenet of testing is based on the principle of measuring human mental performance under different conditions and then making comparisons between people. Another pioneer of
psychometric testing was Walter Dill Scott, a psychology professor at
Northwestern University in Chicago. He rejected the conventional wisdom that
had prevailed since the industrial revolution that one worker was as good as
another. He contended that people had different, innate qualities. His first
psychometric test was used in 1915 to assess 15 engineering graduates at
Western Electric in Chicago for "creative potential" and is accepted as the
first use of psychometric testing in business. |
