Iain Coyne: Games-based assessment

In a previous blog (February 17^th), I detailed the dark side of digital technology within working contexts. Here, I wish to redress the balance by outlining the positive use of technology in employment selection and assessment. Specifically, the developing, novel and highly interactive media of games-based assessment.

Technology in selection

Think back to your first-ever experience of being selected for a job. Did it involve one or more interviews? Did it involve psychometric testing? Was it an assessment centre? Most of us will have experienced some form of assessment when applying for a job, although the type of methods used and the nature of the selection will vary. Surveys show consistently interviews are ubiquitous, with the use of psychometric tests of personality and/or cognitive ability increasing over time.

The traditional perception of psychometric selection testing is one of applicants being invited to attend a group assessment session at a controlled testing centre, monitored by an assessor. Applicants would complete a paper and pencil test typically involving selecting an answer from a number of choices (through circling the correct response or shading in the chosen letter) or writing the response to a question/statement. This test would then be hand-scored by the assessor, who would compose a report for the organisation and ideally provide feedback to the test-taker. Apart from the candidate feedback part, this was my first experience of selection testing – and yes computers had been invented at this time!

Advances in technology and measurement theory means the above scenario is more likely to be the exception than the norm. Current job applicants for many roles are now likely to experience some form of technology-based assessment including telephone/video interviews, computer-based simulations, video-based scenario assessments, internet-delivered testing, virtual assessment centres and avatar-based virtual reality. Computerised testing is not necessarily new; rather the expansion of the use of technology across different selection methods and particularly Internet-delivery of psychometric tests is.

Advantages and concerns of Internet-based testing

Scholars have pointed to the advantages of Internet-delivered tests for test publishers, test users and test takers. These include:

• Quick and low cost collection of test data;
• Increased access to large numbers of test-takers;
• Ability for test-takers to complete the assessment in their own home;
• Ability to test skills and performance in addition to knowledge using novel item types;
• Development of tests which adapt to the performance level of the candidate;
• Enhancement of test-taker engagement through the use of multimedia;
• More reliable scoring in comparison to paper and pencil testing;
• Virtually instantaneous provision of reporting and feedback on performance;
• Availability of practice material for test-takers;
• Enhanced security of test materials and test-taker data;

Not surprisingly, given these advantages the market has increased for technology-based testing, with test developers and employers eager to adopt new, quick, exciting and visually appealing assessment tests. The majority of job applicants are now likely to experience an online psychometric test, often early on, in the selection process. However, just because a test has enhanced graphics, involves avatar-based simulation or includes novel types of items does not mean it is effective. In a selection context, advanced technology does not always equate to test validity or reliability. Poor tests are poor tests regardless of the media used to administer and score them. It is therefore imperative those creating, distributing and using technology-based testing for selection ensure the tool conforms to good practice.

Obviously, this leads to the question of what is good practice? To address this question, my previous research under the aegis of the International Test Commission (ITC) created a set of guidelines on Internet-Delivered and Computer-based testing (https://www.intestcom.org/page/18). This document provides guidance for test publishers, test developers and test users on good practice issues around technology (e.g. ensuring the test can be used correctly across a number of platforms), quality (e.g. checking the test has evidence of validity, reliability etc), control (detailing the level of supervision required) and security (e.g. of test materials and candidate data transferred over the Internet).

More recently, I’ve composed a book chapter (Coyne & Bartram, in review) which posits that those involved in the testing process need to ask three questions before using technology-delivered testing: Is the test fair? Is it secure? Is it psychometrically sound? Hopefully, such guidance will lead to the situation where organisations use fair, secure and sound Internet-based tests and not just visually attractive ones. Online there are instruments which purport to measure psychological constructs in a dynamic, visually attractive and engaging way. However, many do not have the level of evidence behind them needed to make effective and fair judgements in high-stakes selection contexts. My advice here is don’t be swayed by the aesthetics; always check the evidence.

Games-based assessments

Games-based assessments (GBAs) are Internet-delivered (via an app) tests which either simulate job functions and therefore provide a realistic job preview (a form of serious game) or present computer games designed to assess criteria linked to job-related criteria (e.g., personality/ability). They are different to conventional online tests as they immerse the test-taker in a series of game-like activities, collating a large amount of data throughout the testing experience which is then coded to create scores on psychological constructs. As a result, rather than just ticking a box which is then scored, GBAs collect data on reaction time, number of choices, navigation, errors etc.

I am currently engaged closely with one of the leading games-based assessment developers, Arctic Shores (https://www.youtube.com/watch?v=a-60jRMTXHs). This relationship encompasses research activities, consultancy and facilitation of MSc student projects. Indeed five of our MSc Psychology cohort are gaining first-hand experience of Arctic Shores’ approach to games-based assessment.

There is no doubt that GBAs will appeal to the digital native generation of job applicants as well as to organisations who wish to portray a dynamic, future-oriented brand image. As a result, their use in job selection contexts is likely to increase. Currently, the academic research on GBAs is limited and therefore many questions on the utility of this approach are yet to be answered.  Reverting back to the three questions posed earlier; one way to establish utility is to consider the extent GBAs are fair, secure and psychometrically sound.

Fairness of GBAs

Applicant reactions to GBAs may well be positive given the issues outlined above. In particular, we would expect those individuals who regularly play computer games to enjoy the notion of being assessed via a gaming platform. The ‘face validity’ of GBAs (i.e. it looks like it measures what it is meant to measure) is expected to be high and hence may promote perceptions of fairness by job applicants.

On the other hand, research will need to establish perceptions of fairness of judging people via a game and whether some test-takers are at a disadvantage when being assessed by GBAs. Potential impact of demographic features such as gender, age and ethnicity will need to be examined alongside other aspects such as experience with gaming and computer anxiety. GBA developers will need to establish that differences in performance between candidates is due to real differences in the constructs being measured and not other construct irrelevant features.

Psychometric strength of GBAs

As many GBAs are quite new and use proprietary platforms, information about their validity and reliability is difficult to obtain. However, especially given the paradata collated within a GBA test developers need to provide evidence that the GBA assesses the construct it is purported to assess (e.g. how can we be sure certain data in a game measure resilience)? Important for selection contexts, it is essential to establish the extent that GBA scores predict job performance. After all, the whole point of a GBA is to select in and out those predicted to perform well or poorly on the job.

Examination of consistency of scores across different platforms is also important, as candidate scores across different platforms (e.g. lap-top, tablet and mobile phone) should be similar. If not, it raises the question of whether the platform impacts on performance. If comparing candidates playing the game across different platforms then the potential for inaccurate selection decisions if heightened.

Security of GBAs

GBAs do not throw-up any different security issues than would need to be considered for any Internet-based assessment tool. Security protocols would need to ensure protection of the intellectual property of the game, test taker data being transferred over the Internet the storage of test taker results. This involves a combination of explicit policies on how data is collected, stored and who has access to it; with technical solutions such as data encryption, firewalls and secure back-up of data.

Linked to security is candidate authenticity. If the test is taken remotely, then how do know the person taking the test is actually the person applying for the job? The game could be played by someone else or the candidate could receive advice/guidance from another person while playing the game. To some extent this is mitigated by the nature of the assessment as there are not obvious right answers and responses are difficult to fake. This concern is not exclusive to GBAs, with remote forms of assessment liable to questions of authenticity. In fact, authenticity is not solely a problem with digital forms of assessment. Student exams, face-to-face interviews and even driving test examinations have all been subject to cheating and faking.

To counter authenticity a number of approaches can be adopted including developing honesty contracts and re-testing an individual later in the selection process in supervised conditions. Technologically, we can examine response patterns and response latencies to alert to possible cheating. Further, the relatively recent development of remote proctoring, where an individual taking an online test is monitored remotely via a web cam (which could be from a different part of the world), has reduced concerns about authenticity.

The chances of you as the job candidate being asked to ‘put down your pencil’ at the end of a group testing session in exam-like conditions are becoming more and more remote. More likely you’ll be sat in your own home, playing an app-based game on your tablet and when complete, receiving an emailed report of your scores on the constructs being measured. Indeed it is not inconceivable to vision a scenario of the GBA being followed by a remote interview and virtual assessment centre which are being used to select for a remote-working position in an organisation.

This Blog post was written by Dr Iain Coyne, Senior Lecturer in Organisational Psychology and Director of the Work Psychology and Business Psychology MSc programmes. Iain can be contacted on i.j.coyne@lboro.ac.uk