Frequently Asked Questions

---

We get asked a lot of questions by candidates about the assessment process, why companies use it, and how it helps. To make it a little easier to understand, we've broken down the different aspects of the process, explained the how's and why's, and highlighted the research supporting it. 

Hiring managers are often tasked with many responsibilities, competing demands, influences, pressures to perform and meet company goals. These challenges can often cloud their judgment, making it harder for them to feel capable of making objective decisions. Managers that use the assessment process tend to have a realistic view of their ability to manage all these competing demands and are more aware of the potential risk of biases influencing their decisions. The assessment process allows managers to step back and see things more objectively. As third-party talent evaluators, we do not have to work under the same work-related challenges and have no vested interest in who is hired. Our focus is on our clients and ensuring they are armed with additional insights that help them make more fair and objective decisions. 

Ultimately, like with other business decisions, managers/companies use the assessment process to reduce the costs and risks associated with a 'bad hire.' A bad hire doesn't mean the person is bad.  Rather, it means there is a misalignment between the individual and the employer as it relates to the objectives of the role, skills and abilities, needs of the department/company, fit to the team/manager, and alignment with the strategy and company values/culture.  The costs to a company can be staggering and can range from 30% to 2.5 times or more of the position's annual salary (Navarra, 2022).  When there is a misalignment, it usually ends poorly for both parties. There are often missed goals and bottom line costs to the company, animosity, resentment, and declines in morale or even emotional harm experienced by the teams and employee(s). Companies that choose to use the assessment process want to minimize the risks and costs associated with this misalignment.

The assessment process aims to help an employer get an objective look at a candidate’s traits, characteristics, skills, and abilities related to a specific job.  It even gives hiring managers an idea of what it may be like to work with the candidate prior to bringing them onboard.  While it may feel like a one-way street, ultimately, it aims to ensure there is a strong fit between the employer and employee, that both parties are happy with their decisions, and that there is a long-term working relationship that is formed between the two. 

There are a lot of assessments because the objective is to examine a candidate more holistically vs. through a single lens, as you may get with the common interview.  Interestingly, each assessment has its strengths and weakness.  In fact, some assessments like the standard interview are thick with biases (Pronin, Olivola, & Kennedy, 2008; Pronin & Kugler, 2007; Hassin, Uleman, & Bargh, 2005; Roulin, Bangerter, & Levashina, 2015). Additionally, hiring managers frequently overestimate their ability to discern fact from fiction even though they feel like they are a “good judge of character .”  However, research has proven that an interviewer’s ability to pick up on lies and detect deception is shockingly poor - even when highly trained (Schneider, Powell, & Roulin, 2015; Reinhard, Scharmach, and Mu ̈ller, 2013; Erdogan and Liden, 2006; McFarland and Ryan, 2000; Barrick and Mount, 1996; Sackett and Wanek, 1996).  Interestingly, the typical interview is often perceived as fairer despite proof that it’s not (Steiner & Gilliland, 1996; Moscoso, 2000).    

Cognitive testing has also been used for many years and has historically been one of  the strongest predictors of job performance (Gottfredson, 2002; Ree & Carretta, 2002; Schmidt, 2002).  Unfortunately, it has also been found to create a disparaging impact (Roth, Bevier, Bobko, Switzer, & Tyler, 2001).  Personality measures and work simulations are used to offset this impact.  

Fortunately, research has shown over and over again that personality is also a strong predictor of job success (Barrick & Mount, 1991; Hogan & Holland, 2003; Judge & Illies, 2002; Judge, Bono, Illies, & Gerhardt, 2002; Barrick, Mount, & Judge, 2001; Barrick, Stewart, & Piotrowski, 2002) in a variety of job contexts. They have also been shown to be fairer and reduce the chances of adverse impact (Foldes, Duehr, & Ones, 2008; Ones & Anderson, 2002).    

Work simulations, like interviews, can be useful and allow employers to get an understanding of interpersonal and communication skills and how someone approaches their work.  Work simulations are fictitious scenarios designed to mimic the type of work someone may be asked to complete on a regular basis.  An individual’s performance in these scenarios is scored in a semi-structured manner, meaning there are scores assigned to each area; however, individual opinions and judgments are still involved in the scoring, making it somewhat subjective.  Thus, the overall reliability of work simulations has been somewhat mixed.  Some studies found reliability measures to be as low as .53 (which is very poor) to .86, which is considered good (see Cascio & Aguinis, 2018).    

Ultimately, research has helped us learn how limited each assessment was when used alone and how powerful they were when used together to predict job performance and success. 

Please know it’s fairly normal for people to feel some anxiety related to cognitive testing and have concerns about how it is being used and applied to the process.  Many candidates express a fear of being screened out of a job because they are not "smart"; however, we are not using them to determine your intelligence. The assessments we use are only looking at two specific areas of cognition and are not capable of determining anything as complex as someone’s overall IQ.  We are also not using them to screen people out of positions rather to gain more insights into how you think and process work-related information.  We use three different cognitive assessments, two timed reasoning assessments, and one untimed critical thinking assessment.   

The combination of these assessments helps employers understand and answer questions related to a number of different aspects of work-related behaviors, like: 

1) How quickly does the candidate pick up and learn new things? 

2) When training, what is the most effective learning approach to use with the candidate (visual, hands-on, auditory or written)?

3) How much information/data does the candidate gather before making a decision? 

4) How open is the candidate to considering different perspectives? 

5) How well does the candidate deal with ambiguity? 

6) How objective is the candidate when making decisions? 

7) Can the candidate thinking quickly on their feet? 

Ultimately, this information provides helpful insights into how to onboard individuals, understand learning styles, training needs, and even help determine workloads.  

We understand the concern because there has been a great deal of misuse of cognitive assessments in the past.  Research has even shown there is some clear disparaging impact when not applied appropriately.  Rest assured, we take a great deal of care when working with cognitive data to ensure they are applied correctly and leveraged fairly.  

Aligning schedules can take a couple of days to get set up.  However, once established, the testing and interview with someone from our team takes about 1.5 hours for each step (3 hours total).   Once the process is complete, we aim to get our final report to our clients within 48 hours (this may vary slightly during holidays).    

Once we hand over the information, we are removed from our client's hiring process.  If it's been more than three days since you've completed our process, you should contact them directly to inquire about the timing and next steps. 

The short answer is maybe.  There are two ways you can learn about the data you supplied during the assessment process:

1. If you are invited to join the company, you have the option of contacting us for feedback.  The feedback session takes about an hour, and during that time, we walk you through your data, review key strengths and developmental areas and explain how that data aligns with the company's competencies.  Afterward, we will send you a copy of your report.

2. If you are not invited to join or turn down an offer, you can contact us (at no charge to you) for feedback.  During the feedback session, we walk you through your data and provide insights into what was shared with the company.  Since the final report is owned by the client, we cannot provide you with a copy.  However, please note that we never share raw data with anyone - including our clients.  Therefore, the verbal information we provide during the feedback session reflects what was shared with them.  

Yes. All of the assessments used in our process have been heavily studied and subjected to peer review. Collectively, there are over 150 years of science supporting our assessments. We use the California Personality Inventory 260 (CPI-260), the OPQ32, Employment Aptitude Surveys, including verbal and numerical reasoning, and the Watson Glaser. 

The CPI and CPI 260®: The CPI was originally developed in the 1950’s to help identify and predict work-related behaviors of police officers.  Manuals document over 50 years of empirical research establishing validity with non-test correlates in a bibliography of nearly 2,000 titles (Groth-Marnat, 2003; Gough & Bradley, 2002; Gough & Cook, 1996). The CPI 260® scales are highly correlated (rs range from .81 to .97 with a median of .95).  The manual reports internal consistency (alpha) coefficients for the CPI 260® scales, ranging from .54 to .86 with a median of .75.  In terms of validity, the California Psychological Inventory scales used two basic criteria to evaluate validity. The first was that the scales must identify people who will behave in a specified way. Secondly, people with high scores must impress others as having the quality in question (Megargee, 1972). Additionally, Gough subjected the scales to a conceptual analysis to clarify what it is that each scale was assessing as well as exploring them for unexpected relationships and uses (Megargee, 1972). Correlations between individual CPI scales and relevant external criteria fall in the .2 to .5 range. 

OPQ32: The OPQ32 was originally developed in the 1980’s and an occupational model of personality, which describes 32 dimensions or scales of people’s preferred or typical style of behaving, thinking and feeling at work.  The OPQ32 was designed for international use and as part of its development all constructs and items were reviewed for application in different countries and cultures. It has been adapted into 30 languages and is particularly appropriate for use with professional and managerial groups, Reliability data was drawn from 30 country samples shows that IRT based reliabilities are high across different language versions and countries, ranging from 0.68 to 0.94 for individual scales (median = 0.86 across all 32 scales). Test-retest reliabilities were generally high (study 1: median=0.78, study 2: median = 0.85).  Using OPQ32r composite scores to predict specific leadership competencies, composite validities reach as high as 0.29 for manager ratings, 0.30 for colleagues and 0.27 for direct reports (SHL, 2013; Bartram, 2013).  

EAS Reasoning Measures: The EAS dates back to the 1940's when the developers were commissioned to construct a battery of tests to be administered to all applicants for all jobs in a medium-sized factory (PSI, 1994).  While the initial versions of the assessment were not overly impressive, 60 years later, much has improved with regards to their predictability of job performance.  Reliability data ranges from .81 for numerical reasoning to .82 for verbal reasoning (PSI, 1994).  Validity data for the EAS numerical and verbal measures, based on the most recent meta-analysis conducted were generally strong across occupational groupings, ranging from .30 or .40 thresholds which are considered indicators of substantial predictors of performance (PSI, n.d.; PSI, 1994). 

Watson Glaser: The Watson-Glaser Critical Thinking Appraisal as a distinguished history, dating back to its initial development in 1925 and was designed to measure critical thinking abilities.  Over the years it’s been consistently studied and revised and has been used in organizations and in academic settings for nearly 85 years (Watson & Glaser, 2010).  The test-retest intervals ranged from 4 to 26 days, with a mean interval of 11 days. The Watson-Glaser Short Form total score demonstrated acceptable test-retest reliability.  Individual reliability coefficients for each scale were found to be .80-.83 for recognizing assumptions, .55-.57 for evaluating arguments and .65-.70 for drawing conclusions.  The various validity measures (content, convergent, criterion) range from .33-.79 (Watson & Glaser, 2010). 

When examining the viability of an assessment, the reliability and validation studies are crucial.  Assessments used for selection or hiring should be specifically validated for that purpose.  Any assessment that has not gone through validation studies and peer review should be avoided.  The publishers should provide these studies upon request for you to review and examine.  It’s also worth visiting a university library where you can search through peer-reviewed journals to verify the publisher’s data.  Reviewing and understanding the reliability and validity studies is important, particularly if you are going to have an assessment administered and interpreted by a non-psychometrically trained individual inside your organization.  Using assessments that are not validated can get you into legal trouble, particularly if you are sued for discriminatory hiring practices.   

Noted below are the reliability and validity measures and what they mean. 

Reliability coefficient values and their interpretative power are: 

.90 and up are considered excellent 

.80-.89 are considered good 

.70-.79 are considered adequate 

.69 and below are considered poor  

Validity coefficients and their interpretative power are: 

.35 and above are considered very good 

.21-.34 are considered reasonable 

.11-.20 are considered inconsistent 

.10 and below are considered poor 

If your question isn't answered here, please email us at info@sparknu.com

Barrick, M. R., & Mount, M. K. (1991). The Big Five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44, 1–26.  

Barrick, M. R., Mount, M. K., & Judge, T. A. (2001).  Personality and performance at the beginning of the new millennium: What do we know and where do we go next?  International Journal of Selection and Assessment, 9, 9-30.  

Barrick, M. R., Stewart, G. L., & Piotrowski, M. (2002).  Personality and job performance: Test of the mediating effects of motivation among sales representatives. Journal of Applied Psychology, 87, 43-51.  

Cascio, W. F., & Aguinis, H. (2018).  Applied psychology in human resources management (8th ed). Person.  

Erdogan, B., and Liden, R. C. (2006). Collectivism as a moderator of responses to organizational justice: implications for leader-member exchange and ingratiation.  Journal of Organizational Behavior, 27(1), 1-17.  

Foldes, H. J., Duehr, E. E., & Ones, D. S. (2008). Group differences in personality: Meta-analyses comparing five racial groups. Personnel Psychology, 61, 579–616.  

Gottfredson, L. S. (2002).  Where and why g matters: Not a mystery.  Human Performance, 15, 25-46.  

Gough, H. G., & Bradley, P. (1996/2002). CPI Manual (3rd ed.). Mountain View, CA: CPP. Inc.  

Gough, H. G., & Bradley, P. (2005). CPI 260 Manual. Mountain View, CA: CPP, Inc.  

Gough, H. G., & Cook, M. (1996). CPI-434 manual. Oxford: Oxford Psychologists Press.  

Groth-Marnat, G. (2003).  Handbook of psychological assessment (4th ed).  Wiley.  

Hassin, R., Uleman, J., & Bargh, J. (2005). The new unconscious. University Press.  

Hogan, J., & Holland, B. (2003). Using theory to evaluate personality and job performance relations: A socioanalytic perspective. Journal of Applied Psychology, 88(1), 100-112.  

Judge, T. A., Bono, J. E., Illes, R., & Gerhardt, M. W. (2002).  Personality and leadership: A qualitative and quantitative review.  Journal of Applied Psychology, 87, 765-780.  

Judge, T. A., & Illes, R. (2002).  Relationship of personality to performance motivation: A meta-analytic review.  Journal of Applied Psychology, 87, 797-807.  

McFarland, L. A., & Ryan, A. M. (2000). Variance in faking across non-cognitive measures. Journal of Applied Psychology, 85, 812–821. 

Megargee, E. I. (1972). The California psychological inventory handbook. Jossey-Bass Publishers.  

Moscoso, C. (2000).  A review of validity evidence, adverse impact and applicant reactions.  International Journal of Selection and Assessment, 8, 237-247.  

Navarra, K. (2022, April, 22). The real costs of recruitment. SHRM.  Retrieved on April 24, 2022 from, https://www.shrm.org/resourcesandtools/hr-topics/talent-acquisition/pages/the-real-costs-of-recruitment.aspx?_ga=2.260116749.1675676430.1650831388-1590127820.1650831388  

Ones, D. S., & Anderson, N. (2002). Gender and ethnic differences on personality scales in selection: Some British data. Journal of Occupational and Organizational Psychology, 75, 255–276.  

Pronin, E., & Kugler, M. B. (2007). Valuing thoughts, ignoring behavior: The introspection illusion as a source of the bias blind spot. Journal of Experimental Social Psychology, 43, 565–578.  

Pronin, E., Olivola, C. Y., & Kennedy, K. (2008). Doing unto future selves as you would do unto others: Psychological distance and decision making. Personality and Social Psychology Bulletin, 34, 224–236. 

PSI (n.d.). Technical report addendum: A meta-analysis of the validity of the EAS test series.  PSI Services. 

Ree, M. J., & Carretta, T. R. (2002).  g 2 k. Human Performance, 15, 3-23.  

Reinhard, M.-A., Scharmach, M. and Mu"ller, P. (2013). It’s not what you are, it’s what you know: experience, beliefs, and the detection of deception in employment interviews. Journal of Applied Social Psychology, 43(3), 467-479.  

Roth, P. L., Bevier, C. A., Bobko, P., Switzer, F. S., & Tyler, P. (2001).  Ethnic group differences in cognitive ability in employment and educational settings: A meta-analysis.  Personnel Psychology, 54, 297-330.  

Roulin, N., Bangerter, A., & Levashina, J. (2015). Honest and deceptive impression management in the employment interview: Can it be detected and how does it impact evaluations? Personnel Psychology, 68, 395–444. https://doi.org/10.1111/peps.12079    

Sackett, P.R. and Wanek, J.E. (1996). New developments in the use of measures of honesty, integrity, conscientiousness, dependability, trustworthiness, and reliability for personnel selection. Personnel Psychology, 49(4),787-829.  

Schmidt, F. L. (2002).  The role of general cognitive ability and job performance: Why there cannot be a debate.  Human Performance, 15, 187-210.  

Schneider, L., Powell, D., & Roulin, N. (2015).  Cues to deception in the employment interview. International Journal of Selection and Assessment, 23(2), 182-190.  

SHL (2013). OPQ32r technical manual. SLH Steiner, D. S., & Gilliland, S. W. (1996).  Fairness reactions to personnel selection techniques in France and the United States.  Journal of Applied Psychology, 81, 134-141.  

Watson Glaser (2010).  Technical manual and user guide: Watson-Glaser II critical thinking appraisal. Talent Lens.