by Brent W. Hubby, on Nov 10, 2022 1:07:09 PM
To provide value to a business, a behavioral assessment has to be accurate.
Accuracy comes down to two factors: validity and reliability. In other words, a good measure will measure what it is supposed to and do so consistently.
Validity and reliability aren’t just abstract concepts—statistical methods can assess them. But most personality tests and behavioral assessments don’t go to the trouble of doing it. That’s what sets PDP apart from other tests.
PDP’s ProScan Survey was developed in 1976 using a rigorous set of methods to ensure that it was reliable and valid. Since then, we’ve regularly verified that the tests continue to perform well and provide real value for our clients.
Here’s why you can trust PDP behavioral assessments to be true to their findings and make a real impact on your company.
Reliability of PDP Behavioral Assessments
In psychometric testing, reliability has a specific, technical meaning: it refers to how dependably or consistently an instrument measures a characteristic.
For example, imagine stepping on a scale, and it said you weighed some amount. Then you stepped on it again a minute later and said you weighed 25 lbs more than before. The scale is unreliable because it doesn’t provide consistent measurements. That’s a problem because it means that you can’t trust the results.
Behavioral assessments are the same—some are more reliable than others.
We typically measure reliability with two statistics: Chronbach's alpha (α) coefficients and test-retest reliability correlation coefficients.
Chronbach's Alpha
Chronbach’s alpha measures internal consistency—how closely related a set of items is in a group. So, for example, you would look at each of the test items that measure the trait "dominance" and look to see if they are closely related to all of the other items that measure dominance.
Chronbach’s alpha is calculated as a score between 0 and 1 where 0 indicates that items that measure a trait are not at all related to each other and 1 indicates that they are perfectly related to each other. Testing experts consider scores above 0.70 to have adequate reliability.
In its initial research, PDP Global’s ProScan Survey received an excellent reliability score of α = .94. High Chronbach’s alpha coefficients have also been found consistently in each research study since.
Test-Retest Reliability
Test-retest reliability is a measure of reliability that looks at how consistent the results of a test-taker are across time periods. It’s calculated by taking your result on a test and correlating it with your result on the same test a few months later.
Like Chronbach’s alpha, test-retest reliability is measured from 0 to 1, where 0 indicates poor test-retest reliability and 1 indicates that individuals achieve exactly the same results when they take the test again. Testing experts generally suggest that test-retest reliability coefficients above 0.70 are considered good.
In research, PDP Global’s ProScan Survey received test-retest reliability coefficient scores between 0.76 and 0.86.
Validity of PDP Behavioral Assessments
The other aspect of accuracy is validity—does the test measure what it says it measures?
There are several different kinds of validity and ways to measure it. One way to measure validity is by conducting tests like confirmatory factor analysis (CFA). CFAs look at the structure of the test and see if the data is consistent with the traits the test is supposed to measure. There’s a rigorous amount of research needed to attain a CFA, which is why it remains the gold standard of statistical evaluation even moreso than an exploratory factor analysis (EFA).
For the ProScan survey instrument, we needed to see if participants' responses were consistent with the four-factor (Dominance, Extroversion, Pace, Conformity) and one-factor (Logic) structure of the test. The most recent set of CFAs found that the current test structure has excellent fit with the data. We concluded multi-year studies in 2021 that resulted in these three CFAs. The CFAs were conducted in addition to the 11 EFAs we held in 2019 (seven of which included over 3,000 respondents in each study). Altogether, over 39,000 individuals participated in our studies for the 11 EFAs and 3 CFAs, which is a testament to our dedication to solid research.
Another important kind of validity is concurrent validity. Concurrent validity looks at the extent to which one test is related to other, similar tests. PDP Global has consistently demonstrated strong evidence of concurrent validity with correlation coefficients to similar tests ranging from 0.39 to 0.87.
Finally, it’s important to ensure that PDP Global’s tests work. Predictive validity confirms that the assessment can predict future behavior, in this case, future job success. Again, third-party research found that ProScan does successfully predict future job performance. In one study, behavior traits on the test were valid predictors of three workplace KPIs.
Nondiscriminatory Nature of PDP Behavioral Assessments
PDP Global has carefully crafted our assessments to ensure that they do not result in disparate impact.
Disparate impact occurs when members of one applicant group have a greater opportunity for employment because of the test results than another group. Research by the Center for Applied Psychology at the University of Colorado at Denver, a third-party group, conducted a thorough review of the PDP ProScan to assess for disparate impact by gender, race, and age.
The results were clear that disparate impact did not exist, so the assessment does not recommend people from some groups over people from others.
All of our instruments comply with the Equal Employment Opportunity Commission (EEOC) and follow the policies for employment tests and selection procedures. Some labor and employment lawyers applaud PDP for our diligence in designing a survey with legal integrity that is compliant with state and federal discrimination laws. You can rest assured that you comply with all relevant employment laws and policies when using our behavioral assessments.
Continual Testing of PDP Behavioral Assessments
Ensuring that a behavioral assessment is accurate isn’t a one-and-done activity. The assessment needs to be continually tested and improved.
We've engaged in validating our survey instruments since the first validation studies in the 1970s. During our 11 EFAs, we tested 50 trait adjectives in addition to the 30 words we already had at the time. We went through that process because our priority was determining the best set of 30 words with the greatest statistical power, especially since language evolves over time. Then, the CFAs proved that our optimized survey was an accurate model fit for practical application in the real world. Factor analysis from that research found that the Chronbach's alpha score was 0.84, which is higher than the 0.70 criteria that’s considered acceptable.
We're constantly reviewing our survey instruments to ensure that they continue to work well for our clients with rigorous testing procedures that go far beyond that of other companies.
PDP Behavioral Assessments Are Designed for Businesses
Outside of a workplace setting, someone can take simple personality tests for light-hearted amusement. But it's different for businesses. Companies need to make sure that their employee behavioral assessments work and that there will be a return on their investment.
PDP Global’s team has conducted unparalleled research that has made our suite of behavioral assessments the most accurate on the market. PDP Global followed the methods outlined by the American Psychological Association and the Society for Industrial and Organization Psychology for test construction. On the employment side, our instruments meet the requirements and standards created by the US Equal Employment Opportunity Commission and the Americans with Disabilities Act.
PDP Global’s suite of behavioral assessments can help you hire the right people, decrease turnover, increase job satisfaction, and more.
See how we can help you make better people decisions. Try a demo.
