All published articles of this journal are available on ScienceDirect.
The Motivation Competencies That Count Most: An Online International Study
Abstract
Background:
With an online sample of 8,349 people from 123 countries (74.9% from the U.S., Canada, and India), a new test was used to rank eight motivation-related competencies according to how well they predicted desirable, self-reported outcomes. Each of the competencies was derived from empirical studies showing that such competencies were associated with higher levels of motivation. The competencies were: Maintains Healthy Lifestyle, Makes Commitments, Manages Environment, Manages Rewards, Manages Stress, Manages Thoughts, Monitors Behavior, and Sets Goals.
Objective:
The study was conducted to identify and prioritize competencies that are associated with higher levels of motivation.
Methods:
A “concurrent study design” was used to assess predictive validity, which was suggested by a strong association between test scores and self-reported answers to criterion questions about levels of motivation, life satisfaction, and professional success. Regression analyses were conducted to prioritize the competencies. Demographic analyses were also conducted.
Results:
The findings support the value of motivation training; test scores were higher for people who had received such training and were positively correlated with the number of training hours accrued. Effects were found for education, race and age, but no male/female difference was found. Regression analyses pointed to the importance of two of the eight competencies in particular: Sets Goals and Manages Thoughts.
Conclusion:
The study supports the view that motivation competencies can be measured and trained and that they are predictive of desirable motivational outcomes.
1. INTRODUCTION
Motivation has been discussed from many different perspectives since it was first introduced as a useful research construct in the early 1900s [1-4]. Generally speaking, the term is said to refer to hypothetical internal states of an organism that are believed to somehow energize the organism to behave in certain ways [5-7]. The Oxford English Dictionary includes modern-sounding uses of the language of motivation as early as the late-1300s, by which time “motive” was already referred to as “an inward prompting or impulse” [8].
In early formulations of the term in psychology, inner motivational states were often described as aversive – that is, as unpleasant states that an organism sought to reduce or eliminate [1, 9]. The internal states are normally inferred from behavior, but behavior alone is not always a good indicator; one can eat, for example, without being hungry. Context is important in making inferences about motivation [10, 11], and so are aspects of an organism’s history that produce a motivational state – what behavioral psychologists call “establishing operations” [12]. Contemporary discussions about motivation focus on a number of different issues, such as the relative contributions of intrinsic versus extrinsic motivation [13-19], theories of motivation [20-22], environmental and personality factors affecting motivation [23-27], and how people become motivated in particular settings or situations, such as in the work environment [28-31], in sports and exercise activities [32-37] and educational settings [38-41].
A comprehensive review of different approaches to the study of motivation is beyond the scope of this paper. Instead, we will focus on just one approach to motivation that has received relatively little attention, and that is the competencies approach. What types of knowledge and skills – preferably that can be both measured and trained – help to boost motivation? David McClelland’s work is relevant here, given that his early research and theoretical work focused on motivation [42, 43] and given that an article he published in the American Psychologist in 1973 helped propel a competencies movement that continues to impact business, education, government, and other domains in which human performance is important [44]. Much of the research McClelland and his colleagues conducted on competence was performed through his company, McBer, in the 1970s and 80s [45] and was eventually summarized in books published in 1982 and 1993 [46, 47]. The McBer organization focused on measuring and training competencies to improve leadership performance, often by using structured interviews (“behavioral event interviews”) to determine what competencies were common among top performing leaders [47]. McClelland and his associates did not, however, look generally at which measurable and trainable competencies underlie motivation per se. Nor did they develop written tests to assess such competencies [48, 49].
One advantage of a competencies approach is that it avoids labeling. Because competencies can be improved, even low scores on a competencies test can give people hope. A competencies approach breaks complex and sometimes mysterious looking abilities like “leadership” into a set of measurable, trainable skills [50]. Generally, when people know it is possible for them to improve, they feel and perform better [40, 51].
A number of tests have been developed over the years to measure how motivated people are, such as the Motivation Analysis Test [52, 53] and the Managing by Motivation Questionnaire [54], but no tests that we are aware of look at the broad range of competencies that might energize people to behave. Andersen [49] described a new test that specifically measures three important types of motivation – power, achievement, and affiliation – that McClelland himself showed were important in leadership and management.
Other tests that measure level of motivation include the Achievement Motivation Inventory [55], the Athletic Motivation Inventory [56], the Miller Motivation Scale [57], the Sales Motivation Inventory [58], the Work Motivation Inventory [59], and the Volunteer Motivation Inventory [60]. At first glance, the Motivational Sources Inventory [61] would appear to look at competencies of some sort, but in fact, it purports to measure only internal mental components of motivation. The 20th edition of the Mental Measurements Yearbook (MMY) [62] lists 26 validated tests that measure levels of or various aspects of motivation, but none are listed that measure competencies that might boost motivation.
The internet has complicated matters, allowing any individual or group to post any sort of test, with users generally unaware that no attempt has been made to evaluate the test. Non-validated tests of motivation appear on websites such as https://www.psychologytoday.com, https://www.queendom. com/, https://www.seemypersonality.com. Sometimes these tests include disclaimers such as, “this report is intended for personal growth purposes only,” and sometimes they offer minimal results and request a fee to see more detailed results. No regulations exist to limit the posting of such tests, so they will inevitably grow in number over time. With more and more people going first to the internet to get information about physical and mental health [63], the spread of non-validated tests is problematic, and there is no easy solution to this problem.
Several tests (none of which is listed in the MMY) measure competencies that managers or teachers need to motivate other people. Ololube [64] described such a test designed for use with teachers in Nigeria, for example, and in 2008 the first author of the present paper posted a managerial version of the test described herein (the EMCI-m, accessible at https://mymotivationskills.com/managers/). In addition, business training programs have relied on tests that measure a range of managerial competencies, including competencies that increase employee motivation [65]1. Breaking down human performance into measurable, trainable competencies has proved to be valuable in areas such as education [66-68], business [69-71], the military [72], sports [73], and even, more recently, in areas such as creativity [74, 75], stress management [76, 77], and relationships [78-80].
The present paper investigates a competencies approach to the study of motivation, posing the question: Could acquiring certain types of knowledge and skills increase levels of motivation and perhaps even improve life outcomes? We use a “concurrent study design” to evaluate some aspects of the predictive validity of a new test that measures such competencies, and we rank order those competencies according to how well they predict self-reported levels of motivation, professional success, and life satisfaction – desirable outcomes that might conceivably be more likely if people have high levels of motivation competencies. We also look at demographic differences in test scores, specifically by age, gender, race, education level, and location.
2. METHODS
2.1. Participants
The present study analyzed data obtained from a convenience sample of 8,349 individuals in 123 countries who took the EMCI-i at the website https://MyMoti vationSkills.com/ between May 20, 2007 and June 25, 2020. Participants were not actively recruited. They presumably found the test through search engines or through links to the test posted at various times on a variety of different websites, among them: https://motivationshow.com, https://itsallabout focus.com, and https://motivation-goals.com.
Our original sample contained 9,123 cases before cleaning, 8,349 cases after cleaning. The 8,349 cases make up our final sample and are the subject of our statistical analyses. Cases were removed when participants answered fewer than half the questions, reported English fluency levels under 6 (on a scale from 1 to 10, where 10 was the highest possible fluency), or appeared to be under age 13 (see below). When people took the test more than once, we included only data obtained on the first trial in which they answered at least half the questions. Regarding location, 5,084 (63.9% of the number of cases for which location was known) of our participants were from the U.S. or Canada, 874 (11.0%) were from India, 2,000 (25.1%) were from 120 other countries, and location was unknown for 391 (4.7%) people. In total, therefore, the location was known for 7,958 (95.3%) of our participants, and 74.9% of those individuals were from the U.S., Canada, or India.
The mean age of the participants was 29.3 (SD = 12.8, mode = 18, range 13 to 90). A minimum-age cutoff of 13, which was approved by our Institutional Review Board (IRB), was used given that the Flesch-Kincaid reading level of our test questions was 7.1 and given that 13 is the typical age at which students in the U.S. have completed the 7th grade. (We acknowledge that many people at this age read below their grade level.) In all, 658 (7.9%) of our participants (after data cleaning) were under age 18.
3,448 (41.3%) of our participants identified themselves as male, 4,846 (58.0%) as female, and 55 (0.7%) as other. 407 (4.9%) said they lacked a high school degree, 3,042 (36.4%) said they had received a high school degree, 518 (6.2%) said they had received a 2-year associates degree, 2,790 (33.4%) said they had received a college degree, 1,324 (15.9%) said they had received a master’s degree, and 212 (2.5%) said they had received a doctoral degree; educational level was unknown for 56 (0.7%) participants.
4,104 (49.2%) identified themselves as white, 1,481 (17.7%) as Asian, 1,749 (20.9%) as Hispanic, 497 (6.0%) as black, 37 (0.4%) as Native American, and 358 (4.3%) as other; race/ethnicity was unknown for 121 (1.4%) participants. Finally, after cleaning, the average English fluency reported by participants who had answered our fluency question was 9.1 (SD = 1.2).
1,324 (15.9%) of our participants reported that they had training in motivation, and 6,959 (83.4%) said they had not; training history was unknown for 66 (0.8%) participants. When asked how motivated they were (on a scale from 1 to 10 where 10 was the highest possible answer), participants reported an average score of 6.8 (SD = 2.1). When asked how satisfied they were with their life, participants reported an average score of 6.5 (2.3). When asked how much success they had had in their professional life, participants reported an average score of 6.3 (2.3).
2.2. Materials and Design
Our investigation employed a “concurrent study design” that provided convergent validity evidence with related measures, following the most recent guidelines of Standards for Educational and Psychological Testing [79, cf. 80], co-published by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. Specifically, we sought to measure the strength of the relationships between our test scores and the scores on our self-reported criterion questions. This design is called “concurrent” because we obtained test scores and criterion measures at the same time, a strategy that avoids possible temporal confounds. Results from studies employing this design are considered especially robust when the pattern of relationships between test scores and criterion measures proves to be consistent across different demographic groups.
The test instrument employed in this study is called the Epstein Motivation Competencies Inventory for Individuals (EMCI-i) and was developed in 2007 and posted online that year. The EMCI-i has 40-items, each of which requires an answer on a 5-point Likert scale, with points labeled Agree and Disagree at the extremes. It assesses eight competencies that both research and theory suggest are important for motivation. Definitions and relevant references, which have been updated since the test was first posted, are given in Table 1.
Table 1.
|
Maintains a healthy lifestyle: You maintain a healthy lifestyle and good habits. You take good care of your body and brain. Sample item: “I avoid substances that might damage my body or brain.” References: [81–86] |
|
Makes commitments: You make commitments to others in a way that boosts your performance. Sample item: “I always keep my promises.” References: [87 – 94] |
|
Manages environment: You alter and maintain both your social and physical environments to keep yourself motivated. Sample item: “I use ergonomic devices to help me perform at my peak.” References: [95 – 100] |
|
Manages rewards: You manage rewards in a way that boosts your motivation. Sample item: “I often ask others for encouragement and support.” References: [85, 101–106] |
|
Manages stress: You take steps to reduce stress for yourself, practice relaxation techniques, and also take a proactive approach to stress management. You practice techniques that help you avoid or minimize stress. Sample item:“I often try to identify and remove sources of stress in my life.” References: [107–110] |
|
Manages thoughts: You manage your thinking to keep yourself motivated. Sample item: “I frequently try to picture a bright future for myself.” References: [107, 110-119] |
|
Monitors behavior: You monitor your behavior and progress. In almost all situations, keeping records of performance boosts performance. Sample item: “I use many different methods to monitor my behavior.” References: [109, 120 – 127] |
|
Sets goals: You set short and long term goals for yourself. Sample item: “I typically spend a few minutes each day to plan the day.” References: [91, 101, 109, 128-149] |
2.3. Procedure
Participants were first given basic information about motivation and then informed that they would be taking “an inventory of eight types of skills and knowledge that help people boost their motivation or keep themselves motivated.” They were also told: “Most people are able to complete the test in less than 10 minutes, and there are no right or wrong answers. Just select the response that seems best. After you answer the questions, you will be given a detailed report that will give you your scores and explain what they mean.”
Participants were then asked basic demographic questions, along with three criterion questions that they answered on a 10-point Likert scale, as follows: (1) “Generally speaking, how motivated are you?” (scale from Not at all to Highly motivated). (2) “How satisfied are you with your life?” (scale from Not at all to Extremely satisfied). (3) “How much success have you had in your professional life?” (scale from Low to High). Participants were also asked whether they had ever received any training intended to boost their motivation and how many hours of training they had received. The test itself followed.
After completing the test, the test taker clicked on a “Submit” button to generate a detailed report defining the eight competencies and showing his or her total score and the score in each competency area. It also gave people an email address where they could get additional information about the ongoing study or, consistent with standards of the American Psychological Association (APA) and the United States Department of Health and Human Services (HHS), could request that their data be deleted.
Each competency is measured by five items, one of which is a dummy item – a slightly reworded version of another item in the same competency area – that can be used to assess the consistency of responding as soon as an individual completes the test [75]. None of the eight dummy items was scored or included in any of the statistical analyses in the present study.
3. RESULTS
3.1. Reliability and Validity Evidence
Internal-consistency reliability as measured by traditional reliability coefficients was relatively high for total scores: Cronbach’s alpha was 0.88, and the Guttman split-half measure was 0.84. For the eight individual competencies, Cronbach’s alpha ranged from 0.32 (for Maintains Healthy Lifestyle) to 0.72 (for Sets Goals), and the Guttman split-half measure ranged from 0.23 (for Maintains Healthy Lifestyle) to 0.74 (for Manages Thoughts). Test-retest reliability was not measured (see Discussion). We also did not develop an alternate form of the test, so alternate-form reliability could not be estimated.
Regarding evidence of predictive validity, test scores were positively correlated with participants’ answers to all three of our criterion questions: their estimates of their own level of motivation (Spearman’s ρ = 0.55, P < 0.001)2; their self-reported life satisfaction (ρ = 0.46, P < 0.001); and their self-reported professional success (ρ = 0.37, P < 0.001). In addition, test scores were substantially higher among the 15.9% of our participants who reported having had motivation training (Mann-Whitney U = 3,581,047.5, P < 0.001, Myes = 66.9 [SD = 12.7], Mno = 61.6 [13.7]) and were positively correlated with the number of training hours (Spearman’s ρ = 0.10, P < 0.001).
3.2. Demographic Differences
Demographic differences are summarized in Table 2. Small but significant effects were found for the level of education completed with total scores generally increasing with educational level. Test scores were correlated with age, but the correlation was small and probably of little practical significance (Spearman’s ρ = 0.05, P < 0.001). An effect was also found for country (H = 161.2, P < 0.001, MUnitedStates/Canada = 62.0 [13.5], MIndia = 68.1 [11.9], MOther = 61.5 [14.2]), with participants from India substantially outscoring people from all other countries combined (U = 2,415,341.5, P < 0.001, MIndia = 68.1 [11.9], MAllOtherCountries = 61.8 [13.7]) – perhaps indicating that our sample was drawn mainly from urban areas in India, where people have tended since the late 1990s to adhere to a “work hard and get rich” ethos [150, 151].
An effect was found for gender, but the effect disappeared when only males and females were compared (U = 8,198,300.0, P = 0.15, MMale = 62.2 [14.3], MFemale = 62.8 [13.3]). A relatively large difference was also found between males and females combined and those who chose “other” as a designation for their gender (U = 148,329.0, P < 0.001, MMale/Female = 62.5 [13.9], MOther = 52.8 [16.0]).
| Education Level Completed | ||||
|---|---|---|---|---|
| Demographic | Mean Score | SD | H | P |
| None | 54.1 | 15.8 | 250.8 | < 0.001 |
| High School | 60.9 | 13.7 | ||
| Associates | 62.4 | 13.6 | ||
| College | 63.8 | 13.3 | ||
| Masters | 65.7 | 12.5 | ||
| Doctorate | 64.0 | 12.6 | ||
| Race | ||||
| Demographic | Mean Score | SD | H | P |
| American Indian | 65.1 | 18.5 | 161.6 | < 0.001 |
| Asian | 65.9 | 13.4 | ||
| Black | 63.7 | 14.7 | ||
| Hispanic | 63.1 | 12.3 | ||
| White | 61.0 | 13.9 | ||
| Other | 59.2 | 14.4 | ||
| Location | ||||
| Demographic | Mean Score | SD | H | P |
| India | 68.1 | 11.9 | 161.2 | < 0.001 |
| United States / Canada | 62.0 | 13.5 | ||
| Other | 61.5 | 14.2 | ||
| Gender | ||||
| Demographic | Mean Score | SD | H | P |
| Female | 62.8 | 13.3 | 22.2 | < 0.001 |
| Male | 62.2 | 14.3 | ||
| Other | 52.8 | 16.0 | ||
| Criterion | Variable | β | B (95% CI) | SE B | t | Sig. t | r2 |
|---|---|---|---|---|---|---|---|
|
Motivation Level |
Sets Goals | 0.311 | 0.177 (0.164 to 0.191) | 0.007 | 25.956 | < 0.001 | 0.379 |
| Manages Thoughts | 0.248 | 0.162 (0.148 to 0.176) | 0.007 | 22.452 | < 0.001 | ||
| Maintains a Healthy Lifestyle | 0.107 | 0.071 (0.058 to 0.085) | 0.007 | 10.668 | < 0.001 | ||
| Manages Environment | 0.110 | 0.080 (0.064 to 0.097) | 0.008 | 9.564 | < 0.001 | ||
| Manages Stress | -0.094 | -0.068 (-0.085 to -0.052) | 0.008 | -8.231 | < 0.001 | ||
| Monitors Behavior | 0.085 | 0.058 (0.043 to 0.073) | 0.008 | 7.581 | < 0.001 | ||
| Excluded variables: Makes Commitments, Manages Rewards (Model 6) | |||||||
| Life Satisfaction | Sets Goals | 0.235 | 0.143 (0.128 to 0.157) | 0.008 | 18.925 | < 0.001 | 0.283 |
| Manages Thoughts | 0.259 | 0.180 (0.164 to 0.196) | 0.008 | 21.871 | < 0.001 | ||
| Maintains a Healthy Lifestyle | 0.134 | 0.095 (0.080 to 0.110) | 0.008 | 12.500 | < 0.001 | ||
| Manages Environment | 0.109 | 0.084 (0.066 to 0.103) | 0.009 | 8.889 | < 0.001 | ||
| Manages Stress | -0.060 | -0.047 (-0.065 to -0.028) | 0.009 | -4.980 | < 0.001 | ||
| Makes Commitments | -0.023 | -0.018 (-0.033 to -0.002) | 0.008 | -2.229 | 0.026 | ||
| Excluded variables: Monitors Behavior, Manages Rewards (Model 6) | |||||||
| Professional Success | Sets Goals | 0.250 | 0.153 (0.137 to 0.169) | 0.008 | 18.902 | < 0.001 | 0.187 |
| Manages Thoughts | 0.165 | 0.116 (0.098 to 0.133) | 0.009 | 13.053 | < 0.001 | ||
| Manages Environment | 0.129 | 0.101 (0.081 to 0.121) | 0.010 | 10.040 | < 0.001 | ||
| Manages Stress | -0.093 | -0.072 (-0.092 to -0.053) | 0.010 | -7.170 | < 0.001 | ||
| Maintains a Healthy Lifestyle | 0.064 | 0.046 (0.030 to 0.062) | 0.008 | 5.579 | < 0.001 | ||
| Excluded variables: Makes Commitments, Monitors Behavior, Manages Rewards (Model 5) | |||||||
3.3. Competency Differences
The average total score on the test was 62.5% (13.7). Average scores on each of the eight competencies, from highest to lowest, were as follows: Manages Thoughts (M = 73.3 [20.7]), Manages Stress (M = 70.3 [18.6]), Manages Rewards (M = 66.3 [18.0]), Manages Environment (M = 64.9 [18.5]), Sets Goals (M = 64.0 [23.7]), Monitors Behavior (M = 57.5 [19.7]), Makes Commitments (M = 55.3 [18.5]), and Maintains a Healthy Lifestyle (M = 49.1 [20.3]).
3.4. Regressions and Factor Analysis
Linear regression was used to determine which of the eight competencies best predicted the three self-reported criterion measures. These analyses pointed strongly and consistently towards the importance of two of the eight competencies: Sets Goals and Manages Thoughts (Table 3). The ranking of the eight competencies according to their overall predictive value (based on the regression coefficients) was poorly correlated with the ranking of these competencies according to how highly participants scored on each one (ρ = 0.0, r = 0.0). Sets Goals, for example, which ranked highest in predictive value, ranked fifth in competency scores (see Competency Differences, above).
To examine patterns of responding on the test, we performed an exploratory principal components factor analysis that included all 32 scored items. The appropriateness of our data for factor analysis was confirmed by a high Kaiser-Meyer-Olkin measure of sampling adequacy (0.93) and a significant Bartlett’s test of sphericity (P < 0.001). Overall, the analysis yielded seven distinct, interpretable, and statistically sound components, all of which corresponded fairly closely to the competency areas that were used to formulate the test: (1) Improves Performance, (2) Monitors Behavior, (3) Manages Goals and Rewards, (4) Maintains Healthy Lifestyle, (5) Manages Thoughts, (6) Manages Commitments, and (7) Manages Stress (Table 4). The percentage of variance accounted for by each of these components varied from 5.4% to 9.7%. The cumulative total percentage accounted for was 49.2%.
A statistician might argue that we should eliminate test items with low factor loadings, or that we should even revise our competencies model to be consistent with the principal components that emerged in the analysis. We agree that this can be done, but we have reservations about proceeding that way in the present instance. Many psychological tests attempt to measure hypothetical constructs such as self-esteem or idealism. Factor or item reduction is often helpful in the development of such tests because these techniques can yield more stable measures of the construct. The EMCI-i, however, measures trainable skill sets that were derived from published studies. The consistency of items within each competency area is greater than the consistency of items within each factor-analytic component. This is not surprising, given that factor-analytical computations are not based on the actual content of the test items. If only for training purposes, therefore, we believe that our original item clusters have more value than the principal components. That said, the results of our factor analysis could in fact be used to streamline the EMCI-i – eliminating less informative items (such as items 9, 11, 16, 30, and 38) (Table 4) and using principal components to provide guidelines for redefining our original components [152].
| Rotated Component Matrix | |||||||
|---|---|---|---|---|---|---|---|
| Component | |||||||
| Item | Improves Performance | Monitors Behavior | Manages Goals and Rewards | Manages Thoughts | Manages Commitments | Maintains Healthy Lifestyle | Manages Stress |
| i32 | .698 | ||||||
| i35 | .682 | ||||||
| i28 | .678 | ||||||
| i40 | .668 | ||||||
| i17 | .638 | ||||||
| i16 | |||||||
| i18 | .667 | ||||||
| i19 | .620 | ||||||
| i5 | .560 | ||||||
| i22 | .521 | ||||||
| i27 | .510 | ||||||
| i11 | |||||||
| i36 | .723 | ||||||
| i2 | .650 | ||||||
| i25 | .604 | ||||||
| i29 | .504 | ||||||
| i9 | |||||||
| i24 | .724 | ||||||
| i6 | .681 | ||||||
| i10 | .493 | ||||||
| i30 | |||||||
| i13 | .648 | ||||||
| i14 | .568 | ||||||
| i1 | .545 | ||||||
| i20 | .478 | ||||||
| i31 | .591 | ||||||
| i34 | .523 | ||||||
| i39 | .514 | ||||||
| i38 | |||||||
| i8 | .626 | ||||||
| i7 | .567 | ||||||
| i21 | .526 | ||||||
| Year | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 |
|---|---|---|---|---|---|---|---|---|---|
| n | 1143 | 733 | 870 | 698 | 532 | 318 | 1257 | 337 | 455 |
| Mean Score (SD) | 61.3 (13.4) |
62.7 (12.3) | 61.6 (12.5) | 61.8 (13.3) | 63.7 (13.1) | 64.6 (12.7) | 66.7 (12.4) | 57.9 (16.6) | 58.3 (15.9) |
| Mean Age (SD) | 35.2 (12.6) |
29.2 (13.3) | 28.6 (13.3) | 25.9 (11.8) | 25.5 (11.5) | 23.6 (10.7) | 34.1 (13.4) | 25.8 (11.4) | 25.1 (11.4) |
| Gender (% Fem.) | 60.9 | 61.7 | 56.8 | 54.0 | 57.3 | 58.5 | 61.9 | 63.2 | 58.9 |
| Race (% White) | 59.7 | 51.4 | 52.0 | 40.7 | 39.5 | 30.8 | 60.1 | 56.7 | 57.8 |
| Year | 2017 | 2018 | 2019 | Significance Test† |
|---|---|---|---|---|
| n | 191 | 187 | 238 | 6,959 |
| Mean Score (SD) | 60.7(15.1) | 59.4 (14.6) | 56.3 (15.7) | 256.7*** (H) |
| Mean Age (SD) | 25.5(10.9) | 26.0 (11.7) | 27.3 (13.3) | 65.6*** (F) |
| Gender (% Fem.) | 59.7 | 47.1 | 55.0 | 32.8** (Χ2) |
| Race (% White) | 44.5 | 54.0 | 55.5 | 201.6*** (Χ2) |
3.5. Changes Over Time
Because the data were collected over a period of 13 years, we also looked for changes over time. Mean total scores per year did change significantly over time, and so did important demographic characteristics of the sample (Table 5).
We found no evidence, however, of significant trends in our data. When we divided our data into two segments – those collected before December 6, 2013 – the midpoint of the time range over which our data were collected – and those collected on or after that date, we found no significant difference between the mean total scores for those two periods (U = 7,708,321.5, P = 0.54, MPreMidpoint = 62.7 [13.1], MPostMidpoint = 62.1 [14.9]).
4. DISCUSSION
We believe that the EMCI-i is the first test that looks relatively comprehensively at trainable competencies that are associated with high levels of motivation. Moreover, by employing a concurrent study design, we were able to produce converging evidence to suggest that the EMCI-i is a valid measuring instrument [79].
Although we relied on a convenience sample (see Limitations, below), it is unlikely that this sample could have systematically affected our main findings, which are as follows: First, goal setting appears to be the most valuable of the eight competency areas we evaluated. This is good news given that goal setting is a particularly simple concept that is relatively easy to train [153, 154]. Second, there appears to be no relationship between the relative value of our eight motivation competencies (apparent from our regression analyses) and the motivation skills people actually have, which suggests a strong need for training motivation competencies. Third, although our study was correlational, not experimental, our results are consistent with the view that motivation training has value – the more training, the better.
5. LIMITATIONS
The greatest limitation in this study might also be considered a strength – namely, that we had no control over the composition of the sample. Our convenience sample was not necessarily representative of the populations of the countries with the most participants in the study (U.S., Canada, and India), nor, for that matter, of any other country. People drawn to a test of motivation skills might be different in nontrivial ways from people who are not drawn to such a test; at the very least, they are probably more interested in boosting their motivational level. Because our data were collected over a period of 13 years, the sample also varied over time in ways we could not control (Table 4); this occurred, presumably, because the list of websites linking to the test kept changing and also because the ranking of the test in search engines was likely shifting. Our sample was non-representative in one obvious respect: 51.8% of our participants had at least a 4-year college degree. In the U.S., only 33.3% of the adult population has earned such a degree [155].
These limitations notwithstanding, a long-term internet study yields a large and diverse sample – in this case, a cleaned sample of 8,349 people from 123 countries. We suggest that such a sample is more valid than the one that has long dominated the social sciences – namely, a few hundred first- or second-year students from a single university [156-158, cf. 159]. We also suggest that the self-selection that undoubtedly occurred in our sample might have been advantageous: Our ideal sample might indeed consist of people who are interested in ways to increase their motivation.
Because we collected data online without identifying our participants (which we did in order to be granted exempt status by our IRB under HHS and APA ethical rules), we were also limited in how we could assess evidence of reliability and validity. We could not measure test-retest reliability, for example. In theory, we could have administered multiple versions of our test to assess alternate-form reliability, but we did not do so. We also had no way to compare EMCI-i scores to those our participants might have achieved on previously validated tests.
Also of concern: Our study relied entirely on self-reported data, and because responses were collected online from unknown individuals, we had no way to confirm the accuracy of those responses or of the demographic information that was provided. That said, a growing body of evidence suggests that people are especially honest when they take online tests anonymously, especially when they are being asked about socially sensitive matters [160-163]. Self-reported data have also been shown to be good predictors of a variety of different behaviors [164, 165]. Other research has also demonstrated the predictive value of self-reported data, especially when test items pinpoint specific behaviors [46, 47, 166, 167]. All the items on the EMCI-i pinpoint behavior in varying degrees. Thus, instead of saying something vague such as, “I’m very reliable,” the EMCI-i includes items such as “I always keep my promises” and “I always record my progress when working on a tough project.”
5.1. Managerial Implications
Because the EMCI-i could conceivably be used in business, we offer a brief analysis of the test’s possible “adverse impact” under the current standards of the U.S. Equal Employment Opportunity Commission (EEOC), which provides statistical standards that must be met by test instruments that could be used to hire, fire, or promote employees [168-170]. The relevant standard that must be met by such tests is called the “four-fifths rule,” according to which “a disparity is actionable when one group’s pass rate is less than four-fifths (80%) of another group’s pass rate” [171]. Our results do not violate this rule for any of the groups we measured by race, ethnicity, or gender.
5.2. Future Research
Our list of competencies was based on the empirical literature that was available when the test was developed (Table 1); it is not exhaustive. It has long been known, for example, that workplace performance (a concept related to motivation) is tied to factors such as autonomy, task variety, quality of personal relationships, opportunities for advancement and other factors [172-175]. Energy level – another concept related to motivation – is affected by diet, disease, and other factors [176-181]. Future revisions of the EMCI-i could be expanded to include additional competencies derived from literatures that were not reviewed when the original test was developed. Both items and competencies could also be improved using a principal components analysis and other statistical techniques.
Because the present study relied on self-report of criterion measures, future research could also look at the extent to which EMCI-i scores are associated with a variety of different outcome measures that should, in theory, be associated with motivation: evaluations by peers or supervisors, for example, or performance measures in the workplace. The value of the EMCI-i could also be assessed in a training context. The first author of this study has developed and evaluated specific training methods that can boost the strength of the eight competencies evaluated herein [153].
CONCLUSION
Our study presents credible evidence showing the value of using a competencies approach to understanding and boosting motivation, with Goal Setting emerging as the most valuable competency among the eight we compared. As David McClelland and his associates showed decades ago, there are distinct advantages to breaking down complex performances implied by vague concepts such as “motivation” into measurable, trainable competencies [44, 45].
NOTES
1 For examples of such training programs, see: the International Board of Standards of Training, Performance, and Instruction (http://ibstpi. org/training-manager-competencies/), the American Management Association (https://www.amanet.org/training/seminars/management-and-supervisory-skills-training.aspx), or the Oxford Management Centre (http://oxford- management.com/training-courses/certificate-series/). Also see the test at https://www.mindtools.com/pages/article/newTMM_28.htm.
2 Nonparametric statistical tests such as Spearman’s rho, the Mann-Whitney U, and the Kruskal-Wallis H are used throughout this study because scores on the EMCI-i lie on an ordinal scale [181]. Unless otherwise indicated, all test scores are reported as a percentage of total correct rather than as raw scores.
ETHICS APPROVAL AND CONSENT TO PARTICIPATE
Exempt approval was granted by the Institutional Review Board (IRB) of the American Institute for Behavioral Research and Technology (AIBRT). AIBRT is registered with the HHS Office for Human Research Protections (OHRP) under IORG0007755. Our IRB is registered with OHRP under number IRB00009303, and the Federalwide Assurance number for our IRB is FWA00021545.
HUMAN AND ANIMAL RIGHTS
No animals were used in this research. All human research procedures were followed in accordance with the ethical standards of the committee responsible for human experimentation (institutional and national), and with the Helsinki Declaration of 1975, as revised in 2013.
CONSENT FOR PUBLICATION
Informed consent from the participants was obtained.
AVAILABILITY OF DATA AND MATERIALS
Anonymized data are available upon requests sent to info@aibrt.org. The data have not been publicly posted because, under the terms of the exempt approval granted by our IRB, we need to take special precautions to protect the identity of our study participants.
FUNDING
None.
CONFLICT OF INTEREST
The authors declare no conflict of interest, financial or otherwise.
ACKNOWLEDGEMENTS
This paper is based on part on a paper presented at the April 2016 meeting of the Western Psychological Association in Long Beach, California. We thank Philip Cheung, Ronald Robertson, Chelsea Song, and Kim Vu for help during the early stages of this project.
