The E-Proficiency Profile test was developed to assist in the assessment of the outcomes of general education programs to improve the quality of instruction and learning.
The E-Proficiency Profile is a test of college-level skills in critical thinking, reading, writing and mathematics designed to measure the academic skills developed through general education courses, rather than the subject knowledge specifically taught in those courses. All of the subject knowledge required to answer each question is contained in the question itself or in the stimulus materials that accompany the question.
College-level reading questions measure students’ ability to:
-
- interpret the meaning of key terms
- recognize the primary purpose of a passage
- recognize explicitly presented information
- make appropriate inferences
- recognize rhetorical devices
College-level writing questions measure students’ ability to:
-
- recognize the most grammatically correct revision of a clause, sentence or group
- of sentences
- organize units of language for coherence and rhetorical effect
- recognize and reword figurative language
- organize elements of writing into larger units of meaning
Critical thinking questions measure students’ ability to:
-
- distinguish between rhetoric and argumentation in a piece of nonfiction prose
- recognize assumptions
- recognize the best hypothesis to account for information presented
- infer and interpret a relationship between variables
- draw valid conclusions based on information presented
Mathematics questions measure students’ ability to:
-
- recognize and interpret mathematical terms
- read and interpret tables and graphs
- evaluate formulas
- order and compare large and small numbers
- interpret ratios, proportions, and percentages
- read scientific measuring instruments
- recognize and use equivalent mathematical formulas or expressions
There are two versions of the E-Proficiency Profile test:
- a two-hour Standard form, intended to provide information about individual students or groups of students
- a 40-minute Abbreviated form, intended to provide information about groups of 30 or more students
The E-Proficiency Profile test yields two types of scores:
- scaled scores, which are numeric and norm-referenced
- proficiency classifications, which are categorical and criterion-referenced
Standard Test Form
The Standard form of the E-Proficiency Profile test is intended to provide information about individual students, as well as groups of students. It consists of 108 questions, divided into two sections of 54 questions each. The two sections may be administered either in a single two-hour testing session or in separate testing sessions of one hour each.
The Standard form includes:
- 27 questions testing critical thinking skills
- 27 questions testing reading skills
- 27 questions testing writing skills
- 27 questions testing mathematics skills
Most of the questions testing reading and critical thinking skills are in sets of two to four questions based on a common stimulus, such as a reading selection. Many of these sets include both reading and critical thinking questions. If the set includes only critical thinking questions, the stimulus may be something other than a reading passage, e.g., a picture or a graph.
The questions are sequenced in clusters:
- a group of reading and critical thinking questions
- a group of writing questions
- a group of mathematics questions
- a group of critical thinking questions, etc.
This sequence prevents all the questions measuring a particular type of skill from appearing late in the test. Approximately half of the questions testing each type of skill appear in the first section of the test and half in the second section. The questions on the test are arranged in blocks of three to eight questions, with the questions in each block testing the same types of skill: reading and critical thinking skills, writing skills or math skills.
Most of the reading and critical thinking questions are based on information presented in a brief reading selection, picture, graph, etc. Each question measuring critical thinking or reading skills is associated with a particular academic context: humanities, social sciences or natural sciences. The test includes nine critical thinking questions and nine reading questions in each of these three academic contexts. The table that follows illustrates this structure.
Abbreviated Test Form
The Abbreviated form of the E-Proficiency Profile test is not intended to provide information about individual students. It is intended to provide information about groups of at least 30 students. It can be administered in a single 40-minute testing session. The Abbreviated form is a partitioning of the Standard form into three smaller forms of 36 questions each, “Abbreviated Form A,” “Abbreviated Form B” and “Abbreviated Form C.” These three Abbreviated forms are spiraled in alternating sequence, so that each of them is taken by one-third of the students. The 108 questions in the Standard form are assigned to the three Abbreviated forms to make each of them – insofar as possible – a miniature version of the Standard form.
Each of the three Abbreviated forms includes:
- nine questions measuring critical thinking skills
- nine questions measuring reading skills
- nine questions measuring writing skills
- nine questions measuring mathematics skills
The Abbreviated form provides too small a sample of each student's performance to permit the reporting of individual scores (except for a total score based on all 27 questions). A student who takes the Abbreviated form is taking only one-third of the test, and the individual scores are not a reliable indication of the scores the student would have received on the full 108-questions test. However, the large the group of students, the more these inaccuracies will tend to cancel each other out.
Score Data
Eight scaled scores are reported for students taking the E-Proficiency Profile test:
- a total score
- four skills subscores (critical thinking, reading, writing, mathematics)
- three context-based subscores (humanities, social sciences, natural sciences)
The total score is reported on a scale that extends from 400 to 500. The seven subscores are reported on a scale that extends from 100 to 130.
Using the Standard Form versus the Abbreviated Form
Some institutions may choose to use the E-Proficiency Profile test to assess the skills of individual students. Others may use it only to characterize the skills of groups of students, for example, an incoming freshman class or a graduating senior class. In selecting a test to assess general education outcomes, an institution should begin by considering its purpose in wanting to test. How will the test results be used? Is it important to assess the skills of each individual student, or is it sufficient to assess a class of students as a group?
The Abbreviated form requires only 40 minutes of testing time, but it provides only group information – a set of statistical reports for each group of students – or “cohort” – tested plus some additional information on subgroups of the students determined from the demographic data. The Abbreviated form is constructed by dividing the Standard form into three parts, and packaging them in alternating sequence (A, B, C, A, B, C, A, B, etc.) so that each part is taken by one-third of the students. The alternating sequence makes it likely that the groups taking the three parts will be similar, particularly if the number of students is fairly large. This sampling technique (sometimes called “matrix sampling”) makes it possible to obtain reliable information about the group even when no individual student answers enough questions to provide reliable individual subscores.
The Standard form requires two hours of testing time, but it provides scores and proficiency classifications for individual students, in addition to the group information provided by the Abbreviated form. The individual information can be used in advising students and in making decisions about individual students.
Data from both the Standard and Abbreviated forms can also be used to compute statistics that are not routinely provided, for the full group or for subgroups of the students tested. With the eight scaled scores and the nine proficiency classifications for each student tested, it is possible to aggregate and disaggregate the data in ways that expand the institution’s information base.
An institution’s decision to use the Standard form versus the Abbreviated form of the E-Proficiency Profile test will depend mainly on the institution’s purpose in testing. An institution must decide whether to give up the individual scores and proficiency classifications provided by the Standard form in exchange for the reduction in testing time offered by the Abbreviated form. As the needs and priorities of a particular institution evolve, the institution can consider switching from one form to another based on the different benefits these different forms offer. Because the Abbreviated form is derived from the Standard form, the Abbreviated form is also statistically equated to the Standard form – making the scores on each form fully comparable to scores on the other form. The two forms can be used interchangeably and scores from each can be compared with full confidence that they mean the same thing and can be interpreted the same way. This also implies that aggregation of data from both the Standard and Abbreviated forms is possible. However, because subscores and proficiency classifications are not considered adequately reliable at the student level, reports derived from a combination of both Standard form and Abbreviated form test takers only include summary data.
How Colleges Use E-Proficiency Profile
Growth Measurement
The E-Proficiency Profile test can be used to assess students’ growth in reading, writing, critical thinking, and mathematics skills by testing the same students at different times in their educational careers, e.g., as incoming freshmen and as rising juniors or as graduating seniors. However, a group of incoming freshmen will include many students who will not be available to be tested as rising juniors or as graduating seniors. Therefore, this use of the ETS Proficiency Profile test requires individual scores. The Abbreviated form provides individual total scores, but no individual subscores or proficiency classifications. To measure growth in the specific types of skills reflected in the subscores and proficiency classifications, the institution must use the Standard form. The scaled scores are generally better than the proficiency classifications for this purpose, because some students (particularly those who are quite weak initially) may make gains that are not large enough to change their proficiency classifications. These gains will be reflected in their scaled scores.
Trend Indicator
Some colleges use the E-Proficiency Profile test to look for year-to-year changes in the skill levels of their incoming freshmen, their rising juniors, or their graduating seniors. Both the scaled scores and the proficiency classifications can be used for this purpose, although the scaled scores may reflect differences that are not apparent in the proficiency classifications. In these year-to-year comparisons, the groups being compared consist of different students; there is no need to match students’ scores from one testing with their scores from another testing. Either the Standard form or the Abbreviated form (if the groups are large enough) will provide the necessary data.
Comparisons with Other Institutions
Some colleges use the E-Proficiency Profile test to determine how their students’ skills compare with the skills of students at similar institutions. It is important that these comparisons involve students at the same point in their educational careers (entering freshmen, etc.). The information about the performance of students at other institutions can come from the Comparative Data. The norm-referenced scaled scores are particularly useful for this purpose, although the criterion-referenced proficiency classifications can also be used. Either the Standard form or the Abbreviated form (if the groups are large enough) will provide the necessary data.
Counseling Tool
Some colleges test their students to identify those whose reading, writing or mathematics skills need improvement, so that they can advise these students to take courses specifically aimed at improving those skills. Some colleges test their students twice, as entering freshmen and as rising juniors, to identify the skills that the students need to improve as freshmen and the skills that they still need to improve as rising juniors. The proficiency classifications are likely to be especially useful for this purpose, because each proficiency level is associated with a particular set of specific skills. This use of the E-Proficiency Profile test requires the Standard form.
Recruitment Aid
Some colleges administer the Standard form of the E-Proficiency Profile test to the same students as incoming freshmen and as rising juniors or as graduating seniors. They use this information to identify those students whose skills improved substantially. They then look for information in the students’ educational background – information available at the time of admission – that distinguishes these students from those whose skills did not improve. This information can help the college focus its recruitment efforts on students who are likely to benefit from the college’s instructional program. Both the scaled scores and the proficiency classifications are useful for this purpose. This use of the E-Proficiency Profile test requires the Standard form.
Guidelines for Test Use
Institutions planning to use the E-Proficiency Profile test should be aware of its possibilities and its limitations. The following guidelines are provided to assist institutions in using the E-Proficiency Profile test appropriately.
Test Purpose
Review examples of the test items and the list of skills it measures to verify that the skills the test measures are those that the institution seeks to measure. Examine samples of the score reports and statistical reports, to verify that they include the information that the institution needs. If the institution needs information about individual students, make sure to use the Standard form, not the Abbreviated form.
Selecting Students
If the purpose of testing is to make inferences about the performance of groups of students, it is important to test an adequate number of students from each of those groups, selected in such a way that the students tested from each group are representative of the group as a whole. The best way to accomplish this is to test all of the students. If the institution tests only a sample of the students, it is important that the sample include an adequate number of students from each group about which the institution wants information, selected in a way that will permit the results to be generalized to the group as a whole. It is particularly important not to limit the testing to students who volunteer to be tested, unless the institution wants information that applies only to those students.
Student Motivation
Student motivation in assessment testing is a serious concern. If the students are not motivated to do well on the test, their test scores will not reflect their actual skill levels.
Limitations of Test Scores
A test contains only a sample of the tasks that students are expected to be able to do. On another sample of tasks designed to measure the same skills, the students might perform somewhat differently. Information provided on the score reports (see “Confidence Limits”) and elsewhere in this User’s Guide (see “Reliability of Scaled Scores”) enables the user of the scores to determine how much the scores could be expected to differ if a different set of tasks were used.
The reliability of the individual scores of students taking the Standard form – particularly, the skill area scores and the proficiency classifications – should be adequate for counseling purposes and for the identification of students with problems in particular skill areas. These scores are not reliable enough to use as the basis for high-stakes decisions about individual students.
The E-Proficiency Profile test measures a specific collection of skills. It does not and cannot measure all the educational outcomes of interest to institutions of higher learning. When the E-Proficiency Profile test is used to evaluate an institution or any of its programs, it should be used in conjunction with other information. It should never be used as the sole means for evaluating the effectiveness of an institution or the educational progress of the students.
Limitations of Comparative Data
The data in the Comparative Data Guide are drawn entirely from institutions that use the E-Proficiency Profile test. Within any category of institutions, those that use the E-Proficiency Profile test are not likely to be representative of all institutions in that category. In addition, the numbers of students tested and the sampling procedures vary from one institution to another, and it is impossible to verify that the students tested at each institution are representative of all the institution’s students at the relevant class level (freshman, sophomore, etc.).