The process of developing certification examination questions

It is a long and costly process for ASQ that involves several hundred hours of work by teams of certified professionals attending various sessions. The key workshops involved in the exam development process are

(a) Job Analysis Workshop,

(b) Test Specification Workshop,

(c) Item Writing,

(d) Item Review

(e) Exam Review. Participants for every session are to read details of the process and relevant details as preparation before every workshop.

(a) Job Analysis Workshop: This workshop of 12 Subject Matter Experts (SMEs) results in setting the foundation for the Job Analysis Survey. The survey is mailed to about 2000 SMEs in that field (irrespective of ASQ membership) to determine what knowledge and skills are appropriate to test for the specific certification exam.

(b) Test Specification Workshop: The tasks and knowledge areas obtained from job analysis are then assembled into an outline format with topics and categories that logically group to form Body of Knowledge (BOK) for the specific certification category. The SMEs then develop explanatory subtext that clearly describes the particulars of each subtopic. The number of questions and the time limit for taking the exam are also determined at this workshop.

(c) Item Writing Workshop: In this workshop 24 SMEs who are trained in item writing develop question items in small groups. Each item (assessment professionals use this term instead of question) that is developed in this session must have the following:

- Only 1 key (correct answer) that must have team agreement.
- A reference to support the key.
- A rationale for the item which explains what the item is testing and why the key is correct and the other options are not.
- Classification to the most detailed level of the BOK.

Item writing workshops typically produce 200-300 items over a two day period. The items produced are considered `raw’ and will be reviewed by another group of SMEs at an Item Review Workshop.

(d) Item Review Workshop: About 4 weeks before the meeting , the twelve SMEs are sent the items to be reviewed in the workshop. They answer the items as if they were candidates sitting for the exam. While they review the items they are expected to make comments about them and submit those comments to the ASQ test developer. The comments from all SMEs are then compiled into one document, which is used for item review workshop. The SMEs attending this workshop are trained in a variety of assessment issues. Their focus is to review the items to ensure that there is only one key and that the item is clearly written and accurate. They also confirm the reference and rationale for each item. Once all components of the item are verified the item is approved and can subsequently be selected for use in an exam.

(e) Exam-Review Workshop: Once the raw items are reviewed and approved, the ASQ test developer creates a draft exam in accordance with the test specifications, and reviews it for overlap or cueing between items. The12 SMEs attending this workshop take the exam just as any candidate would. As in the case of Item Review discussed above, they are asked to comment on the items and send those comments to ASQ Test Developer. At the workshop the members are trained in a variety of assessment issues and then asked to focus on items identified as “problematic” from the review prior to the workshop. Once all the issues in the exam are resolved, the committee approves the exam and it can then be used for test administration.

The process of determining a passing grade

Whenever an exam is administered with a new body of knowledge, the exam is subjected to a cut score process. A group of 12-14 certified professionals (other than those who were involved in developing the questions or reviewing the exam) is asked to rate each question in the new exam in terms of difficulty for a minimally qualified examinee or candidate. In other words the team is asked "What % of minimally qualified candidates for the exam will get this question correct?"

The cut score study participants consider many factors such as the basic or advanced level of knowledge required to know the right answer, any potential errors a candidate can make because of the various choices given, the way the question is presented (including the use of English language styles and words), etc. Whenever members of the cut score panel show an expected performance score that is more than 25 points apart from other panel’s rating, the difficulty factors for that question are debated and discussed until a consensus is reached by the group.

The final cut score developed by the cut score study panel is presented to the ASQ Certification Board as the recommended percent or portion correct needed to pass the test. See Appendix-B for a sample of the item statistics template exam stats template used for discussions during the cut score study and a sample summary table of what the ASQ Certification Board sees when determining the cut score of a new body of knowledge.

What is described above is a very brief description of the cut score process. I was a member of the cut score study panel for the CQA exam administered in June of this year, and this is the process we used to set the cut for that new body of knowledge. It is the same process ASQ uses on all of its exams.

Note: The procedures ASQ uses to develop tests and establish pass point are in accordance with those described in the ANSI 17024 standard; these procedures are considered accepted practices as outlined in the Standards for Educational and Psychological Testing, which are developed by the American Educational Research Association, (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME).