Scoring Services
Handscoring
sm论坛 hand-scores tens of millions of student responses annually using our state-of-the-art Virtual Scoring Center (VSC). This secure, web-based platform integrates all aspects of the scoring process鈥攆rom hiring and training to scoring and reporting鈥攅nsuring accuracy, consistency, and data security. The VSC includes three core systems:
- VSC Capture Acquires and decodes response data from paper-based tests.
- VSC Train: A secure platform that provides consistent training and practice opportunities for raters and scoring leaders.
- VSC Score: Manages user access, scoring activities, and generates real-time reports.
sm论坛's comprehensive training process equips raters with the knowledge and skills necessary to apply scoring criteria consistently and accurately. Through VSC, raters first receive uniform, in-depth training using rubrics and anchor sets of responses to understand the scoring standards. They then practice scoring student responses, refining their skills while receiving feedback. Finally, raters must pass a qualification stage, ensuring they meet performance standards before participating in operational scoring. This structured approach ensures raters are thoroughly prepared to apply the scoring criteria as intended. Building on this foundation, sm论坛's training methods are reinforced by automated quality assurance processes, ensuring that raters continue to apply scoring criteria consistently during live scoring while maintaining high inter-rater reliability. Our automated tools play a critical role in maintaining scoring quality by providing several layers of monitoring and feedback:
- Automated score resets: If scorer agreement on an item falls below quality expectations, recently scored responses are automatically reset and re-evaluated.
- Automated scorer reassignment: Scorers who fail to meet performance thresholds are automatically reassigned or removed from scoring, ensuring that only high-quality raters remain engaged.
- Automated feedback: sm论坛 is the only company offering scorer-specific automated feedback, providing raters with custom, real-time performance insights to correct issues immediately.
Automated mechanisms reduce the need for subjective human analysis of rater performance, allowing human intervention to take a supporting role. Alongside these automated tools, sm论坛 also employs traditional quality assurance methods, such as read-behinds (or backreads), where scoring leaders review a portion of scored responses to ensure accuracy and address discrepancies. Human review of rater performance data complements this process.
Our handscoring services include:
- Rangefinding proceedings,
- Scoring material development,
- Recruitment, training, and supervision of scoring personnel,
- Real-time monitoring and automated feedback to maintain performance.
In summary, sm论坛's handscoring services combine advanced technology, rigorous training, and automated quality assurance processes to deliver accurate, reliable, and scalable scoring solutions. By integrating human expertise with automated oversight, we ensure the highest standards of performance across every stage of the scoring process.
Automated Scoring
sm论坛's leadership in automated scoring is evident through our success in major public competitions such as the Hewlett Foundation's Automated Student sm论坛 Prize (ASAP) and the NAEP Automated Scoring Challenge. In the NAEP Automated Scoring Challenge, sm论坛 was one of four grand prize winners, producing the most accurate score predictions for Reading open-ended items while also demonstrating model interpretability and fairness across diverse student demographic groups. In ASAP Phase 1, sm论坛 achieved the most accurate predictions for student essays, surpassing two sm论坛 raters in reliability. Similarly, in ASAP Phase 2, sm论坛 led the field in scoring short, open-ended responses in English Language Arts and Science, achieving the highest levels of agreement with human raters.
At the core of our automated scoring solution is the Project Essay Grade (PEG) engine, originally developed by Ellis Page, the "father of automated essay scoring." sm论坛 acquired PEG in 2003 and has since expanded it by incorporating computational linguistics, machine learning, and natural language processing techniques. In addition to hundreds of handcrafted linguistic features, PEG uses advanced technologies such as kernel functions, Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA), and even features derived from fine-tuned large language models (LLMs). These advancements enable PEG to capture local context, recognize patterns in writing, and manage complex linguistic relationships, resulting in the most accurate evaluation of student responses.
A key differentiator in sm论坛's automated scoring approach is the way we train the PEG engine. When selecting the training sample, we exclusively use responses scored by our most accurate raters, rather than all responses scored by the larger population of raters. This practice, unique in the industry, provides strong validity evidence and ensures the highest quality of automated scores.
Hybrid Scoring
sm论坛 offers a best-in-class hybrid scoring approach that seamlessly integrates human expertise with advanced automated scoring technologies. This approach leverages the strengths of both methods to meet or exceed human scoring standards in quality, accuracy, and reliability.
Our hybrid system incorporates several features designed to improve score accuracy and validity compared to human-only scoring. These include a confidence measure for each response, dynamic assessments of scorer accuracy, and intelligent response routing. Low-confidence responses鈥攖hose that are difficult to score accurately as they reflect characteristics of multiple score points鈥攁re automatically flagged and routed to human raters for further review, following a human-in-the-loop approach. This ensures that responses requiring additional scrutiny receive the appropriate level of attention, leading to more accurate scoring outcomes.
sm论坛 has successfully implemented hybrid scoring solutions throughout the United States, consistently delivering reliable, timely, and actionable data. By combining human judgment with machine precision, our hybrid scoring model ensures that even the most challenging responses are evaluated with the highest degree of accuracy.
Psychometric & sm论坛 Services
We offer a full range of psychometric services in addition to comprehensive test development services.
Psychometric Services
Our psychometric team, comprised of psychometricians, data analysts, and software developers, offers extensive educational assessment experience and knowledge of best practices. Guided by the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and the National Council on Measurement in Education, 2014), our work is psychometrically sound and adheres to federal peer review requirements. We offer a full range of psychometric services, including:
- field test planning
- operational test design
- equating
- scaling
- classical item analysis
- IRT analysis
- differential item functioning analysis
- alignment studies
- reliability and validity analyses
- technical report production
- standard setting
Our psychometric team also designs, analyzes, and presents findings of wide-ranging psychometric research. See the research page for details.
sm论坛 Services
Item and Test Development Services
We offer comprehensive test development services for standards-based assessments. In addition to developing items and test forms that illuminate instruction, our capabilities include developing item and test specifications, performance and oral assessments, observation checklists, administration manuals, scoring criteria, and interpretation guides. Our test development staff spans content specialists, project directors, editors, graphic artists, and support staff, who are advised by our psychometric team.
Project and Program Management
sm论坛 believes in building and maintaining relationships with customers and employees alike through effective, honest, and ongoing communication. To this end, we employ a distributed leadership model鈥攔ather than a regimented, hierarchical structure鈥攊n which our project and program managers can lead proactively, make decisions independently, and guarantee we exceed client expectations. sm论坛's team of project and program managers has diverse knowledge around multiple aspects of assessment, including holding previous roles in scoring, test development, and psychometrics. In addition, some individuals also have prior classroom experience and hold graduate degrees or are PMP-certified. This expansive range of backgrounds has provided ample opportunities for several team members to work effectively on numerous state-wide programs. To promote continual growth and expand on their experience, sm论坛's project and program managers are supported by our program management office, which provides protocols, tools, feedback, and sm论坛 development. This approach ensures our project and program managers have both the skills and the resources necessary to consistently meet client needs. Finally, we recognize that all projects are dynamic and that they can require give and take when leading many aspects of successful program management.
Document Production, Distribution, Receipt, and Storage
sm论坛 is capable of producing millions of test booklets annually. In addition to our in-house digital printing equipment, we maintain relationships with several local printers.
We have developed a proprietary Order-Pack-Ship (OPS) application used to scan and track test materials. In addition to allowing us to efficiently manage the selection and shipping of orders, our OPS system is built around extensive quality control procedures to ensure that the correct numbers and types of materials are shipped to schools and districts.
We have similarly developed a sophisticated internal tracking system used for logging and processing all materials returned from the field. This system is capable of locating individual test documents at any time they are in our possession.
We maintain a host of high-volume scanners and imaging equipment and use several time-tested processes that ensure all necessary precautions are taken during scanner and document setup, scanner calibration, data validation, and data export. We utilize a double-blind data correction process to achieve the most accurate reporting of student information and test results.
In addition to secure electronic delivery of assessment results, we are capable of producing reports in-house鈥攗sing a variety of high-speed printers鈥攁nd shipping to schools or districts.
Evaluation & School Improvement Services
Program Evaluation
Our team of accomplished researchers assists organizations in all aspects of the research/evaluation process, including leveraging data for program improvement and sustainability. Whether it is evaluating new programs, conducting statewide survey research, or disseminating resources and information on diverse topics, we offer objective, accurate, and sound information that clients can trust.
Technical Assistance and Professional Development Services
We assist school districts with planning and implementing new
initiatives and developing their capacity to create and sustain
meaningful change. We assess the needs of our clients and
provide a continuum of services鈥攖iered technical assistance鈥攖o
meet those needs. We deliver flexible and adaptable assistance
through site-based and remote consultation, workshops,
conferences, and various resources.
Our areas of expertise include:
- Special Education Systems and Practices
- Multi-tiered Systems of Support (RtI and PBIS)
- Early Literacy
- Leadership Development
- Bullying Prevention
- School Safety and Healthy School Climate
- Career & Technical Education
- STEM
- Student Support Services