Introducing Sociotechnical Scoring Methodology

Abstract

STEM education has outgrown the methods used to evaluate it. Schools, kits, toys, platforms, books, and camps all claim STEM value, yet very few of those claims are ever tested against evidence. The reviews that do exist tend toward one of two failures: unaccountable expert impression, or checklist scoring that mistakes vocabulary for substance.

This paper proposes Sociotechnical Scoring Methodology (SSM) as a structured alternative, and positions it as the evaluation layer of a three-part architecture. CLEO supplies the operating system that classifies and routes recognition; Dynamic Standards Consensus supplies the governance that determines what the standards are; SSM supplies the method by which a routed submission is scored against those standards.

SSM is not a credentialing system. It is a scoring method from which credentials, trustmarks, ratings, or improvement reports may be issued afterward. Within the framework, artificial intelligence processes evidence, detects patterns, and supports reviewer consistency, while human reviewers define the construct, weigh the evidence, and retain final authority over judgment. Consistent with CLEO’s treatment of AI as a replaceable utility, the method is designed to remain fully operable when AI assistance is reduced or unavailable. A machine can locate a claim, but it cannot determine what the claim is worth.

Keywords: Credentialing Logic for Educational Operations (CLEO), Dynamic Standards Consensus (DSC), Sociotechnical Scoring Methodology (SMS), AI-augmented assessment, STEM evaluation, hybrid intelligence, human-in-the-loop assessment, educational credentialing, program evaluation, trustmarks

Introduction: The Problem of Unverified Claims

A STEM label carries public trust. Parents make purchasing decisions on the strength of it, schools adopt curricula because of it, funders allocate resources against it, and manufacturers print it on the box. Almost none of it is independently checked.

STEM.org Educational Research has maintained a categorical taxonomy since its inception, distinguishing certification for individuals, accreditation for institutions, authentication for products, review for published content, and endorsement for aligned initiatives. That taxonomy exists for a substantive reason: a robotics kit and a district STEM program are different objects, and neither can survive a rubric designed for the other. Much of the wider market has not maintained these distinctions, adopting the language of STEM credentialing while altering its structure, so that institutions are certified, products are treated as programs, and participation is mistaken for qualification.

The evaluative question, therefore, should not begin with whether an artifact deserves a seal. It should begin earlier, by asking what is being claimed, what evidence stands behind the claim, how strong that evidence is, and how trained reviewers should interpret it within a defined framework of STEM quality.

Three Layers: Assessment, Scoring, and Credentialing

Much of the confusion in this field comes from collapsing three distinct acts into a single word. It is worth separating them explicitly:

1. Assessment is the process of gathering, organizing, and interpreting evidence.
2. Scoring is the structured conversion of that evidence and the accompanying judgment into a documented evaluative result.
3. Credentialing is the subsequent act of issuing a status, seal, trustmark, or public-facing validation on the basis of that result.

This distinction has precedent in educational measurement. The Standards for Educational and Psychological Testing, issued jointly by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education, distinguish tests from broader assessments and observe that assessment can integrate test information with other sources of evidence. The same source notes that renaming a method does not change its underlying substance, and that sound practice requires validity, technical quality, fairness, and professional judgment.

SSM accordingly treats credentialing as an output condition rather than as the method itself.

Why the Framework Must Be Sociotechnical

A robotics kit is not merely an object. Its educational value depends on the quality of its instructions, the age of the learner, the support available from a teacher, the accuracy of the underlying science, the adequacy of its safety documentation, and the setting in which it is used. A school program is not simply a course schedule; it comprises pedagogy, leadership, teacher preparation, student belonging, partnerships, and outcomes that accumulate over years.

The evaluation itself is subject to the same conditions. The AI model is not an independent judge, but rather one component within a larger human-governed workflow.

The NIST AI Risk Management Framework identifies the characteristics of trustworthy AI as validity and reliability, safety, resilience, accountability, transparency, explainability, privacy, and fairness. It also states that these characteristics are inseparable from social and organizational behavior, and that human judgment should be employed when determining the specific metrics and threshold values attached to them. SSM extends that principle into STEM evaluation, asking not whether AI can score an artifact but how AI, expert reviewers, published standards, evidence, and institutional accountability can together produce a defensible result.

Hybrid Intelligence and the Division of Labor

Human reviewers contribute contextual interpretation, ethical judgment, disciplinary experience, and tolerance for ambiguity, while AI contributes scale, pattern recognition, document comparison, summarization, and consistency support. Neither is sufficient on its own. Research on hybrid intelligence learning environments describes human and AI actors working together to produce outcomes that neither could reliably reach in isolation.

This complementarity matters in STEM evaluation because the evidence base is heterogeneous by nature. A single review may require reading lesson plans, product manuals, curriculum maps, standards alignment charts, video demonstrations, safety documentation, user data, and marketing materials. AI can move through that volume quickly, but only a human reviewer can determine what it means. Work on designing knowledge-in-use science assessments arrives at a similar division, treating human judgment as essential for creativity, ethics, ambiguity, and context, and AI as well suited to volume, pattern detection, and repeatable analytic tasks.

The degree of collaboration should itself be treated as a design decision rather than a fixed default. Research in production management proposes calibrating that degree against factors including data availability, variability, error susceptibility, and decision complexity. The same logic applies here: a high-stakes safety determination and a routine standards crosswalk do not belong at the same point on that scale.

Foundations in Program Evaluation and Educational Measurement

SSM draws on two established traditions rather than inventing criteria from nothing.

Program evaluation already has standards. The Joint Committee on Standards for Educational Evaluation identifies utility, feasibility, propriety, accuracy, and evaluation accountability as the domains of evaluative quality, emphasizing stakeholder usefulness, transparency, valid and reliable information, explicit reasoning, and documentation.

Educational measurement contributes further requirements, including construct clarity, validity evidence, reliability, scoring consistency, and responsible interpretation. The Standards for Educational and Psychological Testing are direct about what is at stake, noting that assessments shape decisions about individuals and programs and that poor use can cause harm.

STEM quality is itself multidimensional. The National Research Council’s work on K-12 STEM education measures success relative to defined goals, including STEM content and practices, positive dispositions, workforce preparation, STEM literacy, and broadened participation. More recent National Academies guidance on undergraduate STEM teaching adds active engagement, student-centered learning, belonging and identity, data-driven improvement, flexibility, and transparency. None of this reduces to whether a package mentions engineering.

Defining Sociotechnical Scoring Methodology

Sociotechnical Scoring Methodology is a structured human-AI evaluation framework for scoring STEM education programs, products, and published content through documented evidence review, rubric-based analysis, expert judgment, AI-supported evidence processing, reviewer calibration, and accountable reporting.

Six commitments define the framework:

• It is construct-driven. Every review begins by defining what STEM quality means for the specific object under examination, because a toy, a school program, a digital platform, and a children’s book cannot be scored against identical assumptions.
• It is evidence-weighted. Scores reflect the quality and relevance of the evidence submitted rather than the confidence of the claims made by applicants, publishers, or manufacturers.
• It is AI-augmented but human-governed. AI may extract evidence, map claims, detect anomalies, and check consistency, but trained human reviewers remain responsible for interpretation and for the final score.
• It is degradable by design. AI is treated as a replaceable utility rather than a foundational component, and the method must remain fully operable under reduced or absent AI assistance, with the same rubric, the same reviewers, and the same evidentiary standard applied manually.
• It is auditable. Every score traces back to specific evidence, rubric language, reviewer notes, model-assisted outputs, and adjudication decisions.
• It is credential-neutral until completion. The methodology produces a score and a recommendation; credential issuance occurs only after the scoring process has concluded.

Position Within the Stack: CLEO, DSC, and SSM

SSM does not operate in isolation, and it should not be read as a freestanding proposal. It occupies the evaluation layer of a three-part architecture, each layer of which answers a different question.

CLEO — Credentialing Logic for Educational Operations is the trustmark operating system. It answers the question of what kind of recognition is at issue, converting intent into classification and classification into evaluation. A teacher’s application routes toward certification; a product submission routes toward authentication. CLEO also establishes two architectural commitments that SSM inherits directly: the separation of authority from automation, and the treatment of artificial intelligence as a replaceable utility rather than a foundational dependency.

Dynamic Standards Consensus (DSC) is the governance layer that runs on CLEO. It answers the question of what the standards actually are, distributing that authority across the verified network of trustmark holders through credential-gated participation rather than concentrating it in a committee or an advisory board. Among the matters DSC governs are the evaluation criteria themselves and the weighting of rubric components.

Sociotechnical Scoring Methodology (SSM) answers the remaining question: how a routed submission is scored against governed criteria. It supplies the evidentiary discipline, the dimensional rubric, the reviewer workflow, and the audit trail.

The dependency runs in one direction, and it matters considerably. CLEO determines which rubric applies. DSC determines what the rubric contains and how its components are weighted. SSM determines what the evidence shows. A methodologist does not get to set the weights; a governance process does. Any version of SSM that fixed its own weights by editorial preference would reintroduce precisely the concentration of authority that DSC exists to prevent.

Two points of scope should be stated plainly. First, SSM as presented here addresses three of the five taxonomic categories — accreditation, authentication, and review — because programs, products, and published content share an evidentiary structure. Certification of individuals and endorsement of aligned initiatives rest on different constructs and are not treated in this paper. Second, the vocabulary should be kept distinct: AI-augmented assessment names the process, Sociotechnical Scoring Methodology names the formal method, hybrid intelligence names the theoretical umbrella, CLEO names the operating system, DSC names the governance model, and credentialing names the downstream recognition event.

That separation prevents a category error. The organization is not using AI to credential STEM products. It operates a trustmark system whose standards are set by distributed consensus, whose logic is deterministic, and whose evaluation method uses AI to process evidence while humans retain authority over judgment.

Object Domains

The framework must accommodate three domains, each with distinct evaluative demands.

Programs include schools, camps, after-school initiatives, professional development systems, courses, and academies. Their evaluation requires attention to learning goals, staff qualifications, curriculum design, learner outcomes, implementation fidelity, assessment systems, safety, equity, and sustainability.

Products include kits, toys, robotics systems, applications, manipulatives, laboratory materials, and simulations. Their evaluation requires attention to age appropriateness, safety, usability, disciplinary accuracy, learning affordances, documentation, accessibility, and whether the learner is doing STEM work rather than passively observing it.

Published content includes books, videos, curricula, lesson collections, digital courseware, and instructional media. Its evaluation requires attention to factual accuracy, conceptual progression, representation of STEM practices, clarity, bias, accessibility, and whether the material genuinely invites inquiry.

These domains overlap frequently. A robotics curriculum sold alongside a kit is both a product and published content, and a digital platform bundled with teacher training may function as all three. Where an artifact spans categories, CLEO’s routing logic determines which rubric governs, and the scoring architecture must be modular enough to accommodate that determination. A single universal rubric would misrepresent what it claims to measure.

The Scoring Architecture

Each dimension is scored on a five-point scale.

The Ten Evaluation Dimensions

The dimensions below represent a proposed starting set rather than a fixed schedule. Under DSC, both their composition and their relative weighting remain subject to revision by the verified network.

1. STEM Integrity. Is the content accurate, meaningful, and educationally defensible? AI can flag STEM claims, terminology, unsupported assertions, and passages requiring factual verification, while human reviewers judge disciplinary accuracy, developmental appropriateness, conceptual depth, and whether the STEM content is authentic rather than decorative.
2. STEM Integration. Do science, technology, engineering, and mathematics genuinely connect? AI can map references to disciplines, standards, practices, and stated learning objectives, while human reviewers determine whether that integration is substantive or merely cosmetic.
3. Inquiry and Design. Does the artifact promote investigation, experimentation, modeling, problem solving, or engineering design? AI can identify inquiry language, design cycles, experimentation prompts, and learner tasks, while human reviewers judge the depth of inquiry, the degree of learner agency, and the authenticity of the design challenge.
4. Pedagogical Quality. Does the artifact support how people actually learn? AI can extract learning objectives, sequencing, scaffolds, vocabulary levels, and formative prompts, while human reviewers evaluate cognitive load, age appropriateness, progression, and fit with established principles of STEM learning.
5. Evidence of Impact. Is there evidence that the artifact advances STEM learning? AI can summarize studies, pilots, usage data, internal reports, and outcome claims, while human reviewers weigh the credibility, relevance, independence, and sufficiency of what has been submitted.
6. Accessibility and Inclusion. Can diverse learners meaningfully access and engage with the artifact? AI can flag accessibility claims, readability, representation patterns, language complexity, and accommodation statements, while human reviewers judge whether inclusion is genuinely embedded or simply asserted.
7. Implementation Support. Can educators, parents, and institutions use the artifact effectively in real settings? AI can compare guides, manuals, onboarding documents, and training materials, while human reviewers judge feasibility, clarity, required expertise, and the time and resource demands placed on the user.
8. Assessment and Feedback. Does the artifact offer meaningful ways to observe learner progress? AI can identify embedded assessments, rubrics, reflection prompts, dashboards, and feedback loops, while human reviewers judge their validity, usefulness, and educational relevance.
9. Safety and Ethics. Are physical, digital, developmental, privacy, and ethical risks disclosed and managed? AI can flag safety warnings, privacy policies, age restrictions, data collection practices, and hazardous materials, while human reviewers determine severity, mitigation quality, and whether the residual risk is acceptable for the intended users.
10. Sustainability and Improvement. Can the artifact improve over time? AI can track revision history, update cycles, versioning, and user feedback, while human reviewers judge organizational capacity, responsiveness, and whether iteration is genuinely built into the model.

The purpose of this structure is not to count STEM vocabulary or to reward polished marketing language, but to determine whether the conditions for credible STEM learning are actually present.

Evidence Strength Ratings

Every score should be reported alongside an evidence strength rating. The score rates the artifact; the evidence rating rates the proof behind the score. Reporting one without the other allows a well-documented product to outrank a better one that is less thoroughly packaged.

• A — Strong Independent Evidence. The artifact is supported by peer-reviewed research, third-party evaluation, rigorous outcome data, or externally validated findings.
• B — Structured Internal Evidence. The artifact is supported by internal evaluation, structured pilot data, documented implementation outcomes, or organization-generated evidence with a clear methodology. Such evidence can be credible, but it is not independent.
• C — Strong Documentation Without Outcome Data. The artifact includes complete materials, clear documentation, sound instructional design, and standards alignment, but offers limited direct evidence of outcomes. This pattern is common among newer products, publications, and programs.
• D — Claim-Based or Limited Evidence. The artifact is supported primarily by applicant assertions, marketing language, testimonials, or incomplete documentation. This does not mean the artifact is weak, but it does mean the claims remain unproven.
• E — Insufficient or Contradictory Evidence. The artifact lacks adequate documentation, contains unverifiable or conflicting claims, or does not make enough evidence available for responsible scoring. This rating may block recognition until the deficiency is resolved.

A product may demonstrate strong design while offering only moderate evidence of impact, and a program may be genuinely promising while lacking formal outcome data. SSM reports both findings rather than averaging them into a single figure that conceals the difference.

The Human-AI Workflow

The methodology can be implemented as an eight-stage process.

1. Routing and intake. CLEO classifies the submission and determines which trustmark category and rubric apply. AI then extracts the explicit and implicit claims contained in the submitted materials, and human reviewers verify the resulting claim map and determine which claims fall within scope.
2. Evidence mapping. AI maps the submitted materials against rubric dimensions and identifies gaps, and human reviewers decide which of those gaps are material to the evaluation.
3. Preliminary AI-assisted analysis. AI produces a non-binding analytic brief containing document summaries, standards crosswalks, possible inconsistencies, safety flags, and reading-level estimates. This output is treated as an aid to review rather than as a decision, and its absence slows the review without invalidating it.
4. Independent human review. At least two trained reviewers with appropriate domain expertise score the artifact independently.
5. Divergence analysis. Reviewer scores are compared, and significant discrepancies trigger adjudication. Disagreement is treated as useful signal that a construct or scoring rule requires closer interpretation, not as a procedural failure.
6. Adjudication and consensus. A lead reviewer or panel resolves the discrepancies, documents the reasoning, and may request additional evidence before proceeding.
7. Scorecard and recommendation. The process produces a scorecard, an evidence profile, narrative findings, a limitations statement, improvement recommendations, and a credential recommendation where applicable.
8. Credentialing after the fact. Only once the score is final does credentialing occur, subject to thresholds, conflict-of-interest review, legal and brand review, and registry publication.

Stages one through three are where AI assistance concentrates, and each carries a defined manual fallback. Stages four through eight are human by construction.

Formal Scoring Logic

The method should be transparent enough that any party can reconstruct how a final score was produced. One workable structure is as follows:

Dimension Score = Human Consensus Score × Evidence Strength Modifier − Risk Adjustment

Final SSM Score = 25 × weighted mean of adjusted dimension scores

Because the base rubric uses a 0–4 scale, multiplying the weighted mean by 25 converts the result to a 100-point scale. The human consensus score remains primary throughout, and AI does not determine the final figure unless that specific AI component has been separately validated for the domain and use case in question.

The weights applied to that mean are not editorial constants. They are governance outputs, set and revised through Dynamic Standards Consensus, recorded in CLEO, and visible to the verified network. Every published score should therefore carry the version of the weighting schedule under which it was produced, so that a result from one period can be read correctly against a result from another.

Both the total score and the dimensional profile should be reported. In most cases the profile is the more useful document, because it explains why the artifact earned its result and identifies what would move it.

Validity, Reliability, and Fairness

A rubric is not valid because its author declares it so. Validity is built over time through evidence that the instrument measures what it claims to measure, that scores are interpreted appropriately, and that the resulting decisions are fair and useful. Several requirements follow:

• Construct validity requires that STEM quality be defined separately by domain, since product quality and school quality are not the same construct.
• Content validity requires that the rubric dimensions genuinely represent the domain of STEM educational quality.
• Inter-rater reliability requires reviewer training, calibration exercises, double scoring, and periodic drift analysis.
• AI reliability requires benchmark testing of model outputs, prompt and model version tracking, hallucination monitoring, and systematic error review.
• Fairness requires attention to resource differences, cultural context, and learner populations, and vigilance against a system that rewards polished documentation over actual educational value.
• Consequential validity requires monitoring how scores and credentials are used in marketing, procurement, and adoption, including the identification of overclaiming and misuse.

A living standard complicates validation rather than simplifying it. Where distributed governance revises criteria or weights, the affected validity evidence must be revisited rather than assumed to carry forward, and each revision must be dated so that historical scores remain interpretable.

Governance and Accountability

An AI-augmented scoring system requires governance at three levels.

Technical governance covers model selection, prompt management, data privacy, cybersecurity, version control, audit logs, confidence thresholds, and human review requirements. AI-generated summaries must trace to source documents, and unverified AI statements must not reach a final report. The Govern, Map, Measure, and Manage functions of the NIST framework provide a workable structure, and the requirement that the system continue to function without AI is itself a technical governance obligation rather than an aspiration.

Professional governance covers reviewer qualifications, conflict-of-interest disclosure, calibration, appeal procedures, rubric revision cycles, and documentation standards. The JCSEE Program Evaluation Standards address evaluator credibility, explicit reasoning, and evaluation documentation directly, and translate cleanly into this context.

Institutional governance covers credential thresholds, brand integrity, registry rules, renewal cycles, revocation processes, and the separation of commercial interests from evaluative judgment. This last point carries particular weight when a single organization both reviews artifacts and licenses the resulting trustmark, and it is the specific problem that distributed standard-setting is designed to address. Where the criteria are set by a verified network rather than by the issuing body, the issuer’s commercial interest loses its most direct route into the rubric.

Credentialing as a Downstream Act

Under SSM, the credential is not the score. It is a public-facing signal that a scored artifact met defined criteria under a defined review process, at a defined moment in time.

Placing credentialing last produces several advantages. It protects the integrity of the review, because reviewers score evidence rather than casting a yes-or-no vote. It allows a single evidence architecture to support multiple outputs, so that accreditation, authentication, and reviewed status can rest on common foundations. It permits conditional and developmental findings rather than forcing every artifact into a pass or fail. It makes renewal rational, since a revised product or expanded program produces a revised score. Finally, it settles the role of AI, which supports the assessment but never issues the mark.

This also completes a shift from episodic to continuous operation. A credential issued and then forgotten records a single moment. A credential maintained within an operating system, scored by a documented method, and governed by an evolving standard can be re-examined whenever the artifact changes or the standard moves.

Risks and Limitations

Seven risks deserve explicit management.

1. Automation bias. Reviewers may defer too readily to AI-generated summaries or suggested scores, and must be trained to challenge them.
2. Documentation bias. Well-resourced organizations submit polished evidence packages, while smaller programs with strong practice may document poorly. The method should score both evidence quality and observed quality, and should state clearly where claims remain unproven.
3. Rubric rigidity. Any scoring methodology can calcify into bureaucracy and penalize innovation or nontraditional learning models. Distributed revision mitigates this risk but does not eliminate it.
4. Governance volatility. The opposite failure is equally real. If criteria or weights move too readily, scores lose comparability across time. Change thresholds, participation minimums, and dated versioning are the safeguards against it.
5. Model drift. AI tools change over time, and a model used in 2026 will not behave identically to one used in 2027. Model versions must be documented and workflows periodically revalidated.
6. Credential misuse. A mark indicates that criteria were met at the time of review, not that the artifact is superior to its competitors. Public-facing language must be governed accordingly.
7. Conflict of interest. Where applicants pay for review, the organization must separate commercial intake from evaluative judgment, publish its standards, and protect reviewer independence.

Research Agenda

A mature implementation should be studied empirically. Priority questions include the following:

• How consistently do reviewers agree across different artifact types?
• What effect does AI assistance have on both the speed and the quality of review?
• Does AI-assisted evidence mapping measurably improve scoring consistency?
• How far do scores diverge between AI-assisted and fully manual review of the same artifact?
• Do SSM scores correlate with independently measured learning outcomes?
• How do applicants use scoring feedback to improve their programs, products, or content?
• Is scoring fair across organization size, geography, learner population, and resource level?
• Does distributed revision of criteria improve or degrade measurement quality over successive cycles?
• How does the public actually interpret credentials derived from the method?

Answering these questions would move SSM from an internal quality-assurance process toward a genuine contribution to STEM education measurement and evaluation.

Final Thoughts

The future of STEM evaluation should be neither fully automated nor fully manual. The defensible middle path uses AI for scale, pattern recognition, and consistency support while preserving human authority over meaning, context, ethics, and final judgment.

Sociotechnical Scoring Methodology supplies that path at the evaluation layer, and it depends on the two layers around it. Without an operating system, scoring has no consistent classification to work from. Without distributed governance, the criteria being scored are merely one organization’s preferences. Without a disciplined scoring method, the standards that governance produces are applied by impression.

The credential is a downstream signal rather than the core intellectual act. The substantive work happens upstream: defining what quality means, gathering evidence, scoring dimensions, documenting reasoning, adjudicating uncertainty, and reporting findings honestly.

STEM quality does not reside solely in an artifact, a curriculum, a technology, or a brand claim. It emerges through the relationship among design, use, learner experience, evidence, context, and judgment. That is precisely why the methodology must be sociotechnical.

Closing Statement

This paper completes a sequence of three. CLEO: The World’s First STEM Trustmark Operating System introduces the operating system. Dynamic Standards Consensus: A Living Standards Model for STEM Education introduces the governance layer.

Andrew B. Raupp is the Founder / Executive Director @stemdotorg. Resolutely preserving the rights and freedoms of the STEM education community through sound policy & practice…