README.md

This package contains of type definitions for processing assessment data. Additionally, you find TypeScript interfaces and classes, so these data structures can be used in a coherent way.

• Iqb specifications
• Ausführliche Darstellungen zum Kodieren

Installation

npm i @iqb/responses

.NET 10

The repository also contains the native .NET package Iqb.Responses. It uses the same camel-case JSON contracts and does not require Node.js at runtime.

dotnet add package Iqb.Responses --version 5.2.2-preview.1

using System.Text.Json;
using Iqb.Responses;

var scheme = CodingScheme.Parse(File.ReadAllText("coding-scheme.json"));
var responses = JsonSerializer.Deserialize<List<Response>>(
    File.ReadAllText("responses.json"),
    IqbJson.Options) ?? [];

var coded = CodingSchemeFactory.Code(responses, scheme.VariableCodings);

The native SOLVER supports numeric literals, null, booleans, parentheses, arithmetic and comparison operators, ternary expressions, IQB placeholders, fragment access, and the documented default/ERROR/INC policies. Its scalar mathjs subset also includes:

• Constants: pi/PI, e/E, tau, phi, Infinity, and NaN.
• Numeric functions: abs, sqrt, cbrt, ceil, floor, fix, round, sign, min, max, gcd, pow, mod, square, cube, and nthRoot.
• Exponential and logarithmic functions: exp, log, log10, and log2.
• Trigonometric functions: sin, cos, tan, asin, acos, atan, atan2, and hypot.

Matrices, units, assignments, user-defined functions and other unrestricted mathjs syntax are not supported. Unknown syntax, complex values and non-finite final results produce DERIVE_ERROR. Transcendental functions can differ from V8 by one final IEEE-754 bit because .NET and JavaScript use different platform math implementations. Direct, finite SOLVER results that use exp, logarithmic, trigonometric or hypot functions may differ by at most one IEEE-754 ULP in differential tests. Status, error and downstream coding results are still compared exactly. gcd requires at least two finite integer arguments and is always compared exactly.

Number-to-string conversion and Unicode lowercasing follow JavaScript semantics in both runtimes. MATCH_REGEX deliberately uses a portable ASCII subset: character classes, groups, alternation and ordinary quantifiers are supported; lookaround, backreferences, Unicode property classes, Unicode/hex escapes and non-ASCII pattern text are rejected during validation and produce a controlled coding error if validation is bypassed.

Differential parity testing

npm run test:differential compares all checked-in golden vectors and 20,000 reproducible generated cases with the native .NET implementation. A failure is automatically minimized and written to artifacts/differential/. Replay it with npm run test:differential:generated -- --case <minimized.request.json>.

Before a stable release, npm run test:differential:stress runs 100,000 cases. Locale, Unicode, regex-dialect and SOLVER-language boundaries are tracked separately in test/differential/boundaries/manifest.json. Large-number formatting, Unicode lowercasing and rejection of non-portable regular expressions are blocking parity cases.

Documentation

• Quick Reference Guide - Common operations, examples, and troubleshooting
• Coding Process Documentation - Detailed architecture and logic flow with diagrams
• Iqb specifications
• Ausführliche Darstellungen zum Kodieren

Public API (Exports)

The package exports (among others):

• Coding responses
  • CodingFactory (also exported as CodingEngine / ResponseCoder)
  • CodingSchemeFactory (also exported as CodingSchemeEngine / SchemeCoder)
• Rendering / formatting
  • ToTextFactory (also exported as CodingTextRenderer / CodingFormatter)
  • CodingSchemeTextFactory (also exported as CodingSchemeTextRenderer)
• Utilities
  • VariableList

Concepts

Core data model

• Response (@iqbspecs/response)
  • Identified by id (string)
  • Holds value (single value or array) and a status
  • May have code, score and (optionally) subform
• VariableCodingData (@iqbspecs/coding-scheme)
  • Defines how one variable is obtained and/or coded
  • Important fields you will see often:
    • id and alias
    • sourceType + deriveSources (for derived variables)
    • codes + processing + fragmenting (for rule-based coding)

id vs alias

In many consuming applications variables are addressed by alias, while internally the coding scheme also has an id.

• When you call CodingSchemeFactory.code(), responses are temporarily mapped from alias to id, processed, and mapped back.
• When building tooling around this package, treat alias as the “external name” and id as the “stable internal key”.

Coding pipeline (high level)

CodingSchemeFactory.code() processes a list of responses together with a coding scheme and runs the following steps:

1. Group by subform
   • Responses with a subform are coded per subform group.
2. Normalize statuses
   • Some statuses are normalized to VALUE_CHANGED depending on variable/source parameters.
3. Resolve derived/base conflicts
   • Base responses that are shadowed by derived variables can be removed before deriving/coding.
4. Plan derivations
   • A dependency graph is computed so derived variables are processed in the right order.
5. Ensure responses exist
   • If a variable is present in the coding scheme but missing from input responses, placeholder responses can be created.
6. Derive values (if configured)
   • Derived variables are computed from their source responses.
7. Code values
   • Rule-based coding is applied to responses with status === VALUE_CHANGED.

Status and error handling

• Status strings are defined by the @iqbspecs/response / @iqbspecs/coding-scheme specs and are used throughout this package.
• Typical flow for base variables is:
  • VALUE_CHANGED -> CODING_COMPLETE / CODING_INCOMPLETE / NO_CODING
• For derived variables:
  • derivation may set DERIVE_ERROR (and value is cleared)
• If internal processing throws unexpectedly:
  • CODING_ERROR / DERIVE_ERROR is set and the optional onError callback is called.

Debugging & internal extension points

If you are working on the internals:

• Rule evaluation is implemented in src/rule-engine/*.
• Derivations are implemented in src/derive/*.
• Alias mapping lives in src/mapping/alias-mapper.ts.
• Subform grouping lives in src/subform/*.

Quick start: code all responses of a unit

Use CodingSchemeFactory.code() to apply derivations (if configured in the coding scheme) and then apply coding rules.

import type { Response } from '@iqbspecs/response/response.interface';
import type { VariableCodingData } from '@iqbspecs/coding-scheme';
import { CodingSchemeFactory } from '@iqb/responses';

const responses: Response[] = [
  { id: 'A1', value: 'foo', status: 'VALUE_CHANGED' },
  { id: 'A2', value: 2, status: 'VALUE_CHANGED' }
];

const variableCodings: VariableCodingData[] = /* from a coding-scheme JSON */ [];

const coded = CodingSchemeFactory.code(responses, variableCodings, {
  onError: (err) => {
    // optional hook: called when derive/coding encounters unexpected errors
    console.error(err);
  }
});

Notes

• Aliases vs ids: CodingSchemeFactory.code() maps response ids between alias and id internally and returns the final responses mapped back.
• Subforms: responses may include a subform property; coding is performed per subform group.
• Missing responses: for variables that are present in the coding scheme but missing from responses, placeholder responses may be created (except for sourceType: 'BASE_NO_VALUE').

Code a single response

If you already have a VariableCodingData for a base variable and want to code exactly one Response, use CodingFactory.code().

import type { Response } from '@iqbspecs/response/response.interface';
import type { VariableCodingData } from '@iqbspecs/coding-scheme';
import { CodingFactory } from '@iqb/responses';

const response: Response = { id: 'A1', value: 'foo', status: 'VALUE_CHANGED' };
const coding: VariableCodingData = /* coding for variable A1 */ {} as VariableCodingData;

const codedResponse = CodingFactory.code(response, coding);

Validate a coding scheme against base variables

import type { VariableInfo } from '@iqbspecs/variable-info/variable-info.interface';
import type { VariableCodingData } from '@iqbspecs/coding-scheme';
import { CodingSchemeFactory } from '@iqb/responses';

const baseVariables: VariableInfo[] = /* list of base variables */ [];
const codingScheme: VariableCodingData[] = /* coding scheme variables */ [];

const problems = CodingSchemeFactory.validate(baseVariables, codingScheme);
// each problem is a { type, breaking, variableId, variableLabel, ... }

Derivation dependency tree

To inspect derivation order / dependencies:

import type { VariableCodingData } from '@iqbspecs/coding-scheme';
import { CodingSchemeFactory } from '@iqb/responses';

const variableCodings: VariableCodingData[] = [];
const tree = CodingSchemeFactory.getVariableDependencyTree(variableCodings);

Get required base variables for a set of target variables

This is useful if you want to load only the required base responses needed to derive/code a set of variables.

import type { VariableCodingData } from '@iqbspecs/coding-scheme';
import { CodingSchemeFactory } from '@iqb/responses';

const targetVarAliases = ['V1', 'V2'];
const variableCodings: VariableCodingData[] = [];

const requiredBaseVarAliases = CodingSchemeFactory.getBaseVarsList(targetVarAliases, variableCodings);

Render coding information as text

import type { VariableInfo } from '@iqbspecs/variable-info/variable-info.interface';
import { ToTextFactory } from '@iqb/responses';

const varInfo: VariableInfo = /* ... */ {} as VariableInfo;
const lines = ToTextFactory.varInfoAsText(varInfo);

Status values

Status strings used by this package include (among others) VALUE_CHANGED, CODING_COMPLETE, CODING_INCOMPLETE, CODING_ERROR, DERIVE_ERROR, INVALID, NO_CODING, UNSET, ...

Versionsänderungen npm-package @iqb/responses

5.2.2

• Ein von einem SOLVER-Ausdruck explizit zurückgegebenes null wird als VALUE_CHANGED weitergegeben und kann über IS_NULL oder RESIDUAL_AUTO kodiert werden.
• NaN, positive und negative Unendlichkeit sowie andere nichtnumerische SOLVER-Ergebnisse führen weiterhin zu DERIVE_ERROR.

5.2.1

• Ein eindeutiger öffentlicher Alias darf der technischen ID einer anderen Variable entsprechen, sofern deren öffentlicher Bezeichner abweicht. Responses bleiben extern aliasbasiert, während deriveSources weiterhin technische IDs referenzieren.
• Doppelte IDs, doppelte Aliasse und echte öffentliche Kollisionen bleiben ungültig.
• Unerwartete sourceType-Werte führen kontrolliert zu DERIVE_ERROR, statt geerbte Object-Properties als Handler aufzulösen.

5.2

• SOLVER-Ausdrücke können jetzt Fragmente aus Quellvariablen referenzieren, z. B. ${VAR[0]}.
• SOLVER-Placeholders unterstützen Policies für leere/fehlende und nicht-numerische Werte, z. B. ${VAR:0}, ${VAR:INC} oder ${VAR:0:INC}.
• Die Validierung des Coding-Schemes prüft weitere Regelparameter, Numeric-Ranges, Fragment-Regeln, valueArrayPos und Typkompatibilität.
• Abgeleitete Variablen können gezielt den Alias ihrer Base-Source überschreiben; die Base-Response wird danach korrekt entfernt.
• NUMERIC_MATCH wertet null und leere Strings nicht mehr als numerische 0.

5.1

• @iqbspecs/coding-scheme 3.4.0
• Implementiert die neuen CodingProblem Typen:
  • RULE_REGEX_INVALID,
  • RULE_PARAMETER_INVALID,
  • RULE_NUMERIC_RANGE_INVALID,
  • RULESET_VALUE_ARRAY_POS_INVALID

5.0

• umfangreiches Refactoring/Modularisierung (u. a. Ableitungen, Rule-Engine, Dependency-Planung)
• verbesserte Validierung des Coding-Schemes (z. B. doppelte IDs/Aliase, ungültige Quellen, RULE_PARAMETER_COUNT_MISMATCH)
• robustere Fehlerbehandlung (z. B. ungültige RegEx, ungültige Solver-Expressions)
• Performance-Verbesserungen durch Map-basierte Lookups
• deutlich erweiterte Unit-Tests sowie CI/Dependabot-Workflows

4.0

Übernimmt folgende Spezifikationen:

• @iqbspecs/coding-scheme
• @iqbspecs/response
• @iqbspecs/variable-info

3.7

• TAKE_NOT_REACHED_AS_VALUE_CHANGED neuer Parameter für die Verarbeitung von Quell-Variablen

3.6

• NUMERIC_FULL_RANGE Regel >= Wert <= hinzugefügt

3.5

• Antworten in Unterformularen werden kodiert
• Setze RuleMethodParameterCount -1 for NUMERIC_MATCH Regel

3.4

• BASE_NO_VALUE source type wird nicht zu den Antworten hinzugefügt
• Status INTENDED_INCOMPLETE und code type INTENDED_INCOMPLETE hinzugefügt

3.3

• Ableitungsregeln angepasst

3.2

• manuell kodierte abgeleitete und eingespielte Variablen werden nicht versucht abzuleiten
• Missing Regeln werden bei der Ableitung beachtet
• Aliase aus dem CodingScheme werden berücksichtigt

3.1

• neuer Wert für Property valueArrayPos: LENGTH; Werte Array muss angebene Länge haben
• neuer Wert für Property valueArrayPos: ANY_OPEN; dann wird - im Gegensatz zu ANY - erlaubt, dass Werte im Array sind, für die der Regelsatz nicht zutrifft
• Funktionalität für valueArrayPos - ANY_OPEN und ANY hinzugefügt bzw. korrigiert
• IGNORE_CASE führt jetzt zum i-Flag bei RegEx
• ein Werte-Array mit Länge 0 wird jetzt auch als 'leer' klassifiziert

3.0

• neue Property version; daher ist das gesamte Coding Scheme nicht mehr ein Array, sondern ein Objekt!
• neue Werte für code type: RESIDUAL, RESIDUAL_AUTO; ersetzt ELSE-Regel
• ELSE-Regel entfernt
• codingModel ist jetzt beschränkt auf NONE, RULES_ONLY und MANUAL_ONLY

2.2

• Neuer Wert für codeModel: RULES_ONLY - Ausblenden der Controls für die manuelle Kodierung
• Variable erhält neben der ID einen alias
• Code erhält einen type, um die Dokumentation zu erleichtern und die UI zu vereinfachen. Werte UNSET, FULL_CREDIT, PARTIAL_CREDIT, NO_CREDIT - label wurde hierfür immer missbraucht

2.1

• Neue Funktion getVariableDependencyTree() im Kodierschema, um den Graph der Ableitungen abzubilden

2.0

• Umsetzung der Änderungen Datenstruktur coding-scheme v2.0:
  • Neue Ableitungsmethoden UNIQUE_VALUES, SOLVER
  • neue Parameter 'sourceParameters' mit den Eigenschaften 'solverExpression' und 'processing' (mögliche Werte TO_LOWER_CASE, TO_NUMBER, REMOVE_ALL_SPACES, REMOVE_DISPENSABLE_SPACES, TAKE_DISPLAYED_AS_VALUE_CHANGED, SORT, TAKE_EMPTY_AS_VALID)
  • processing REMOVE_WHITE_SPACES entfernt; stattdessen IGNORE_ALL_SPACES, IGNORE_DISPENSABLE_SPACES, SORT_ARRAY
• Umbau der Tests auf Jest (für coding-scheme und die Validierung der JSON-Schema)
• Ersetzen createCodingVariableFromVarInfo mit createCodingVariable in CodingFactory