Interactivity

December 16, 2024 · View on GitHub

Contributors

  • ISO/IEC SC29 WG3 (MPEG Systems) - Scene Description Breakout Group
  • Contacts

Status

Based on ISO/IEC DIS 23090-14 2nd Edition

Dependencies

Written against the glTF 2.0 spec.

Overview

Interactivity is supported at the scene level and at the node level through the definition of two extensions MPEG_scene_interactivity and MPEG_node_interactivity. When present, the MPEG_scene_interactivity extension shall be included as extension to the scene object. When present, the MPEG_node_interactivity extension shall be included as extension to a node object. The MPEG_node_interactivity extension is used to complement the interactivity extension defined at the scene level. One particular case is the definition of the parameters for a physics engine. That is, when an MPEG_node_interactivity extension contains a trigger of type TRIGGER_COLLISION without being referenced by a trigger of type TRIGGER_COLLISION at the MPEG_scene_interactivity extension, this node shall not be considered for collision detection and instead only be used by the physics engine.

Semantics

Semantics at scene level

Semantics at node level

Semantics at scene level

The semantic of the MPEG_scene_interactivity extension is based on the definition of trigger, action and behavior objects as shown in Table 1.

Table 1: Semantic of the MPEG_scene_interactivity extension

NameTypeRequiredDefaultDescription
triggersMPEG_scene_interactivity.trigger [1-*]✓ YesContains the definition of all the triggers used in that scene.
actionsMPEG_scene_interactivity.action [1-*]✓ YesContains the definition of all the actions used in that scene.
behaviorsMPEG_scene_interactivity.behavior [1-*]✓ YesContains the definition of all the behaviors used in that scene. A behavior is composed of a pair of (triggers, actions), control parameters of triggers and actions, a priority weight and an optional interrupt action.
recommendedPhysicsHighPrecisionBooleanNofalseDetermines whether the application should enable a more deterministic and precise physic simulation
gravityNumberNo-9.81Determine the gravity for the whole scene. In meter per second square (m.s-2) as defined in the international unit system.
recommendedPhysicsFrameRateNumberNo50Provides the recommended frame rate at which the Physics Engine should operate. In frame per second as defined in the international unit system.
bounceThresholdnumberNo1A contact with a relative velocity below this threshold will not result in a bounce. In meter per second (m.s-1) as defined in the international unit system.

The semantic of a trigger is provided in Table 2.

Table 2: Semantic of a trigger

NameTypeRequiredDefaultDescription
typeenumeration✓ YesOne element of Table 3 that defines the type of the trigger.
if (type == TRIGGER_COLLISION){
nodesarray✓ YesIndices of the nodes in the nodes array to be considered for collision determination. Any detection of collision shall activate the trigger
primitivesarray(Primitive)NoN/AList of primitives used to activate the proximity or collision trigger. Semantics of Primitive are presented in Table 4.
}
if (type == TRIGGER_PROXIMITY){
referenceNodenumberNoN/AIndex in the nodes array, of the node to consider for the proximity evaluation.
In the absence of the referenceNode attribute, the active camera managed by the application shall be used.
distanceLowerLimitnumberNo0Threshold minimum in meters for the proximity calculation, based on the distance between the nodes and the referenceNode.
distanceUpperLimitnumber✓ YesThreshold maximum in meters for the proximity calculation, based on the distance between the nodes and the referenceNode.
nodesarray✓ YesIndices of the nodes in the nodes array to be considered. All the nodes shall have a distance from the origin of their local space to
the referenceNode above the distanceLowerLimit and below the distanceUpperLimit to activate the trigger
primitivesarray(Primitive)NoN/AList of primitives used to activate the proximity or collision trigger. Semantics of Primitive are presented in Table 4.
}
if (type== TRIGGER_USER_INPUT){
userInputDescriptionstring✓ YesDescribes the user body part and gesture related to the input. The format shall follow the OpenXR input path description as defined in [OpenXR] section 6. An example is: “/user/hand/left/grip”.
nodesarrayNoN/AIndices of the nodes in the nodes array to be considered for this user input.
}
if (type== TRIGGER_VISIBILITY){
cameraNodenumber✓ YesIndex to the node containing a camera in the nodes array for which the visibilities are determined.
The visibility trigger shall be evaluated only if the related camera is active.
nodesarray✓ YesIndices of the nodes in the nodes array to be considered. All the nodes shall be visible by the camera to activate the trigger.
}

Table 3: type of trigger

Trigger typeDescription
TRIGGER_COLLISION=0Collision Trigger
TRIGGER_PROXIMITYProximity Trigger
TRIGGER_USER_INPUTUser_Input Trigger
TRIGGER_VISIBILITYVisibility Trigger

The semantics of a trigger primitive are defined in Table 9.

Table 4: Semantics of MPEG_scene_interactive.trigger.primitive properties

NameTypeRequiredDefaultDescription
typeenumerationNoBV_SPHEROIDDescribes the type of primitive used to activate the proximity trigger. The available options are:
BV_CUBOID = 0,
BV_PLANE_REGION = 1, BV_CYLINDER_REGION = 2, BV_CAPSULE_REGION = 3,
BV_SPHEROID = 4
The default is BV_SPHEROID. Semantics are presented in Table 5.
boundarynumberNo0.0Defines the region of intersection within the primitive. if zero, then all area of the primitive activates the trigger. Otherwise, the region of intersection decreases following the normal direction of all sides of the primitive from its centroid. For the capsule primitive, it should be applied over the radius, top, and base attributes.
transformationMatrixarrayNo[1.0,0.0,0.0,0.0,
0.0,1.0,0.0,0.0,
0.0,0.0,1.0,0.0,
0.0,0.0,0.0,1.0]
Floating-point 4x4 matrix that defines the initial orientation, translation, and scale of a primitive. Formatted in column-major order. The primitive shall follow x+ for width, y+ for height, z+ for length . The matrix transformation allows to transform any primitive after initialization.

Table 5: Semantical description of each primitive region

NameTypeRequiredDefaultDescription
if (type == BV_CUBOID) {
widthnumber✓ YesWidth of the box.
heightnumber✓ YesHeight of the box.
lengthnumber✓ YesLength of the box.
centroidVEC3✓ YesCentroid 3D coordinate (x,y,z) of the cube.
}
if (type == BV_PLANE_REGION) {
widthnumber✓ YesWidth of the plane.
heightnumber✓ YesHeight of the plane.
centroidVEC2✓ YesCentroid 2D coordinate (x,y) or (x,z) or (y,z) of the plane.
}
if (type == BV_CYLINDER_REGION) {
radiusnumber✓ YesRadius of the cylinder.
lengthnumber✓ YesLength of the cylinder.
centroidVEC3✓ YesCentroid 3D coordinate (x,y,z) of the cylinder
}
if (type == BV_CAPSULE_REGION) {
radiusnumber✓ YesRadius of the capsule.
baseCentroidVEC3✓ YesCentroid 3D coordinate (x,y,z) of the base semi-sphere of the capsule.
topCentroidVEC3✓ YesCentroid 3D coordinate (x,y,z) of the top semi-sphere of the capsule.
}
if (type == BV_SPHEROID) {
radiusnumber✓ YesRadius of the sphere.
centroidVEC3✓ YesCentre 3D coordinate (x,y,z) of the sphere.
}

The semantic of an action is provided in Table 6.

Table 6: Semantic of action

NameTypeRequiredDefaultDescription
typeenumeration✓ YesOne element of Table 7 that defines the type of the action.
delaynumberNo0.0Duration of delay in second before executing the action.
if (type== ACTION_ACTIVATE){
activationStatusenumeration✓ YesENABLED=0: the node shall be processed by the application
DISABLED =1: the node shall be skipped by the application.
nodesarray✓ YesIndices of the nodes in the nodes array to set the activation status.
}
if (type== ACTION_TRANSFORM){
transform✓ YesA 4x4 transformation matrix to apply to the nodes.
nodesarray✓ YesIndices of the nodes in the nodes array to be transformed.
}
if (type== ACTION_BLOCK){
nodesarray✓ YesIndices of the nodes in the nodes array to lock their related transforms.
}
if (type == ACTION_ANIMATION){
animationnumber✓ YesIndex of the animation in the animations array to be considered.
animationControlenumeration✓ YesOne element of Table 8 that defines the control of the animation.
}
if (type == ACTION_MEDIA){
medianumber✓ YesIndex of the media in the MPEG_media array to be considered.
mediaControlenumeration✓ YesOne element of Table 9 that defines the control of the media.
}
if (type == ACTION_MANIPULATE){
manipulateActionTypeenumeration✓ YesOne element of Table 10 that defines the action manipulate type.
axisarrayNoUp(x,y,z) coordinates of the axis used for rotation and sliding. These coordinates are relative to the local space created by the user input described in the userInputDescription. For example a “/user/hand/left/pose” user input trigger creates a local space attached to the user left hand.
userInputDescriptionstring✓ YesDescribe the user input related to the manipulation action. The format shall follow the OpenXR input path description as defined in [OpenXR] section 6. An example is: “/user/hand/left/aim/pose”.
nodesarray✓ YesIndices of the nodes in the nodes array to be manipulated.
}
if (type == ACTION_SET_MATERIAL){
materialnumber✓ YesIndex of the material in the materials array to apply to the set of nodes.
nodesarray✓ YesIndices of the nodes in the nodes array to set their material.
}
if (type == ACTION_HAPTIC){
hapticActionNodesarray(HapticActionNode)✓ YesList of haptic action nodes as defined in Table 13.
}
if (type == ACTION_SET_AVATAR) {
avatarActionstring✓ YesThe avatarAction is a URN that uniquely identifies the avatar action.
For the MPEG reference Avatar, a set of avatar actions and their respective URNs is defined in Table H.4.3.2 of Annex H.
For example, the URN “urn:mpeg:sd:2023:avatar:actionslist/speech” referenced in Table H.4.3.2 of Annex H sets the optional attributes “microphone” and “media”. Considering only the boolean attribute “microphone”, the nodes in the node array will activate/deactivate their “microphone” ability accordingly when this action is launched.
nodesarray✓ YesIndices of the nodes in the nodes array to launch their avatar actions.
}

Table 7: Type of action

Action typeDescription
ACTION_ACTIVATE=0Set activation status of a node
ACTION_TRANSFORMSet transform to a node
ACTION_BLOCKBlock the transform of a node
ACTION_ANIMATIONSelect and control an animation
ACTION_MEDIASelect and control a media
ACTION_MANIPULATESelect a manipulate action
ACTION_SET_MATERIALSet new material to nodes
ACTION_HAPTICGet haptic feedbacks on a set of nodes
ACTION_SET_AVATARGet avatar related actions

Table 8: Control of animation

Animation ControlDescription
ANIMATION_PLAY=0Play the animation from time 0 or from any other time provided by a control.
ANIMATION_PAUSEPause the animation
ANIMATION_RESUMEResume the animation from the last pause position.
ANIMATION_STOPStop the animation

Table 9: control of media

Media ControlDescription
MEDIA_PLAY=0Play the media from time 0 or from any other time provided by a control.
MEDIA_PAUSEPause the media
MEDIA_RESUMEResume the media from the last pause position.
MEDIA_STOPStop the media

Table 10: Action Manipulate Type

Action Manipulate Typedescription
ACTION_MANIPULATE_FREE=0The nodes follow the user pointing device and its rotation.
ACTION_MANIPULATE_SLIDEThe nodes move linearly along the provided axis by following the user pointing device.
ACTION_MANIPULATE_TRANSLATEThe nodes translate by following the user pointing device.
ACTION_MANIPULATE_ROTATEThe nodes rotate around the provided axis by following the user pointing device.
ACTION_MANIPULATE_SCALEPerforms a central scaling of the nodes by following the user pointing device.

The semantics of a behavior is provided in Table 11.

Table 11: semantic of behavior

NameTypeRequiredDefaultDescription
triggersarray✓ YesIndices of the triggers in the triggers array considered for this behavior
actionsarray✓ YesIndices of the actions in the actions array considered for this behavior.
triggersCombinationControlstring✓ YesSet of logical operations to apply to the triggers
A ‘#’ indicates the trigger index, ‘&’ indicates a logical AND operation, ‘|’ a logical OR operation and ‘’ a NOT operation. Parenthesis are used to group some operations. Such a syntax may give the following string: “#1&#2|(#3&#4)”.
An empty string is understood as a logical OR between all the triggers.
A regex expression (https://json-schema.org/understanding-json-schema/reference/regular_expressions.html) is specified in the JSON schema to validate this string.
triggersActivationControlenumeration✓ YesIndicates when the combination of the triggers shall be activated for launching the actions. One element of Table 12 that defines when the combination of the triggers shall be activated for launching the actions.
actionsControlenumeration✓ YesDefines the way to execute the defined actions.
SEQUENTIAL=0: each defined action is executed sequentially in the order of the actions array.
PARALLEL=1: the defined actions are executed concurrently.
interruptActionnumberNoN/AIndex of the action in the actions array to be executed if the behavior is still on-going and is no more defined in a newly received scene update.
prioritynumberNo0Integer value defining the priority associated to the behavior When several behaviors are in concurrence to affect the same node(s) at the same time, the behavior having the highest priority value shall be processed. The lower priority behavior(s) shall not be processed. In the case of behaviors having the same priority, the application should apply its own criteria.

Table 12:Trigger Activation Control

Trigger Activation ControlDescription
TRIGGER_ACTIVATE_FIRST_ENTER=0Activated when the conditions are first met
TRIGGER_ACTIVATE_EACH_ENTERActivated each time the conditions are first met
TRIGGER_ACTIVATE_ONActivated as long as the conditions are met
TRIGGER_ACTIVATE_FIRST_EXITActivated when the conditions are first no longer met
TRIGGER_ACTIVATE_EACH_EXITActivated each time the conditions are no longer met
TRIGGER_ACTIVATE_OFFActivated as long as the conditions are not met

Table 13: Semantic of HapticActionNode object

PropertyTypeRequiredDefaultDescription
nodeinteger✓ YesIdentifier of the node in the glTF nodes array.
hapticObjectintegerNoN/AIndex to a haptic object in the hapticObjects array of the MPEG_haptic extension.
actionLocationintegerNo0xFFFFFFFFBody part mask specifying where on the body the signal can be rendered.
Possible values are detailed in Table 17.
washoutbooleanNoFalseSpecifies whether the action should trigger a washout (reset to the origin) of the associated devices.
useColliderbooleanNoFalseUsed with a Collision trigger. If True, the rendering engine shall use collision information to estimate the desired location of the haptic feedback on the body. For haptic materials, the presentation engine retrieves the associated haptic texture media and generates haptic feedback based on texture information and collision position. If false, the signal shall be rendered based on the information specified in the Haptic file.
materialHapticModalityarray(enumeration)NoN/AList of haptic material modalities that shall be rendered. Possible values are detailed in Table 16.
hapticActionMediasarray(HapticActionMedia)✓ YesList of Haptic Action Media.

Table 14: Semantic of the HapticActionMedia object

PropertyTypeRequiredDefaultDescription
mediaIndexinteger✓ YesIndex in the accessors array of the associated haptic data.
perceptionIndicesarray(integer)✓ YesIndices of the perceptions of the media that shall be rendered. If the list if empty all perceptions shall be rendered
hapticModalityarray(enumeration)NoN/AList of haptic modalities that can be rendered. Possible values are described in Table 15.
hapticControlenumerationNoHAPTIC_PLAYOne element of Table 49 that defines the control of the haptic rendering.
loopbooleanNoFalseSpecifies if the haptic rendering of the data should be continuously looping.

Table 15: List of haptic modalities

Pressure = 0
Acceleration
Velocity
Position
Temperature
Vibrotactile
Water
Wind
Force
Electrotactile
Vibrotactile Texture
Stiffness
Friction
Other

Table 16: List of haptic Material Modalities

Stiffness = 0
Friction
Vibrotactile Texture
Temperature
Vibration
Custom

Table 17: Body part masks

NameBody_part_mask (binary)HexadecimalDecimal
0Unspecified000000000000000000000000000000000x000000000
1Head Face000000000000000000000000000000010x000000011
2Head Back/Neck/Ears000000000000000000000000000000100x000000022
3Mouth Bag000000000000000000000000000001000x000000044
4Lower Jaw000000000000000000000000000010000x000000088
5Upper Jaw000000000000000000000000000100000x0000001016
6Eye Left000000000000000000000000001000000x0000002032
7Eye Right000000000000000000000000010000000x0000004064
8Chest Front000000000000000000000000100000000x00000080128
9Chest Back000000000000000000000001000000000x00000100256
10Upper Arm Left000000000000000000000010000000000x00000200512
11Lower Arm Left000000000000000000000100000000000x000004001 024
12Hand Left000000000000000000001000000000000x000008002 048
13Upper Arm Right000000000000000000010000000000000x000010004 096
14Lower Arm Right000000000000000000100000000000000x000020008 192
15Hand Right000000000000000001000000000000000x0000400016 384
16Abdomen Front000000000000000010000000000000000x0000800032 768
17Abdomen Back000000000000000100000000000000000x0001000065 536
18Upper Leg Left000000000000001000000000000000000x00020000131 072
19Lower Leg Left000000000000010000000000000000000x00040000262 144
20Foot Left000000000000100000000000000000000x00080000524 288
21Upper Leg Right000000000001000000000000000000000x001000001 048 576
22Lower Leg Right000000000010000000000000000000000x002000002 097 152
23Foot Right000000000100000000000000000000000x004000004 194 304
24-32Reserved00000000010000000000000000000000 - 100000000000000000000000000000000x00800000 -0x800000008 388 608 - 2 147 483 648

Table 18: Examples of body part combinations

NameBody_part_mask (binary)HexadecimalDecimal
Right arm000000000000000001110000000000000x0000700028 672
Left arm000000000000000000001110000000000x00000E003 584
Right leg000000000111000000000000000000000x007000007 340 032
Left leg000000000000111000000000000000000x000E0000917 504
Upper body000000000000000111111111111111110x0001FFFF131 071
Lower body000000000111111000000000000000000x007E00008 257 536
Full body111111111111111111111111111111110xFFFFFFFF4 294 967 295

Table 19: List of Haptic controls

Haptic ControlDescription
HAPTIC_PLAY = 0Start the rendering of the haptic data from time 0 or from any other time provided by a control
HAPTIC_PAUSEPause the rendering of the haptic data
HAPTIC_RESUMEResume the rendering of the haptic data from the last pause position.
HAPTIC_STOPStop the rendering of the haptic data

Semantics at node level

In complement to the interactivity objects defined in the glTF scene-level extension, some additional data could be provided at the level of the affected glTF nodes to specialize the trigger activation. The semantic of the MPEG_node_interactivity extension is shown in Table 50.

Table 20: Semantic of the MPEG_node_interactivity extension

NameTypeRequiredDefaultDescription
triggersarray✓ YesArray of node triggers (as defined in Table 21). Only distinct types are allowed.
The minimum size of this array is 1, and the maximum size is size of trigger types as defined in this specification.

Table 21: Semantic of the MPEG_node_interactivity.trigger extension

NameTypeRequiredDefaultDescription
typeenumeration✓ YesOne element of Table 3 that defines the type of the trigger.
if (type == TRIGGER_COLLISION){
colliderinteger✓ Yesthe index of the mesh element that provides the collider geometry for the current node.
The collider mesh may reference a material.
isStaticboolean✓ YesIf True, the collider is defined as a static collider.
usePhysicsboolean✓ YesIndicates if the object shall be considered by the physics simulation.
if (usePhysics) {
needPreciseCollisionDetectionBooleanNofalseIf true, the physics engine should handle the collision detection more accurately by increasing the detection rate for this node.
linearDampingNumberNo0A non-negative value, in second-1 (s-1), as defined in the international unit system. It defines the linear drag coefficient which corresponds to the rate of decrease of the linear velocity over time.
It is used to compute a new velocity value V(t) at each simulation step (dt):
V(t+dt) = V(t)(1-linearDampingdt), the velocity being clamped to 0.
angularDampingnumberNo0A non-negative value, in second-1 (s-1), as defined in the international unit system. It defines the angular drag coefficient which corresponds to the rate of decrease of the angular velocity over time.
It is used to compute a new velocity value V(t) at each simulation step (dt):
V(t+dt) = V(t)(1-angularDampingdt), the velocity being clamped to 0.
useGravityboolean✓ YesIndicates if the gravity affects the object
massnumber✓ YesMass of the object in kilogram as defined in the international unit system..
restitutionnumber✓ YesProvides the ratio of the final to initial relative velocity between two objects after they collide.
staticFrictionnumber✓ YesUnitless friction coefficient as defined in the Coulomb friction model. Friction is the quantity which prevents surfaces from sliding off each other. StaticFriction is used when the object is lying still. It will prevent the object from starting to move.
dynamicFrictionnumber✓ YesUnitless friction coefficient as defined in the Coulomb friction model. When a large enough force is applied to the object, the dynamicFriction is used, and will attempt to slow down the object while in contact with another.
}
primitivesarray(Primitive)NoN/AList of primitives used to activate the proximity or collision trigger. Semantics of Primitive are presented in Table 9.
}
if (type == TRIGGER_PROXIMITY){
allowOcclusionboolean✓ YesIndicates if occlusion by other nodes should be considered
upperDistanceWeightnumberNo1The weight applied
to the distanceUpperLimit parameter defined at scene level
lowerDistanceWeightnumberNo1The weight applied
to the distanceLowerLimit parameter defined at scene level
primitivesarray(Primitive)NoN/AList of primitives used to activate the proximity or collision trigger. Semantics of Primitive are presented in Table 9.
}
if (type ==TRIGGER_USER_INPUT){
userInputParametersarray✓ YesProvides additional information related to the user inputs (eg “max speed = 0.5”)
}
if (type== TRIGGER_VISIBILITY){
allowsPartialOcclusionboolean✓ YesThe visibility computation shall take into account both the occultation by other node(s) and the camera frustrum. If the allowsPartialOcclusion Boolean is TRUE, then a partial visibility of this node activates the trigger.
If the allowsPartialOcclusion Boolean is FALSE, then this node shall be fully in the camera frustrum and not be occluded by any other node(s) except the nodes listed in the nodes array to activate the trigger.
nodesarrayNoN/ASet of nodes that shall not be considered for the visibility computation, when the allowsPartialOcclusion is FALSE.
meshnumberNoN/AIndex of the mesh in the scene meshes array that will be used to compute visibility.
}

Processing model

When a scene description file becomes available, the Presentation Engine

  • parses the related glTF file
  • creates each behavior composed of triggers and actions described at the interactivity scene level extension
  • specializes the trigger for each affected node with the additional data provided at the interactivity node level extension if present At runtime, the presentation Engine iterates on each behavior and performs the following tasks:
  • checks the activation status of each trigger of this behavior by following the procedure detailed in the figure below,
  • checks the logical combination of these trigger status,
  • if this status satisfies the triggersActivationControl value, launches the corresponding actions.
processing model for the activation of a single trigger.

When several behaviors are in concurrence to affect the same node(s) at the same time, the behavior having the highest priority value shall be processed. The lower priority behavior(s) shall not be processed. In the case of behaviors having the same priority, the application should apply its own criteria. When a new scene description update is received, the application follows the procedure detailed in the figure below. An on-going behavior corresponds to a behavior:

  • having its triggers status verifying the triggersActivationControl value for that frame,
  • or having previously launched a play action related to a media/animation,
  • or having previously launched an action with a delay not yet expired. To check if the behavior is still defined, the application should check if the scene description update leads to:
  • removal of this behavior in the behaviors array.
  • modification of any parameter of this behavior (e.g. by adding a new trigger, by changing the trigger activation control,…).
  • removal of any of the referenced nodes by the action. The application shall process the interrupt action only if the timing requirement of the scene description update is met.
Processing model when a new scene description update is received.

If the scene description document contains a description of physics properties based on another physics model, then that physics model shall take precedence in the processing of the scene.

Otherwise, the application shall handle a physics simulation if the usePhysics Boolean is TRUE on any of the collision trigger extensions defined at the node level. When a collision occurs between two nodes, the application should calculate the combination of the restitution, static friction and dynamic friction values based on the values provided by the collision trigger extension of the two nodes.

Schema

Known Implementations

Resources

  • ISO/IEC FDIS 23090-14, Information technology — Coded representation of immersive media — Part 14: Scene Description
  • ISO/IEC WD 23090-24, Information technology — Coded representation of immersive media — Part 24: Conformance and Reference Software for Scene Description for MPEG Media

License

Copyright ISO/IEC 2022

The use of the "MPEG scene description extensions" is subject to the license as accessible here: https://standards.iso.org/ and is subject to the IPR policy as accessible here: https://www.iso.org/iso-standards-and-patents.html.