4.1 Introduction and topics

4.1.1 Introduction

This clause describes key concepts in this part of ISO/IEC 19776. This includes how the X3D constructs defined in ISO/IEC 19775-1 are encoded as a binary file.

4.1.2 Topics

Table 4.1 lists the topics is this clause.

4.1.3 Conventions used

The following conventions are used throughout this part of ISO/IEC 19776.

Italics are used for event and field names and are also used when new terms are introduced and equation variables are referenced.

A fixed-space font is used for URL addresses and source code examples. ISO/IEC 19776 Classic VRML encoding (see ISO/IEC 19776-2) examples appear in bold, fixed-space font.

Node type names are appropriately capitalized (e.g., “The Billboard node is a grouping node...”). However, the concept of the node is often referred to in lower case in order to refer to the semantics of the node, not the node itself (e.g., “To rotate the billboard...”).

The form "0xhh" expresses a byte as a hexadecimal number representing the bit configuration for that byte.

Throughout this part of ISO/IEC 19776, references are denoted using the “x.[ABCD]” notation, where "x" denotes which clause or annex the reference is described in and “[ABCD]” is an abbreviation of the reference title. An exception to this convention is that International Standards are referenced by the number of the standard.

EXAMPLE 2.[ABCD] refers to a reference described in Clause 2 and C.[ABCD] refers to a reference described in Annex C.

4.2 Overview

4.2.1 Introduction

This X3D encoding provides a compact transmission format that minimizes delivery size and maximizes parsing speed. Privacy and ownership protections are optionally provided using XML Security mechanisms. Size, parsing and security capabilities are typically superior to that provided by simple gzip compression alone.

There is an essential one-to-one match between the nodes and fields of the abstract X3D scene graph and corresponding constructs in the binary encoding. This section describes the design patterns that govern the correspondence.

The following compression steps are performed as part of the X3D compressed binary encoding. The actual processing order and methods of accomplishment may vary between applications. An implementation may skip any step so long as the resultant binary file is the same as if these steps were followed.

• Conversion of X3D scene into .x3d XML encoding, either as a file or alternatively as an XML-based memory representation
• Optional domain-specific compression of nodes and their children.
• Normalizing XML structures by applying X3D canonicalization as an alternate formulation for XML canonicalization (see 2.[XML-Canonicalization]).
• Optional encryption and/or digital authentication of scene fragments using XML encryption (see 2.[XML-Encryption]) and XML signature (see 2.[XML-Signature]).
• XML-based compression using Fast InfoSet algorithms as specified in ISO/IEC 24824-1.
• Optional gzip compression of entire compressed scene as specified in 2.[ZLIB].

Figure 4.1 — Data flow/production chain

A registry of node compression methods is maintained under ISO control. A file conforming to this part of ISO/IEC 19776 includes only compressors that have been standardized or have approved for inclusion in the registry. See Annex B Node comporessors for more information on registered node compressors.

ISO/IEC 24824-1 specifies the technique used to encode a binary stream representing an X3D scene graph. This technique has the name of Fast InfoSet. Conceptually, a slightly modified X3D XML-encoded document is fed to a Fast InfoSet processor to write the binary stream. Implementations are not required to follow this conceptual model.

4.2.2 The structure of an X3D file

An X3D file is structured as specified in ISO/IEC 19775-1.

4.2.3 X3D canonical form

Conceptually, the X3D scene input to the Fast InfoSet encoder is an XML-encoded document with certain restrictions. X3D canonical form eliminates file ambiguities that have no impact on the 3D content but which otherwise would negatively impact security issues, compression or parsing performance.

The following X3D canonicalization restrictions are applied to an X3D scene (or scene fragment) prior to encryption, signature or compression:

1. Whitespace rules:
   • All whitespace characters are converted to a normalized (single-occurrence) blank character.
   • Whitespace includes comma separators between all MF-type array values, including MFString arrays.
   • All literal characters within an SFString value are retained verbatim.
   • All literal characters within MFString array values are retained verbatim.
2. Quotation marks and apostrophes:
   • Individual MFString array values are bounded by "double quote" characters, each separated by a single space.
   • The overall MFString attribute is contained within 'apostrophe' characters.

   EXAMPLE 1  <NavigationInfo type='"WALK" "EXAMINE" "ANY"'/>

   • Apostrophe characters within an MFString value are replaced by the ′ character entity.
   • Quotation characters within an MFString value are replaced by the " character entity (and escaped by a leading backslash \ character).

3. Default or substitute DTD as specified in Annex A of ISO/IEC 19776-1 is included following the <?xml version="1.0" encoding="UTF-8"?> header in canonical form.

   NOTE 1  The default DTD is not included in the final compressed binary encoding, only substitute DTD values are compressed.

4. Default or substitute X3D schema (see Annex B of ISO/IEC 19776-1)attributes are included in the root <X3D> element.

   NOTE 2  The default X3D Schema attributes are not included in the final compressed binary encoding, only substitute X3D schema attributes are compressed.

5. Floating point and double precision.
   • Use of scientific notation is optional.
   • Numeric values using scientific notation use a lower-case 'e' character.
   • Leading plus signs are omitted, both for mantissa and exponent.

   EXAMPLE 2  2.004e3 (equal to 2004.0).

   • Floating point values do not have to be converted into scientific notation.
   • Decompressed values will be equal but will not necessarily match the form of original values.
6. Attributes with empty values are eliminated. Attributes with default values are eliminated in order to reduce encoded file size. Default values can be determined from the X3D DTD (see Annex A of ISO/IEC 19776-1) or X3D schema (see Annex B of ISO/IEC 19776-1). This rule supercedes the XML canonicalization rule that all attribute values are provided.
7. Attribute-value pairs for DEF, USE and (non-default) containerField shall appear before other attributes, which then follow in alphabetic order. This ordering typically provides higher parsing performance during subsequent decoding. This step supercedes the XML canonicalization rule that all attribute values are provided in alphabetic order.

8. All MFNode content for a child field shall be provided in a contiguous block with no intermixed containerField usage.

   EXAMPLE 3  The following code exhibits X3D not in this form:

         <Collision>
            <Shape containerField="children" />
            <Shape containerField="proxy" />
            <Shape containerField="children" />
         </Collision>

   The canonical form with proper child-element grouping is:

         <Collision>
            <Shape containerField="proxy" />
            <Shape containerField="children" />
            <Shape containerField="children" />
         </Collision>

9. Comment are always preserved. Since default values for X3D compressed binary encoding are lossless, comments are retained by default.
10. SFImage data is written in hexadecimal form separated by normalized whitespace.
11. SF/MFInt32 values will be converted to decimal form if in hexidecimal form.
12. Empty tag paris are replaced with a singleton that closes.

    EXAMPLE 4  The construct:

    <Group DEF="someDEF" class="someClass"></Group>

    is converted to:

    <Group DEF="someDEF" class="someClass"/>

13. CDATA sections (typically, ECMAcript or shader source code) are not converted into character entities. This step superecedes the XML canonicalization rule that CDATA sections are replaced with their character content.
14. Except where specifically overridden by the preceding rules, apply the rules of XML canonicalization, summarized here:
    1. comments may optionally be included
    2. normalize line feeds
    3. normalize attribute values
    4. resolve character and parsed entity references
    5. sort attributes lexicographically
15. X3D scenes in canonical form shall be well-formed, validated XML. This property is a prerequisite to subsequent XML-based compression techniques.

4.2.4 Header

EXAMPLE 1  <?xml encoding='finf'?>'11100000000000000000000000000000000'

EXAMPLE 2  <?xml version='1.0' encoding='finf' standalone='yes'> '11100000000000000000000000000000000'

Tow additional fields are appended to the X3D tag to denote X3D compressed binary version information. These attributes are:

<X3D
binaryVersion="3.1" serializerVersion="Fast InfoSet 2004" >     </X3D>	SFString SFString	[initializeOnly] [initializeOnly]

4.2.5 Prototype representations

All Prototype-related elements are treated in a manner consistent with the representations of the X3D XML encoding. Thus, Prototype instances are treated as PrototypeInstance constructions in XML.

EXAMPLE

  <ProtoDeclare name="ExampleSensor">
   <ProtoInterface>
     <field accessType="initializeOnly" name="enabled" type="SFBool" value="true"/>
   </ProtoInterface>
   <ProtoBody>
     <TouchSensor description="touch to activate">
      <IS>
       <connect nodeField="enabled" protoField="enabled"/>
      </IS>
     </TouchSensor>
    </ProtoBody>
   </ProtoDeclare>
   <ProtoInstance name="ExampleSensor">
    <fieldValue name="enabled" value="false"/>
   </ProtoInstance>

The following "native-tag" representation is not legal and shall not be used within the compressed binary encoding:

   <ExampleSensor enabled="false"/>

4.3 X3D compressed binary encoding file syntax

4.3.1 Node compressors

4.3.1.1 Process overview

Node compressors are responsible for compressing a node and all its children. Typically, this compressor will have domain-specific knowledge about how to compress a node's data.

EXAMPLE  Using geometric compression techniques to store an IndexedFaceSet

The compressor shall restore the exact same scenegraph structure as was compressed, including any ProtoInstance usage. In addition, no reordering of data is allowed; i.e., vertex values in a Coordinate node point array shall not change ordinal position.

A compressor shall only be assigned to handle a single node and all its children. Other constructs like Prototypes and Routes may not be compressed using Node compressors. Furthermore, nodes that implement the X3DScriptNode or X3DProgrammableShaderObject shall not be assigned a node compressor. A node compressor can be used to compress the child nodes of these node types.

Node compressors shall not be nested. Once a Node compressor is used it shall handle all children nodes. A compressor may leave a node or field unaltered in the compressed stream.

4.3.1.2 Compression

A node compressor transforms a scenegraph fragment using the following steps:

1. Create a MetadataSet named ".x3db" with a default reference of "Web3D". This value may be omitted to further reduce size.
2. Move user specified-metadata (if any) to become the first child of the created MetadataSet. This step is ignored if no user-specified metadata is provided.
3. Add a MetadataInteger entry for the encoding, named "encoding" with an integral value attribute. This must be an encoder specified in Annex B Node compressors.
4. Add a MetadataInteger entry for the resulting compressed data, named "payload".
5. It is recommended that replacement values be URI, pointing to alternative guidance.
6. Remove fields from the original scene graph handled by the encoder. Leave all remaining scene graph structure intact.
7. Add Encoder-specific metadata. Only one metadata node is allowed and it shall be named "Encoder Metadata"

EXAMPLE 1 The following illustrates compressing an IndexedFaceSet

An IndexedFaceSet containing a MetadataString and a Coordinate node are to be compressed using a geometric compression algorithm (encoding 1). The input scene graph looks like this:

<IndexedFaceSet coordIndex="0, 1, 2, 3, 4, 5, -1 ...">
  <MetadataString name="myData" value="myValue"/>
  <Coordinate DEF="COORD" point="1.0 2.0 3.0, 4.0 5.0 6.0, ..."/>
</IndexedFaceSet>

The geometric encoder compresses and encodes the coordIndex and point fields into the payload section. The resulting form sent to Fast Infoset looks like this:

<IndexedFaceSet>
  <MetadataSet name=".x3db" reference="Web3D">
    <MetadataString name="myData" value="myValue"/>
    <MetadataInteger name="encoding" value="1"/>
    <MetadataInteger name="payload" value="..."/>
    <MetadataSet name="Encoder Metadata">
      <MetadataSet name="Binary Statistics">
        <MetadataString name="rate" value="12:1"/>
      </MetadataSet>
    </MetadataSet>
  </MetadataSet>
  <Coordinate DEF="COORD"/>
</IndexedFaceSet>

Then the final form parsed by the X3D binary decoder engine is:

<IndexedFaceSet coordIndex="0, 1, 2, 3, 4, 5, -1 ...">
  <MetadataString name="myData" value="myValue"/>
  <Coordinate DEF="COORD" point="1.0 2.0 3.0, 4.0 5.0 6.0, ..."/>
</IndexedFaceSet>

4.3.1.3 Decompression

1. Move user-specified metadata back to its original position as immediate child of the parent node.
2. Restore the exact original scene graph structure using the payload data and the unprocessed nodes and fields.
3. Remove the MetadataSet named ".x3db" and a reference of "Web3D" along with all its children.

4.3.1.4 Preservation

An optional MetadataString may be specified before the encoding MetadataInteger name using the keyword "preserved". If the corresponding "preserved" keyword is specified, the decompressed form shall retain all ".x3db" MetadataSet data including the payload in addition to the uncompressed values in the original node. The user's metadata is retained but will not be returned to its original location.

EXAMPLE 2 The following example illustrates compressing an IndexedFaceSet with preserved parameters

An IndexedFaceSet containing a MetadataString and a Coordinate node is to be compressed using a geometric-compression algorithm (encoding 1). The input scene graph looks like this:

<IndexedFaceSet coordIndex="...">
  <MetadataSet name=".x3db" reference="Web3D">
    <MetadataString name="myData" value="myValue"/>
    <MetadataInteger name="preserved" value="true"/>
    <MetadataInteger name="encoding" value="1"/>
    <MetadataSet name="Encoder Metadata">
      <MetadataSet name="Binary Params">
        <MetadataString name="lossy" value="true"/>
        <MetadataString name="coordinateBits" value="15"/>
        <MetadataString name="normalBits" value="6"/>
        <MetadataString name="colorBits" value="9"/>
        <MetadataString name="textureBits" value="9"/>
      </MetadataSet>
    </MetadataSet>
  </MetadataSet>
  <Coordinate DEF="COORD" point="..."/>
</IndexedFaceSet>

The compressed X3D canonical form sent to Fast Infoset looks like this:

<IndexedFaceSet>
  <MetadataSet name=".x3db" reference="Web3D">
    <MetadataString name="myData" value="myValue"/>
    <MetadataInteger name="encoding" value="1"/>
    <MetadataInteger name="payload" value="..."/>
    <MetadataSet name="Encoder Metadata">
      <MetadataSet name="Binary Params">
        <MetadataString name="lossy" value="true"/>
        <MetadataString name="coordinateBits" value="15"/>
        <MetadataString name="normalBits" value="6"/>
        <MetadataString name="colorBits" value="9"/>
        <MetadataString name="textureBits" value="9"/>
      </MetadataSet>
    </MetadataSet>
  </MetadataSet>
  <Coordinate DEF="COORD"/>
</IndexedFaceSet>

The final form parsed by the X3D engine is:

<IndexedFaceSet coordIndex="...">
  <MetadataSet name=".x3db" reference="Web3D">
    <MetadataString name="myData" value="myValue"/>
    <MetadataInteger name="encoding" value="1"/>
    <MetadataSet name="Encoder Metadata">
    <MetadataSet name="Binary Params">
      <MetadataString name="lossy" value="true"/>
        <MetadataString name="coordinateBits" value="15"/>
        <MetadataString name="normalBits" value="6"/>
        <MetadataString name="colorBits" value="9"/>
        <MetadataString name="textureBits" value="9"/>
      </MetadataSet>
    </MetadataSet>
  </MetadataSet>
  <Coordinate DEF="COORD" point="..."/>
</IndexedFaceSet>

4.3.1.5 User-specified parameters

An X3D compressed binary processor compressing a node that already contains a MetadataSet with the name ".x3db" and reference "Web3D" shall generate the payload block using the specified encoder and Encoder Metadata parameters. Any unknown or incorrect values may be ignored for compression purposes.

4.3.2 Additional field compression

Further type-specific field compression is specified in 5 Encoding of fields.

4.3.3 XML binary compression using Fast InfoSet

This part of ISO/IEC 19776 uses Fast InfoSet to serialize and compress an X3D document. It uses several techniques that reduce the size of an X3D document and that increase the speed of creating and processing such documents. These techniques are based on the use of vocabulary tables that allow small (typically) integer values (vocabulary table indexes) to be used instead of character strings.

EXAMPLE 1  Character strings that form the names of nodes or fields in an X3D document.

A further optimization uses the encoding algorithm vocabulary table. This table identifies specialized encodings that can be employed for commonly occurring strings, again with a number of built-in algorithms.

EXAMPLE 2  If there is a string which looks like the decimal representation of an integer in the range -32768 to 32767, that string can be encoded by identifying that this vocabulary table is being used, giving the vocabulary table index, and then encoding the integer as a two-octet signed integer. Floating-point numbers and arrays of such numbers can be supported in the same way.

This part of ISO/IEC 19776 defines several unique encoders for X3D that utilize type-aware knowledge to further compress data. Further details are specified in 5 Encoding of fields.

4.4 X3D files and the World Wide Web

4.4.1 gzip compression

An optional final step is gzip compression of the entire compressed scene. This step supports global compression, while the other compression techniques described in this part of ISO/IEC 19776 are localized to individual chunks of data.

NOTE  gzip compression of compressed binary encoded scenes may have a negative impact on parser processing performance during decompression.

4.4.2 File extensions and MIME types

X3D compressed binary encoded files shall use the file extension “.x3db”. The MIME-type associated with that binary compressed X3D encoded file is “model/x3d+binary”.

X3D compressed binary encoded files that have been “gzipped” shall use a file extension of “.x3db.gz”. The MIME type is “model/x3d+binary”. The content-encoding value is “gzip”.

The concept of MIME types is defined in 2.[MIME].