BinPack

A space-efficient binary JSON serialization format based on JSON Schema. More...

Topics
	Integer Encodings
	Number Encodings
	Any Encodings
	String Encodings
	Array Encodings
	Object Encodings

Classes
class	sourcemeta::blaze::BinPackDecoder
class	sourcemeta::blaze::BinPackEncoder
class	sourcemeta::blaze::BinPackEncodingError

Typedefs
using	sourcemeta::blaze::BinPackEncoding

Functions
auto	sourcemeta::blaze::binpack_compile (sourcemeta::core::JSON &schema, const sourcemeta::core::SchemaWalker &walker, const sourcemeta::core::SchemaResolver &resolver, std::string_view default_dialect="") -> void
auto	sourcemeta::blaze::binpack_canonicalize (sourcemeta::core::JSON &schema, const sourcemeta::core::SchemaWalker &walker, const sourcemeta::core::SchemaResolver &resolver, std::string_view default_dialect="") -> void
auto	sourcemeta::blaze::binpack_load (const sourcemeta::core::JSON &input) -> BinPackEncoding

Detailed Description

A space-efficient binary JSON serialization format based on JSON Schema.

This functionality is included as follows:

#include <sourcemeta/blaze/binpack.h>

JSON BinPack is a binary JSON serialization format with a strong focus on space efficiency. It encodes any JSON document, and it runs in both schema-driven and schema-less modes given a matching JSON Schema 2020-12 definition.

Highly space-efficient

A reproducible benchmark study has proven JSON BinPack to be space-efficient in comparison to 12 alternative binary serialization formats in every considered case. It also typically provides a higher average size reduction than general purpose compressors such as GZIP, LZMA and LZ4 configured with their highest compression levels, specially for small and medium-sized documents. For large documents, combining JSON BinPack with a compression format may achieve greater space-efficiency still. The benchmark is live at https://benchmark.sourcemeta.com.

Based on JSON Schema

JSON BinPack adopts the industry-standard schema language for JSON documents, so the schemas you already have in your OpenAPI, AsyncAPI, RAML or other specifications can be reused as they are. There is no additional domain-specific schema language to learn, and no translation step between them. For the same reason, JSON BinPack works alongside any schema evolution tooling from the wider JSON Schema ecosystem.

Optional schema

JSON BinPack is a hybrid serialization format that runs in both schema-driven and schema-less modes. Thanks to JSON Schema being a constraint system rather than a structural definition language, a schema can be as loose or as specific as the use case demands, even within the same document, without reaching for a different serialization technology. Free-form data and unknown fields encode without trouble as long as the corresponding schema validates the data.

Space efficiency over runtime efficiency

When transmitting data over the Internet, time is the bottleneck, which makes computation cheap in comparison. Leaving extremely low-powered devices aside, trading more CPU cycles for better compression is a sensible choice. JSON BinPack excels at improving network performance over unreliable and low-bandwidth connections on mobile and IoT systems. If instead the use case demands extreme runtime efficiency or zero-copy serialization, such as intra-service or inter-process communication, consider a format such as FlatBuffers or Cap'n Proto.

Research

JSON BinPack is the result of binary serialization research at the Department of Computer Science of University of Oxford:

• [JSON BinPack: A space-efficient schema-driven and schema-less binary serialization specification based on JSON Schema](https://www.jviotti.com/dissertation.pdf)
• Benchmarking JSON BinPack
• A benchmark of JSON-compatible binary serialization specifications
• A survey of JSON-compatible binary serialization specification

---

Class Documentation

sourcemeta::blaze::BinPackDecoder

class sourcemeta::blaze::BinPackDecoder

Inheritance diagram for sourcemeta::blaze::BinPackDecoder:

Public Types
using	Stream
Public Member Functions
	BinPackDecoder (Stream &input)
auto	read (const BinPackEncoding &encoding) -> sourcemeta::core::JSON

Member Typedef Documentation

Stream

using sourcemeta::blaze::BinPackDecoder::Stream

Initial value:

 std::basic_istream<sourcemeta::core::JSON::Char,
                                    sourcemeta::core::JSON::CharTraits>

sourcemeta::blaze::BinPackEncoder

class sourcemeta::blaze::BinPackEncoder

Inheritance diagram for sourcemeta::blaze::BinPackEncoder:

Public Types
using	Stream
Public Member Functions
	BinPackEncoder (Stream &output)
auto	write (const sourcemeta::core::JSON &document, const BinPackEncoding &encoding) -> void

Member Typedef Documentation

Stream

using sourcemeta::blaze::BinPackEncoder::Stream

Initial value:

 std::basic_ostream<sourcemeta::core::JSON::Char,
                                    sourcemeta::core::JSON::CharTraits>

sourcemeta::blaze::BinPackEncodingError

class sourcemeta::blaze::BinPackEncodingError

An error that represents an encoding that the runtime does not implement

Inheritance diagram for sourcemeta::blaze::BinPackEncodingError:

Public Member Functions
	BinPackEncodingError (std::string name)
auto	what () const noexcept -> const char *override
auto	name () const noexcept -> std::string_view
	The name of the encoding that could not be recognized.

Typedef Documentation

BinPackEncoding

using sourcemeta::blaze::BinPackEncoding

Initial value:

 std::variant<
    BOUNDED_MULTIPLE_8BITS_ENUM_FIXED, FLOOR_MULTIPLE_ENUM_VARINT,
    ROOF_MULTIPLE_MIRROR_ENUM_VARINT, ARBITRARY_MULTIPLE_ZIGZAG_VARINT,
    DOUBLE_VARINT_TUPLE, BYTE_CHOICE_INDEX, LARGE_CHOICE_INDEX,
    TOP_LEVEL_BYTE_CHOICE_INDEX, CONST_NONE, ANY_PACKED_TYPE_TAG_BYTE_PREFIX,
    UTF8_STRING_NO_LENGTH, FLOOR_VARINT_PREFIX_UTF8_STRING_SHARED,
    ROOF_VARINT_PREFIX_UTF8_STRING_SHARED,
    BOUNDED_8BIT_PREFIX_UTF8_STRING_SHARED, RFC3339_DATE_INTEGER_TRIPLET,
    PREFIX_VARINT_LENGTH_STRING_SHARED, FIXED_TYPED_ARRAY,
    BOUNDED_8BITS_TYPED_ARRAY, FLOOR_TYPED_ARRAY, ROOF_TYPED_ARRAY,
    FIXED_TYPED_ARBITRARY_OBJECT, VARINT_TYPED_ARBITRARY_OBJECT,
    NUMBER_VARINT_TUPLE>

Represents an encoding

Function Documentation

binpack_canonicalize()

auto sourcemeta::blaze::binpack_canonicalize	(	sourcemeta::core::JSON &	schema,
		const sourcemeta::core::SchemaWalker &	walker,
		const sourcemeta::core::SchemaResolver &	resolver,
		std::string_view	default_dialect = "" )->void

Transform a JSON Schema into its canonical form to prepare it for compilation. Keep in mind this function mutates the input schema. Also, the binpack_compile function already performs canonicalization. This function is exposed mainly for debugging and testing purposes. For example:

#include <sourcemeta/blaze/binpack.h>
#include <sourcemeta/core/json.h>
#include <sourcemeta/core/jsonschema.h>
 
#include <iostream>
 
auto schema{sourcemeta::core::parse_json(R"JSON({
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "string"
})JSON")};
 
sourcemeta::blaze::binpack_canonicalize(
    schema, sourcemeta::core::schema_walker,
    sourcemeta::core::schema_resolver);
 
sourcemeta::core::prettify(schema, std::cout);
std::cout << std::endl;

binpack_compile()

auto sourcemeta::blaze::binpack_compile	(	sourcemeta::core::JSON &	schema,
		const sourcemeta::core::SchemaWalker &	walker,
		const sourcemeta::core::SchemaResolver &	resolver,
		std::string_view	default_dialect = "" )->void

Compile a JSON Schema into an encoding schema. Keep in mind this function mutates the input schema. For example:

#include <sourcemeta/blaze/binpack.h>
#include <sourcemeta/core/json.h>
#include <sourcemeta/core/jsonschema.h>
 
#include <iostream>
#include <sstream>
 
auto schema{sourcemeta::core::parse_json(R"JSON({
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "integer",
  "minimum": -100,
  "maximum": 100
})JSON")};
 
sourcemeta::blaze::binpack_compile(
    schema, sourcemeta::core::schema_walker,
    sourcemeta::core::schema_resolver);
 
const auto encoding{sourcemeta::blaze::binpack_load(schema)};
 
std::ostringstream output;
sourcemeta::blaze::BinPackEncoder encoder{output};
encoder.write(sourcemeta::core::JSON{5}, encoding);
 
std::istringstream input{output.str()};
sourcemeta::blaze::BinPackDecoder decoder{input};
sourcemeta::core::prettify(decoder.read(encoding), std::cout);
std::cout << std::endl;

binpack_load()

auto sourcemeta::blaze::binpack_load

(

const sourcemeta::core::JSON &

input

)

->BinPackEncoding

Load an encoding schema into its in-memory representation