Test

A JSON Schema test runner. More...

Classes
struct	sourcemeta::blaze::TestCase
struct	sourcemeta::blaze::TestOutcome
struct	sourcemeta::blaze::TestTarget
struct	sourcemeta::blaze::TestSuite
class	sourcemeta::blaze::TestParseError

Typedefs
using	sourcemeta::blaze::TestTimestamp = std::chrono::steady_clock::time_point
using	sourcemeta::blaze::TestTargetResolver

Detailed Description

A JSON Schema test runner.

This functionality is included as follows:

#include <sourcemeta/blaze/test.h>

---

Class Documentation

sourcemeta::blaze::TestCase

struct sourcemeta::blaze::TestCase

Represents a single test case in a test suite

Static Public Member Functions
static auto	parse (const sourcemeta::core::JSON &test_case_json, const sourcemeta::core::PointerPositionTracker &tracker, const std::filesystem::path &base_path, const sourcemeta::core::Pointer &location, const sourcemeta::core::PointerPositionTracker::Position &position) -> TestCase
	Parse a single test case.
Public Attributes
sourcemeta::core::JSON::String	description
	The optional description of the test case.
bool	valid
	Whether the test data is expected to be valid against the schema.
sourcemeta::core::JSON	data
	The test data to validate.
std::optional< sourcemeta::core::JSON >	rdf
	The expected promotion of the test data to expanded-form JSON-LD.
sourcemeta::core::PointerPositionTracker	tracker
	The position tracker for error reporting on the data.
sourcemeta::core::PointerPositionTracker::Position	position
	The position of this test case in the test suite file.

sourcemeta::blaze::TestOutcome

struct sourcemeta::blaze::TestOutcome

Represents the outcome of evaluating a single test case against a target

Public Attributes
bool	passed
	Whether the test case passed overall.
bool	valid
	The actual validity outcome of the test data against the target.
std::optional< sourcemeta::core::JSON >	rdf
	The actual expansion, when RDF promotion ran and succeeded.
std::optional< JSONLDResolutionError >	rdf_error
	The resolution error, when RDF promotion failed.

sourcemeta::blaze::TestTarget

struct sourcemeta::blaze::TestTarget

Where a target of a test document lives, as the caller resolved it

The document and the frame are borrowed rather than copied, and the exhaustive template of a target is compiled on the first request, so both must outlive the test suite

Public Attributes
const sourcemeta::core::JSON &	document
const sourcemeta::core::SchemaFrame &	frame
sourcemeta::core::JSON::String	entrypoint

Member Data Documentation

document

const sourcemeta::core::JSON& sourcemeta::blaze::TestTarget::document

The document that holds the schema under test, which may be that schema or a wrapper that holds it, such as an OpenAPI description

entrypoint

sourcemeta::core::JSON::String sourcemeta::blaze::TestTarget::entrypoint

The URI of the schema under test, which the frame must locate. This is typically the target itself, as what a test document names a schema by and what framing the document it lives in keys that schema by are the same URI. An empty string names whatever the frame was rooted at

frame

const sourcemeta::core::SchemaFrame& sourcemeta::blaze::TestTarget::frame

The frame of the schemas that the document holds. For a wrapper, this is what framing the places it keeps its schemas in reports, such as what sourcemeta::core::OpenAPIFrame::schemas returns

sourcemeta::blaze::TestSuite

struct sourcemeta::blaze::TestSuite

Represents a test suite containing multiple test cases

A test suite does not resolve or frame anything. The caller hands over the document that holds each target, a frame of the schemas in it, and where in it the schema under test sits. That is what lets a target name a schema inside a document that is not a schema itself, such as an OpenAPI description, without this module knowing anything about wrappers

Public Types
using	Callback
	A callback invoked for each test case during execution.
Public Member Functions
auto	fast (std::size_t target_index) const -> const Template &
	The compiled schema template for fast validation of the given target.
auto	exhaustive (std::size_t target_index) -> const Template &
auto	run (const Callback &callback) -> Result
Static Public Member Functions
static auto	parse (const sourcemeta::core::JSON &document, const sourcemeta::core::PointerPositionTracker &tracker, const std::filesystem::path &base_path, const TestTargetResolver &target_resolver, const sourcemeta::core::SchemaResolver &schema_resolver, const sourcemeta::core::SchemaWalker &walker, const Compiler &compiler, const std::optional< Tweaks > &tweaks=std::nullopt) -> TestSuite
Public Attributes
std::vector< sourcemeta::core::JSON::String >	targets
std::vector< TestCase >	tests
	The list of test cases in the suite.
Evaluator	evaluator
	The evaluator instance used for validation.

Member Typedef Documentation

Callback

using sourcemeta::blaze::TestSuite::Callback

Initial value:

 std::function<void(
      const sourcemeta::core::JSON::String &target, std::size_t target_index,
      std::size_t index, std::size_t total, const TestCase &test_case,
      const TestOutcome &outcome, TestTimestamp start, TestTimestamp end)>

A callback invoked for each test case during execution.

Member Function Documentation

exhaustive()

auto sourcemeta::blaze::TestSuite::exhaustive

(

std::size_t

target_index

)

->constTemplate &

The compiled schema template for exhaustive validation of the given target, compiled on the first request and cached from then on. A test suite must not be shared across threads

parse()

auto sourcemeta::blaze::TestSuite::parse ( const sourcemeta::core::JSON & document, const sourcemeta::core::PointerPositionTracker & tracker, const std::filesystem::path & base_path, const TestTargetResolver & target_resolver, const sourcemeta::core::SchemaResolver & schema_resolver, const sourcemeta::core::SchemaWalker & walker, const Compiler & compiler, const std::optional< Tweaks > & tweaks = std::nullopt )->TestSuite

static

Parse a test suite from a JSON object, resolving and compiling every target that it names. For example:

#include <sourcemeta/blaze/test.h>
#include <sourcemeta/blaze/compiler.h>
 
#include <sourcemeta/core/json.h>
#include <sourcemeta/core/jsonpointer.h>
#include <sourcemeta/core/jsonschema.h>
 
#include <cassert>
#include <filesystem>
#include <functional>
 
const auto input{R"JSON({
  "target": "https://example.com/string",
  "tests": [
    { "data": "foo", "valid": true },
    { "data": [], "valid": false, "description": "Not a string" }
  ]
})JSON"};
 
sourcemeta::core::PointerPositionTracker tracker;
sourcemeta::core::JSON document{nullptr};
sourcemeta::core::parse_json(input, document, std::ref(tracker));
 
const auto schema{sourcemeta::core::parse_json(R"JSON({
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.com/string",
  "type": "string"
})JSON")};
 
const sourcemeta::core::SchemaFrame frame{
    sourcemeta::core::SchemaFrame::Mode::References, schema,
    sourcemeta::core::schema_walker, sourcemeta::core::schema_resolver};
 
const auto suite{sourcemeta::blaze::TestSuite::parse(
    document, tracker, std::filesystem::current_path(),
    [&schema, &frame](const sourcemeta::core::JSON::String &target)
        -> sourcemeta::blaze::TestTarget {
      return {.document = schema, .frame = frame, .entrypoint = target};
    },
    sourcemeta::core::schema_resolver, sourcemeta::core::schema_walker,
    sourcemeta::blaze::default_schema_compiler)};
 
assert(suite.targets.size() == 1);
assert(suite.targets.front() == "https://example.com/string");
assert(suite.tests.size() == 2);

The frame that a target resolves to must contain reference information for the document it frames, and that document must be bundled, which are the same pre-conditions that the overload of sourcemeta::blaze::compile taking a frame states. The frame must also locate the entry point, so a document is framed under the base that the target was resolved against rather than under no base at all

run()

auto sourcemeta::blaze::TestSuite::run

(

const Callback &

callback

)

->Result

Run all test cases in the suite, invoking the callback for each. For example:

#include <sourcemeta/blaze/test.h>
#include <sourcemeta/blaze/compiler.h>
 
#include <sourcemeta/core/json.h>
#include <sourcemeta/core/jsonpointer.h>
#include <sourcemeta/core/jsonschema.h>
 
#include <filesystem>
#include <functional>
#include <iostream>
 
const auto input{R"JSON({
  "target": "https://example.com/string",
  "tests": [
    { "data": "foo", "valid": true, "description": "a string" }
  ]
})JSON"};
 
sourcemeta::core::PointerPositionTracker tracker;
sourcemeta::core::JSON document{nullptr};
sourcemeta::core::parse_json(input, document, std::ref(tracker));
 
const auto schema{sourcemeta::core::parse_json(R"JSON({
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://example.com/string",
  "type": "string"
})JSON")};
 
const sourcemeta::core::SchemaFrame frame{
    sourcemeta::core::SchemaFrame::Mode::References, schema,
    sourcemeta::core::schema_walker, sourcemeta::core::schema_resolver};
 
auto suite{sourcemeta::blaze::TestSuite::parse(
    document, tracker, std::filesystem::current_path(),
    [&schema, &frame](const sourcemeta::core::JSON::String &target)
        -> sourcemeta::blaze::TestTarget {
      return {.document = schema, .frame = frame, .entrypoint = target};
    },
    sourcemeta::core::schema_resolver, sourcemeta::core::schema_walker,
    sourcemeta::blaze::default_schema_compiler)};
 
const auto result{suite.run(
    [](const sourcemeta::core::JSON::String &target,
       std::size_t, std::size_t index, std::size_t total,
       const sourcemeta::blaze::TestCase &test_case,
       const sourcemeta::blaze::TestOutcome &outcome,
       sourcemeta::blaze::TestTimestamp start,
       sourcemeta::blaze::TestTimestamp end) {
      std::cout << target << " " << index << "/" << total << ": "
                << test_case.description << " - "
                << (outcome.passed ? "PASS" : "FAIL")
                << "\n";
    })};
 
std::cout << result.passed << "/" << result.total << " passed\n";

A target that names a schema inside a document that is not a schema itself is reached the same way, by handing over that document and a frame of the schemas it holds. Nothing else about it differs, as framing a document under the base it was read from keys a schema within it by that base and the place it sits in, which is what the target spells:

// A frame can neither be copied nor moved, and this one has to outlive
// the suite, so the caller keeps it somewhere of its own and hands over
// a reference to it
const auto &frame{*frames.emplace_back(
    std::make_unique<sourcemeta::core::OpenAPIFrame>(
        description, walker, resolver, base))};
return {.document = description, .frame = frame.schemas(),
        .entrypoint = target};

Member Data Documentation

targets

std::vector<sourcemeta::core::JSON::String> sourcemeta::blaze::TestSuite::targets

The target schema URIs or file paths, resolved against the location of the test document

sourcemeta::blaze::TestParseError

class sourcemeta::blaze::TestParseError

An error that occurs when parsing a test file

Inheritance diagram for sourcemeta::blaze::TestParseError:

Public Member Functions
	TestParseError (const char *message, sourcemeta::core::Pointer location, std::uint64_t line, std::uint64_t column)
	TestParseError (std::string message, sourcemeta::core::Pointer location, std::uint64_t line, std::uint64_t column)=delete
	TestParseError (std::string &&message, sourcemeta::core::Pointer location, std::uint64_t line, std::uint64_t column)=delete
	TestParseError (std::string_view message, sourcemeta::core::Pointer location, std::uint64_t line, std::uint64_t column)=delete
auto	what () const noexcept -> const char *override
auto	location () const noexcept -> const sourcemeta::core::Pointer &
auto	line () const noexcept -> std::uint64_t
auto	column () const noexcept -> std::uint64_t

Typedef Documentation

TestTargetResolver

using sourcemeta::blaze::TestTargetResolver

Initial value:

 
    std::function<TestTarget(const sourcemeta::core::JSON::String &target)>

How to reach what a target names, invoked once per target of a test document in the order that the document lists them

TestTimestamp

using sourcemeta::blaze::TestTimestamp = std::chrono::steady_clock::time_point

The monotonic timestamp type used for timing measurements