Crate pallet_schemas

Schema Management for Message and Stateful Storage

Quick Links

• Configuration: Config
• Extrinsics: Call
• Runtime API: SchemasRuntimeApi
• Custom RPC API: SchemasApiServer
• Event Enum: Event
• Error Enum: Error

Schemas Pallet

The Schemas Pallet provides universal schema registration for data flowing through Frequency.

Summary

This pallet provides an on chain repository for Schemas, Intents, and IntentGroups, thereby allowing participants of the network to flexibly interact and exchange messages with each other with the correct human intent and data structure. All Messages and Stateful Storage content must be attached to both an Intent and a Schema Identifier so that the content can be correctly located, parsed, and interpreted.

Data Structure vs Human Intent

Schemas provide for consistent data structures, while Intents encapsulate the human intent of the message. Some schemas may be structurally the same, but have a different interpretation of the contents. For example, two schemas might both only have a hash for contents, but one is a recording of the hash for time validation purposes, while the other is to mark an off-chain vote.

• Intents designate the semantic meaning or purpose of a payload, as well as designating its location.
• Schemas indicate how the data of a payload is to be parsed or interpreted. Many schemas can implement the same Intent as a data format evolves over time, but the semantic meaning or purpose does not change, nor does the storage lcoation.

Groups

Intent Groups are lists of Intents that are maintained on-chain. These groups are intended mostly for off-chain discovery by dApps to determine which Intents need to be requested as delegations for different purposes.

Name Resolution

Intents & Intent Groups can be looked up by name (Schemas cannot). Names are of the form <protocol>.<descriptor>.
NOTE: until all nodes implementing deprecated legacy RPCs are fully removed, Schemas can still be looked up by their associated Intent name using the SchemasPallet_get_schema_versions_by_name runtime API..

Structure

Intent Parameters

• Settings: Various options for the Intent like signature requirements, append-only behavior, etc.
• Payload Location: Where data associated with this Intent is stored.

Schema Parameters

• Model: The actual JSON representing the data structure.
• Model Type: The type of serialization used. (Parquet, Avro, etc…)
• Intent ID: The Intent that the Schema implements
• NOTE: The state storage for Schemas includes a copy of the associated Intent’s settings and payload_location. This is an optimization introduced so that Intent storage does not need to be read when writing to a Schema.

Model Types

• Parquet: Designed for lists or large numbers of records; especially in the publication of record batches by a Provider on behalf of multiple MSAs.
• AvroBinary: Useful for most generic data structures, preferred for on-chain data (OnChain, Itemized, and Paginated payload locations)

Settings

• AppendOnly
  • Prior data is immutable and all new data is appended to existing data.
  • For Payload Locations: Itemized
• SignatureRequired
  • An MSA control key signature is required instead of a delegation.
  • For Payload Locations: Itemized or Paginated

Payload Locations

• OnChain: Data is stored directly in the Messages pallet data storage, usually as AvroBinary.
• IPFS: Data is stored in IPFS and Messages pallet stores the CID.
• Itemized: Data is stored in the Stateful Storage pallet as an array of individual items.
• Paginated: Data is stored in the Stateful Storage pallet as a list of paged blobs.

Mainnet vs Testnet Entity Creation

On Mainnet, Schemas, Intents, and Intent Groups can only be created through the use of the propose_to_create_XXX extrinsics, which require approval by the Frequency Council in order to be executed. This is to prevent malicious schemas and increase the documentation around the schemas available.

On Testnets, these entities can be created by anyone using the create_XXX extrinsics directly, so there are no guarantees around schema correctness or quality. If you want to use a trusted Schema or Intent on a testnet, it is suggested that you use specific Schema Ids or publish the necessary entity yourself.

On all chains, the create_XXX_via_governance extrinsics exist only to support executing a proposal, and are not meant to be invoked directly.

Note, both Testnet and local development chains are seeded with the Intents, Groups, and Schemas from Mainnet, to facilitate ease of testing.

Actions

The Schemas pallet provides for:

• Registering or proposing new Schemas, Intents, and Intent Groups.
• Retrieving entities by their Id (all) or name (Intents and Intent Groups).
• Retrieving last registered Schema/Intent/IntentGroup Id.
• Updating the status of a Schema
• Modifying (overwriting) the Intents contained within an IntentGroup

Interactions

Extrinsics

Name/Description	Caller	Payment	Key Events	Runtime Added
set_max_schema_model_bytes Governance action to alter the maximum byte length of Schema models	Governance	Tokens	SchemaMaxSizeChanged	1
propose_to_create_schema_v3 Creates a proposal to the Frequency Council for a new schema	Token Account	Tokens	Proposed	?
create_schema_via_governance_v3 Governance action version of create_schema_v3	Frequency Council	Tokens	SchemaCreated	?
create_schema_v4 Creates a new Schema.	Mainnet: Governance Testnet: Token Account	Tokens	SchemaCreated	?
create_intent Creates a new Intent (local/testnet only)	Mainnet: filtered Testnet: Token Account	Tokens	IntentCreated	?
create_intent_via_governance Governance action version of create_intent	Frequency Council	Tokens	IntentCreated	?
propose_to_create_intent Creates a proposal to the Frequency Council for a new Intent	Token Account	Tokens	Proposed	?
create_intent_group Creates an Intent Group	Mainnet: prohibited Testnet: Token Account	Tokens	IntentGroupCreated	?
create_intent_group_via_governance Governance action version of create_intent_group	Frequency Council	Tokens	IntentGroupCreated	?
propose_to_create_intent_group Creates a proposal to the Frequency Council for a new Intent Group	Token Account	Tokens	Proposed	?
update_intent_group Overwrites an existing IntentGroup with a new list of Intents	Mainnet: prohibited Testnet: Token Account	Tokens	IntentGroupUpdated	?
update_intent_group_via_governance Governance action verison of update_intent_group	Frequency Council	Tokens	IntentGroupUpdated	?
propose_to_update_intent_group Creates a proposal to the Frequency Council to update an existing Intent Group	Token Account	Tokens	Proposed	?

See Rust Docs for more details.

State Queries

Name	Description	Query	Runtime Added
Get Max Current Schema Identifier	Fetch current Schema Identifier maximum	currentSchemaIdentifierMaximum	1
Get Max Current Intent Identifier	Fetch current Intent Identifier maximum	currentIntentIdentifierMaximum	??
Get Max Current Intent Group Identifier	Fetch current IntentGroup Identifier maximum	currentIntentGroupIdentifierMaximum	??
Get Schema Model Max Bytes	Fetch maximum number of bytes per Schema Model as set by Governance	governanceSchemaModelMaxBytes	1
Get a Schema Info	Fetch the metadata and settings for a schema	schemaInfos	62
Get Schema Payload/Model	Fetch the payload/model JSON for the specified Schema	schemaPayloads	62
Get Intent Info	Fetch the metadata for an Intent	intentInfos	??
Get Intent Group Info	Fetch the list of Intents registered in an Intent Group	intentGroups	??
Get Intent/IntentGroup IDs by Name	Fetch matching Intent/IntentGroup IDs by protocol and descriptor	nameToMappedEntityIds	??

See the Rust Docs for additional state queries and details.

RPCs

Note: May be restricted based on node settings and configuration.

Name	Description	Call	Node Version
Get Schema by Id	Retrieves the schema for the given Schema Id	getBySchemaId	v1.0.0+

* Must be enabled with off-chain indexing

See Rust Docs for more details.

Runtime API

Name	Description	Call	API Version Added	Runtime Added
Get Schema by Id (deprecated)	Retrieves the schema for the given Schema Id	getBySchemaId	1	1
Get Schema by Id (version 2)	Retrieves the schema for the given SchemaId	getSchemaById	3	?
Get Schema Versions by Name (deprecated)	Retrieves the ordered list of Schema Ids for the given name(s)	getSchemaVersionsByName	2	66
Get registered entities (Intent, IntentGroup) by Name	Retrieves the entities belonging to the given name(s)	getRegisteredEntitiesByName	3	?
Get Intent by Id	Retrieves the Intent for the given IntentId	getIntentById	3	?
Get IntentGroup by Id	Retrieves the IntentGroup for the given IntentGroupId	getIntentGroupById	3	?

See Rust Docs for more details.

Implementations

• SchemaValidator: Functions for accessing and validating Schemas. This implementation is what is used in the runtime.
• SchemaProvider: Allows another pallet to resolve schema information.

Re-exports

pub use pallet::*;

pub use weights::*;

Modules

migration

Storage migrations

pallet

The pallet module in each FRAME pallet hosts the most important items needed to construct this pallet.

weights

Autogenerated weights for pallet_schemas

Structs

GenesisIntent

Genesis Intents need a way to load up and this is it!

GenesisIntentGroup

Genesis IntentGroups need a way to load up and this is it!

GenesisSchema

Genesis Schemas need a way to load up and this is it!

GenesisSchemasPalletConfig

Overall Genesis config loading structure for the entire pallet

IntentGroup

A structure defining an IntentGroup

IntentInfo

A structure defining Intent information

SchemaInfo

A structure defining Schema information (excluding the payload)

SchemaName

A structure defining name of a registered entity in this pallet. Names consist of a protocol (namespace) and a descriptor. Currently, names may be registered for Intents and IntentGroups. See name-resolution for more info. (Note, the type name SchemaName is a relic from the previous implementation of this pallet; Schemas themselves can no longer be directly associated with names.)

SchemaVersionId

A structure defining a vector of SchemaIds representing monotonically increasing versions of a Schema.

Constants

DESCRIPTOR_MAX

The maximum size of a schema descriptor

DESCRIPTOR_MIN

The minimum size of a schema descriptor

MAX_INTENTS_PER_DELEGATION_GROUP

Maximum number of intents allowed per delegation group

MAX_NUMBER_OF_VERSIONS

maximum number of versions for a certain schema name -1 is to avoid overflow when converting the (index + 1) to SchemaVersion in SchemaVersionId

PROTOCOL_NAME_MAX

The maximum size of a namespace in schema

PROTOCOL_NAME_MIN

The minimum size of a namespace in schema

SCHEMA_NAME_BYTES_MAX

The maximum size of a fully qualified name, including all parts and separators

SCHEMA_STORAGE_VERSION

Current storage version of the schemas pallet.

SEPARATOR_CHAR

separator character

Traits

ConvertToResponse

trait to create a response from a name and entity identifier

Type Aliases

FullyQualifiedName

A structure defining a fully qualified name of an entity

SchemaDescriptor

schema descriptor type

SchemaNamePayload

A fully qualified name has the following structure PROTOCOL.DESCRIPTOR

SchemaPayload

Type alias for SchemaPayload

SchemaProtocolName

schema namespace type