Overview
This document is intended a starting point for development of an open synchronized review messaging format in which multiple different applications can participate. The intention is that there is a single presenter and one or more participants with whom the presenter shares his current review session. With the proposed schemas below, the presenter can control playback as well as make annotations on the media and communicate these changes to the participants across a network using a message queuing service. The messages include key exchanges so that they can be encrypted end-to-end to ensure privacy between host and participants.
The schemas are all in the human-readable json format, and in some cases existing schemas from the OpenTimelineIO format are used within the LiveReview schemas. The RationalTime, TimeRange from OTIO are used to synchronize to specific times within the media and custom OTIO effects schemas, like Paint and Text and Point are used for the annotations. These are not official OTIO schemas, but can be created with the schema-extension feature offered by OTIO.
Use Cases
The messages are intended to be used in a presenter-participant style review where there is a single presenter controlling the review session who actions will be pushed over a network connection to one or more participants where each user is running their own instance of the application that will reflect the presenter's actions. It is not intended as collaborative shared session where participants can draw annotations together or any other type of collaborative actions taken. However, if necessary any one of the participants can become the presenter, and doing so will change the current presenter's role to participant.
The details of how the messages are transmitted to the user, and the infrastructure required to do so are not within the scope of this proposal. And as such, no particular message queueing system, database or other requirements are specified. As long as the message arrive in order to the participants, any backend infrastructure should be able to be used.
The messages are also intended to be agnostic of any particular application so that it can be used to share sessions across any application that decides to support the them. However, any application using this system is assumed to support basic review functionality such as media playback and annotations. The messages should also contain mechanisms for embedding application-specific data beyond the scope of synchronizing participants so that applications do not require a separate parallel system to ensure their application-specific features stay in sync.
Design Decisions
The decision to use OTIO in the messages was made because of several reasons. Among the most important is that it is rich open source format that many applications already understand, so it seems to be a natural choice to be able to share timelines across both web and desktop applications in order to reduce the burden of adopting the messaging format as much as possible. Another important factor in using OTIO is that a lot of time and effort has already been expended by that team on how to best express and serialize time, a key component of any synchronization. It also provides a rich API around time, eliminating the need to do the (sometimes tricky) mathematical computations around it. Lastly, custom schemas can easily be added to express any effects on the media if need be – like CDLs or other effects for basic luminance, gamma and colour synchronization between apps.
Another important aspect of the messaging service is encryption. The encryption works by using a common shared cryptographic key among participants allowing all participants to encrypt and decrypt messages from each other. When a new participant joins a session, the shared key is returned to them by the server after it has been encrypted using their public key so that it can be decrypted using their private key. This is done to ensure that no sensitive or private data that may be in filenames, storage locations, etc is accidentally leaked during transmission.
An additional design decision was to try to keep the messaging system as lightweight as possible. When a user initially joins a session, they receive a full OTIO Timeline describing the current state of the review. Thereafter, they only receive the changes since their last sync. It's worth nothing that the OTIO format does not currently support incremental changes as was designed as a file storage format more so than a networking protocol. Because of this, schemas that modify a particular clip (like annotations) are always applied to the clip currently being viewed. Exploring partial/incremental updates to OTIO could be a subject worth exploring in the future.
Lastly, the schema system was designed with minimalism in mind. Only messages that directly effect synchronization are transmitted with the aim of performing as much of the work as possible on the local system. This includes the actual playback of the media – only media times are sent across the network no actual pixels are shared as participants are expected to be able to handle playback and access to the media locally.
Event Data Flow
The below diagrams illustrate the dataflow of the messages between a synced review application, a session store that tracks the current session and message queue for distribution of the messages. The schema for each message is provided in the section following this one.
Schema Proposals
The schemas take the form of schema name with a version and a payload. So, in most cases, the json keys come in pairs like "command" and "command_schema" or "event" and "event_schema".
Session Schemas
The first type of schemas centers around the changes to the LiveReview session itself, such as changing the presenter or adding participants.
New Presenter
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "LIVE_SESSION_1.0",
"command": {
"event": "NEW_PRESENTER",
"payload": presenter_hash
}
}
}
Presenter_hash is a hash uniquely identifying the presenter
New Participants
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "LIVE_SESSION_1.0",
"command": {
"event": "NEW_PARTICIPANTS",
"payload": empty
}
}
}
Payload is empty in this case
Shared Key Request
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "SHAREDKEY_1.0",
"command": {
"event": "GET",
"payload": public key
}
}
}
Public key is the user's public cryptographic key
Shared Key Response
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "SHAREDKEY_1.0",
"command": {
"event": "SET",
"payload": hash
}
}
}
Hash is the shared key, encrypted, base64 encoded. It is a symmetric key used between users of the same session, signed with the users public key.
Get Session
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "OTIO_SESSION_1.0",
"command": {
"event": "GET",
"payload": {
user: hash,
app: id
}
}
}
}
The payload is the current user hash and an identified for the application they are using.
Update Session
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "OTIO_SESSION_1.0",
"command": {
"events": "SET",
"payload": {
"otio": OTIO Timeline,
"rv"?:
"xstudio"?:
"xview"?:
}
}
}
}
The otio payload is an OTIO Timeline representing the clip(s) being played. The fields marked with '?' are optional and are meant for sending application-specific information.
Playback Schemas
The next set of schemas are to synchronize playback between users.
Request Sync Playback
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "PLAYBACK_SETTINGS_1.0",
"command": {
"event": "GET",
"payload": empty
}
}
}
The payload is empty here.
Sync Playback
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "PLAYBACK_SETTINGS_1.0",
"command": {
"event": "SET",
"payload": {
"looping"?: true,
"playing"?: false,
"muted"?: false,
"playback_range"?: {
"enabled": false,
"zoomed": false,
"range": {
"OTIO_SCHEMA": "TimeRange.1"
"start_time": {
"OTIO_SCHEMA": "RationalTime.1",
"value": 0.0,
"rate": 30.0
},
"duration": {
"OTIO_SCHEMA": "RationalTime.1",
"value": 1.0,
"rate": 30.0,
}
}
},
"current_time"?: {
"OTIO_SCHEMA": "RationalTime.1",
"value": 1,
"rate": 24
},
"scrubbing"?: false,
"output_bounds"?: {
"OTIO_SCHEMA": "Box2d.1",
"min": {
"OTIO_SCHEMA": "V2d.1",
"x": -8.0,
"y": -4.5
},
"max": {
"OTIO_SCHEMA": "V2d.1",
"x": 8.0,
"y": 4.5
},
},
"source": "/asklfja/3409809".
"source_index": 0
}
}
}
}
The fields with '?' are optional. The full payload is sent upon first sync and thereafter only the fields that changed since the last sync are sent.
Annotation Schemas
The next set of schemas involve user drawn annotations. The event are based on entering the annotation mode, drawing points, ending the annotation mode and clearing annotations.
Paint Start
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "ANNOTATION_1.0",
"command": {
"event": "PAINT_START",
"payload": {
"source_index": 0,
"paint": {
"OTIO_SCHEMA": "Paint.1",
"points": [ ],
"rgba": [ 1.0, 1.0, 0.0, 1.0 ],
"type": "COLOR",
"brush": "circle",
"visible": true,
"name": "Paint",
"effect_name": "Paint",
"layer_range": {
"OTIO_SCHEMA": "TimeRange.1"
"start_time": {
"OTIO_SCHEMA": "RationalTime.1",
"value": 0.0,
"rate": 30.0
},
"duration": {
"OTIO_SCHEMA": "RationalTime.1",
"value": 1.0,
"rate": 30.0
},
},
"hold": false,
"ghost": false,
"ghost_before": 3,
"ghost_after": 3
}
}
}
}
}
Paint Point
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "ANNOTATION_1.0",
"command": {
"event": "PAINT_POINT",
"payload": {
"point_target": {
"source_index": 0,
"range": {
"OTIO_SCHEMA": "TimeRange.1"
"start_time": {
"OTIO_SCHEMA": "RationalTime.1",
"value": 0.0,
"rate": 30.0
},
"duration": {
"OTIO_SCHEMA": "RationalTime.1",
"value": 1.0,
"rate": 30.0,
}
},
}
"point": {
"OTIO_SCHEMA": "Point.1",
"x": 5.669550344154889,
"y": 2.0246346746398025,
"size": 0.05
}
}
}
}
}
Paint End
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "ANNOTATION_1.0",
"command": {
"event": "PAINT_END",
"payload": { }
}
}
}
Clear
{
"live_schema": "SYNC_REVIEW_1.0",
"live_payload": {
"command_schema": "ANNOTATION_1.0",
"command": {
"event": "CLEAR",
"payload": {
"clear_all": false,
"clear_target"?: {
"source_index": 0,
"range": {
"OTIO_SCHEMA": "TimeRange.1"
"start_time": {
"OTIO_SCHEMA": "RationalTime.1",
"value": 0.0,
"rate": 30.0
},
"duration": {
"OTIO_SCHEMA": "RationalTime.1",
"value": 1.0,
"rate": 30.0,
}
}
}
}
}
}
}