Audio & Video

Core is an image viewer. A canvas whose painting bodies are Sound or Video gets core's unsupported presentation: a notice that keeps the canvas in layout, navigation, and the thumbnail strip, but plays nothing.

@triiiceratops/plugin-av claims those canvases (takes over rendering them from core) and plays their media instead: a media stage over the canvas rect, playback controls in the viewer's own control bar, waveforms, captions, a transcript panel, and an AVState object your application can command playback through.

Install

pnpm add @triiiceratops/plugin-av

npm install @triiiceratops/plugin-av

bun add @triiiceratops/plugin-av

triiiceratops, @triiiceratops/plugin-sdk, and svelte are peers.

Registering it

AvPlugin is exported ready to use with no configuration. Add it to the viewer like any other plugin. See adding a plugin to your viewer for the full per-framework code:

import 'triiiceratops/element/register';
import { AvPlugin } from '@triiiceratops/plugin-av';

viewer.plugins = [AvPlugin];

import { TriiiceratopsViewer } from 'triiiceratops/react';
import { AvPlugin } from '@triiiceratops/plugin-av';

const plugins = [AvPlugin];

export function Reader() {
    return (
        <TriiiceratopsViewer
            manifestId="https://example.org/manifest.json"
            plugins={plugins}
            style={{ display: 'block', height: '600px' }}
        />
    );
}

<script setup lang="ts">
import { TriiiceratopsViewer, type SdkPlugin } from 'triiiceratops/vue';
import { AvPlugin } from '@triiiceratops/plugin-av';

const plugins: readonly SdkPlugin[] = [AvPlugin];
</script>

<template>
    <TriiiceratopsViewer
        manifest-id="https://example.org/manifest.json"
        :plugins="plugins"
        style="display: block; height: 600px"
    />
</template>

<script>
    import { TriiiceratopsViewer } from 'triiiceratops/svelte';
    import 'triiiceratops/style.css';
    import { AvPlugin } from '@triiiceratops/plugin-av';
</script>

<TriiiceratopsViewer
    manifestId="https://example.org/manifest.json"
    plugins={[AvPlugin]}
/>

As a script tag (IIFE)

This plugin differs from every other Triiiceratops plugin in two ways.

Host the dist directory, not one file out of it. The lazy chunks are sibling ES modules the entry import()s by a URL resolved against its own document.currentScript.src. Copy node_modules/@triiiceratops/plugin-av/dist/ somewhere your server serves it and point the script tag inside that directory. A deployment that copies only iife.js works until a user reaches a feature served by a chunk (an HLS stream, a canvas with linked waveform data, a canvas whose duration is tiled by several media files, or a caption track that loaded with cues in it), and then fails with 404 errors.

Core's script must load first. Unlike other plugins, this one does not bundle a Svelte runtime; it reads core's off window.Triiiceratops, which keeps it small. See script order for what the diagnostic looks like when the order is wrong.

<script src="/assets/triiiceratops-element.iife.js"></script>
<script src="/assets/plugin-av/iife.js"></script>

<triiiceratops-viewer id="viewer"></triiiceratops-viewer>
<script>
    // Loading the script only registers a factory; activation is per-viewer.
    document.getElementById('viewer').plugins = [
        window.Triiiceratops.plugins.get('@triiiceratops/plugin-av'),
    ];
</script>

The shared runtime is a first-party arrangement, not a pattern to copy: svelte/internal is private, unversioned API, and it is only safe here because core and this plugin are built and released from one repository at one Svelte version. The plugin pins coreRange to the 1.x line to say so, and refuses to activate against anything else. If you are writing your own plugin, bundle your own Svelte runtime.

What it renders

• Stage layout. The claimed canvas rect is divided into lanes in canvas space, so the whole stack pans and zooms with the viewer. Video takes the whole rect as a visual lane. Audio takes the whole rect as a timeline lane, which draws a waveform when the canvas links audiowaveform data and a ruler otherwise. When core paints a companion canvas into the rect, the stage draws no lanes. The rect belongs to the renderer, and the stage contributes only a tap target, the play-state glyph and the "can't play" notice. That covers a canvas with an accompanyingCanvas, and a canvas with a placeholderCanvas until its first play, so a recording published with a poster shows no waveform until it starts playing. Because the lane is a region of the rect, it keeps the canvas's own aspect ratio: a publisher that declares a very wide, very short audio canvas gets a correspondingly shallow lane.

• Timeline projection. Canvas x maps linearly to media time, so the viewer's own zoom doubles as temporal zoom into the lane's ruler or waveform, and a tap on the timeline lane is a seek.

• Transport. Play/pause, a real role="slider" scrubber with arrow-key seeking and buffered ranges, elapsed/total time, mute and volume, and a captions control when tracks exist. It is registered into core's control bar through the transport-chrome extension point, so it is themed and placed like the rest of the viewer's chrome. There is only one, and it drives the current canvas. Every other claimed canvas on screen carries a decorative play-state glyph on its own stage, so a user with several recordings in view can tell which one is playing. Every label comes from the plugin's own locale catalog in the viewer's active locale. Native controls are never shown.

  While the transport is registered, nav.align is inert and the control bar spans its full width, because a wider seek bar allows more precise seeking. nav.style, nav.edge, and controls are unaffected. See nav alignment and playback controls.

• Waveform. Drawn from audiowaveform data linked from the canvas, parsed into one peaks model regardless of file format. Temporal zoom is limited to the data's own resolution.

• Transcript panel. A list of timestamped, keyboard-operable buttons: the entry at the playhead is marked, clicking one seeks without starting playback, and following the playhead stops the moment you scroll away. It lists entries from whichever of these sources the current canvas offers:

  Source	IIIF shape
  Caption cues	the WebVTT track behind the captions control
  The manifest's timed notes	commentary annotations whose target carries #t= and whose body is a text/plain TextualBody, earliest first (Cookbook 0103). motivation is never inspected
  An untimed transcript	one text/plain file linked from the canvas's rendering (Cookbook 0017)

  The panel works for audio and video alike. For audio, it is the only place caption text is shown, since an <audio> element has no area to paint captions in. The captions toggle is video-only for that same reason. The toolbar button is absent on a canvas offering none of the three, so the panel never opens empty. Its label matches its content: Transcript when there is one, Notes when the canvas offers only the manifest's timed commentary.

Commanding playback: AVState

The plugin publishes its playback state through core's published-state extension point, so a host commands media the same way it commands the viewer: through viewer state, never by importing the object. getAVState is a typed accessor over viewerState.getPluginState('av'); it returns null whenever the plugin is not active on that viewer (absent, failed, or retrying).

import { getAVState } from '@triiiceratops/plugin-av';

const av = getAVState(viewer.viewerState);
if (av) {
    av.seek(30);
    av.play();

    // Batched, payload-free: a notification means "read what you need".
    const stop = av.subscribe(() => {
        console.log(av.paused ? 'paused' : 'playing', 'of', av.duration);
    });

    // The playhead has its own, finer cadence and does NOT notify `subscribe`.
    const stopClock = av.subscribeFrame(() => {
        scrubber.value = String(av.currentTime);
    });
}

Three rules apply:

• All times are canvas time. duration is the canvas's duration and seek() takes a canvas-time position, even when several media files tile that canvas. seek clamps to [0, duration].

• No command throws. A play() the browser's autoplay policy refuses leaves the state paused rather than producing a rejected promise or an exception.

• Commands against a non-AV current canvas are refused, through the plugin error channel's command phase, not by throwing.

Multi-target addressing (seek(canvasId, t)) is a possible future extension and is not in this version. There is one transport, anchored to the current canvas, and every other visible AV canvas shows a play-state glyph.

To open at a media time rather than command one after the fact, pass a temporal offset to setCanvas. See media time.

Manifest shapes it understands

CORS

Media must be CORS-readable. Every media element is created in CORS mode (crossOrigin = 'anonymous'), because a <track> is only fetched when it is, and nearly every Cookbook caption recipe is cross-origin. As a result, the requirement applies to the media as well: a file or stream served without an Access-Control-Allow-Origin header that admits your viewer's origin will fail to load and show the "can't play" notice, even though the same URL plays in a plain <video> tag.

This is the first thing to check when media that works elsewhere will not play here. No credentials are ever sent, so Access-Control-Allow-Origin: * is enough.

Your Content Security Policy needs a media-src that admits the media host, the caption host, and blob: for HLS. See audio and video under a strict CSP.

A stream that cannot play (a dead URL, a CORS or CSP refusal, an offline user) shows a localized "can't play this" notice in that canvas's stage. The rest of the viewer keeps working.

Configuring its UI

The plugin's uiId is av, so it is controlled through config.plugins.av like any other plugin. See controlling plugin UI through config. That key governs the transcript panel and its toolbar button; the media stage and the transport are not plugin chrome and are not switchable.

viewer.config = {
    plugins: {
        av: { position: 'right', open: true },
    },
};

Documented limitations

These limits are intentional in this release:

• No MPEG-DASH. HLS and progressive files only.

• A brief gap at each segment seam. A temporally composed canvas swaps media elements at the boundary; gapless MediaSource stitching is not implemented.

• WebVTT captions only. No TTML, no SRT, no annotation-derived captions, and no caption styling controls.

• Only embedded annotations are read, for caption tracks and for the panel's timed notes alike. A canvas whose annotation page is an external reference contributes neither; no page is ever fetched.

• Spatially placed media renders full-rect or not at all. A painting body targeted at #xywh= is not placed within the canvas. The Cookbook's 0489-multimedia-canvas is the one recipe affected. It renders its image body alone and warns on the developer console, and is counted as unsupported, because compositing is the point of that recipe.

• No waveform on a temporally composed canvas, and no waveform computed in the browser: peaks come from data the manifest links.

• No quality picker, playback rate, fullscreen, or picture-in-picture in the transport. Rendition switching is host-driven through the viewer's choice-selection commands.

• AV canvases are excluded from image and PDF export by documented contract (see Image Download and PDF Export), and a claimed canvas is dropped from annotatableCanvasIds, so the annotation editor offers no tool over a video.

• Timed annotations are read, not drawn. An annotation targeting a #t= range is listed in the transcript panel and seekable from it, but nothing paints it against the playhead or onto the stage, and there is no authoring surface for one. A range's end is carried and not enforced.

Bundle

dist/iife.js is about 15.9 KB gzip. Registered alongside core's element, that makes roughly 131 KB gzip for the pair, which CI holds under a budget measured against TIFY, the nearest audiovisual-capable viewer. The four lazy chunks are outside that figure:

Figures are gzip at level 9, rounded. They vary by tens of bytes between builds, and the byte-exact table lives with the measurement rather than here.

hls.js alone is larger than core and the plugin combined, so it is a separate chunk: a manifest of MP4s never loads it.

Design notes

The plugin is authored entirely on the framework-neutral plugin SDK, and core never imports it. It uses four core extension points (capabilities) and declares each in requiredCapabilities so it fails closed rather than half-working:

Why audiovisual support is a plugin built on a generic canvas claim, rather than a core feature, is recorded in ADR 0017; the published-state extension point behind AVState is covered in ADR 0018.