Georeferenced media API

Georeferenced images, video and dynamic frames (16.1)

Common scripts use MapLibre-shaped image/video/canvas sources, raster layers and setCoordinates/play/pause. Applications build their own layer panels and playback controls. This stage accepts host-decoded RGBA frames. Video URLs are connected in16.4 below; Canvas input is connected below; terrain draping is connected within terrain-quad; custom rendering extensions remain in progress; this is not complete media compatibility.

// @gis-api 0.5-session
// @permissions ["project.read", "map.write"]
import { map } from 'sqsmap';
map.setProjection({ type: 'mercator' });
map.addSource('camera', {
  type: 'video',
  coordinates: [[-4, 4], [4, 4], [4, -4], [-4, -4]],
  attribution: 'Your media source'
});
map.addLayer({ id: 'camera-layer', type: 'raster', source: 'camera' });
await map.sqs.flush();
const camera = await map.getSource('camera');
await camera.sqs.submitFrame({ sequence: 1, width: 1, height: 1, data: [255, 0, 0, 255] });
await camera.pause();
await camera.play();
await map.sqs.flush({ render: true });
export default await camera.sqs.getState();

Coordinates are WGS84 longitude/latitude corners in top-left, top-right, bottom-right, bottom-left order. The footprint must be finite, convex, nonintersecting and within Mercator bounds, without crossing the date line. Flat/pitched Mercator and spherical media displays are connected; terrain draping is connected within terrain-quad. Both source types accept frames; only video accepts playback control. raster-opacity supports constants and existing zoom expressions, not feature properties, feature state or filters. Standard layer APIs control opacity, zoom range, visibility and ordering. Actual four-corner georeferencing is rendered rather than its bounding box. Media has no queryable features.

Sequence numbers are positive safe integers and must increase strictly. Duplicate/late frames return CONFLICT. Paused input is discarded with applied:false, preserving sequence, revision and pixels. Resuming accepts later sequences. Input is copied before atomically replacing the sole latest frame; failures retain the old frame. Removing/recreating a source invalidates old handles and late decoder deliveries with STALE_HANDLE. Remove referencing layers before removing a source; map destruction releases resources.

Dimensions are 1–4096, with exact width×height×4 RGBA bytes, 32MiB per frame, 64MiB per map and at most 32 media sources. Script JSON arrays additionally obey script message budgets. For large frames, use the host binary API: standalone SDK source.sqs.submitFrame takes Uint8Array; WASM mediaFrame and C sqsm_map_media_frame share identity, sequence and budget semantics. C input is borrowed only during the call; output is independently owned and freed using sqsm_map_bytes_free. Failed output allocation does not commit a frame. The library includes the portable pure-Rust MP4/H.264 decoder below; it does not own browser GL contexts or HTTP. Host-decoded frames still use the same binary input.

getSourceInfo includes mediaState and loading state; a source is not loaded before its first frame. Existing sourcedataloading/sourcedata events describe source changes. Command confirmation means application; render:true waits for a completed current display. Desktop, native QuickJS and browser main-thread/strict Worker use the same acceptance script. Local software-GPU evidence does not replace external-device or formal-release acceptance.

URL image sources and updateImage (16.2)

Image sources accept a credential-free HTTP(S) url. Hosts download through the shared resource policy; the core decodes PNG/JPEG/WebP with at most4MiB encoded input,4096 pixels per edge and32MiB RGBA. No browser image object is required; existing512px sprite limits remain unchanged.

map.addSource('photo', {type:'image',url:'https://your.example/orthophoto.png',coordinates:[[-4,4],[4,4],[4,-4],[-4,-4]]});
map.addLayer({id:'photo-layer',type:'raster',source:'photo'});
await map.sqs.flush();
const photo=await map.getSource('photo');
await photo.updateImage({url:'https://your.example/new.png',coordinates:[[-5,5],[5,5],[5,-5],[-5,-5]]});

Confirmation only registers the new request; use sourcedata/isSourceLoaded for download/decode completion. New pixels and optional corners commit together. Failure retains old pixels and georeferencing, emits error and endsloading while hasData remains true. Generations reject late responses; another update or removal cancels the old request. setCoordinates returns CONFLICT while an update has captured corners; an update without corners allows independent coordinate changes. Explicit submitFrame cancels pending image requests and clears URL snapshot ownership, preventing old responses from replacing the new frame.

Capability queries expose mediaImageURLSources/mediaImageUpdates/mediaFrames/mediaRasterPlanar/mediaRasterPitched. mediaVideoURLSources is connected in16.4; Canvas named frames are connected; terrain andcustom rendering still returnfalse. Shared capability flags do not establish mobile script-host acceptance.

The complete HTML example includes styles, all button handlers, image loading events, attribution and destruction cleanup.

Spherical media rendering (16.3)

Registered image/video RGBA media and URL image sources can use map.setProjection({type:'globe'}); mediaRasterGlobe is true. Handles, frame sequences, pause state, coordinates, opacity, visibility and layer order retain their meanings. Switching Mercator/globe preserves media. Actual four-corner spherical meshes reuse shared depth, lighting and texture caches rather than a planar screenshot. Coordinates still obey the published Mercator georeferencing bounds; date-line crossing is unsupported.

Spherical lighting can change pixel values. Acceptance checks channels and actual layer occlusion rather than requiring bytes identical to flat rendering. Media produces no queryable features; other vector queries keep their spherical semantics. Cancellation/destruction releases resources. Frame confirmation checks camera,density,business revisions and media frame signatures, preventing an old texture from passing a render barrier.

URL videos and shared playback clock (16.4)

import { map } from 'sqsmap';
map.addSource('film', {
  type: 'video',
  urls: ['https://your.example/clip.mp4'],
  coordinates: [[-4,4],[4,4],[4,-4],[-4,-4]],
  attribution: 'Video author and license'
});
map.addLayer({ id:'film-layer', type:'raster', source:'film' });
await map.sqs.flush();
const film=await map.getSource('film');
await film.pause();
await film.play();
export default await film.sqs.getState();

urls contains1–8 credential-free HTTP(S) alternatives, downloaded/decoded in order under the same source identity. A terminal error occurs after all alternatives fail; RESOURCE_LIMIT/CANCELLED do not try another URL. The core does not perform HTTP: browser, desktop and C callers use the shared resource policy. sourcedata.isSourceLoaded becomes true only after an accepted decoded first frame. Registration confirmation is not loading completion. Subscribe with map.on; pausing before loading still permits the first frame.

Every host uses the same pure-Rust ordinary MP4/H.264 decoder:8-bit progressive YUV420. Fragmented MP4,WebM,HEVC,audio playback and browser-specific getVideo are unsupported. Encoded input is64MiB per file/map, with at most2 URL videos per map,1 hour/216000 samples. Coded edges are at most2048 and total pixels at most1920×1088. Display frames retain the desktop960×540 downscale rule and limited-range BT.601 conversion. Reorder buffering is at most16 frames; each map-owner tick decodes at most32 frames per video, never predecoding a whole movie. Unsupported formats fail explicitly. The existing64MiB total latest-RGBA budget still applies.

Playback starts automatically on the map-owned monotonic clock. pause freezes pixels/time;play resumes;end-of-file loops. State adds duration/currentTime in seconds;currentTime describes the accepted presentation frame. sourcedata reports frame changes. Display failure pauses and emits error while retaining the previous accepted pixels. Removal/recreation and map destruction release downloads,compressed input and decoders;late responses cannot revive a source. Explicit sqs.submitFrame takes over media ownership and stops URL decoding;paused external frames still discard.

C resource responses now allow64MiB globally and validate the smaller request budget before copying. GeoJSON8MiB and image4MiB remain unchanged;ABI1 symbols/layout/output ownership remain unchanged. Flat,pitched and spherical raster media are connected;terrain draping remains pending. mediaVideoURLSources=true describes the matching shared implementation, not mobile-host/device acceptance. The same video script and complete HTML example include loading events,play/pause,attribution and cleanup.

Canvas sources

The type:'canvas', canvas, coordinates, and animate vocabulary follows MapLibre CanvasSource; SqsMap implements it with its own core. animate defaults to true. False accepts the initial frame and stays paused; play resumes, pause freezes the latest pixels. getStyle retains the provider name and initial animate setting; query playback through sqs.getState.

Portable scripts pass a string such as canvas:'drawing'. The host produces RGBA pixels and calls source.sqs.submitFrame({sequence,width,height,data}). Script data is a byte array; SDK data is Uint8Array. Sequence numbers are increasing positive safe integers. Readiness requires an accepted first frame. sqs.fail({code:'IO_ERROR',message:'Canvas unavailable'}) ends loading and emits error while retaining accepted pixels. Play and a new valid frame recover. Old handles cannot change a recreated source.

The browser SDK accepts a HTMLCanvasElement or an existing element ID, capturing readable 2D pixels at roughly30fps with at most one submission in flight per source. Capture stays on the main thread even with strict Worker rendering. source.getCanvas() returns the bound browser element and is absent from portable script/native declarations. A string without a corresponding DOM canvas registers a manual named provider. Tainted,empty or non2D canvases fail automatic capture; WebGL/OffscreenCanvas hosts should submit RGBA explicitly. Explicit submission takes ownership and stops automatic capture. Limits remain4096 pixels per edge,32MiB per frame and64MiB for the map's retained frames.

Removal/destruction stops capture and rejects late work by source instance. Raster display supports flat,pitched and spherical views. Terrain draping and custom GL callbacks remain pending. canvasFrameSources/mediaCanvasSources identify the shared frame model; canvasDOM is reported only by the browser SDK. The mobile script host remains unimplemented. The complete HTML/CSS/JS example includes drawing,playback buttons and cleanup; the portable script shows named frame submission.

13.6c supports custom GeoJSON/TileJSON/georeferenced image/video source URLs and updates, retaining decoder/cancellation/identity rules. See source protocols; Style/glyph/loadImage/live entries retain their own scope.

Shared planar composition capability

When a matching SDK reports commonPlanarLayerComposition:true, the shared Mercator view with zero pitch/roll and no terrain/extrusion combines media, backgrounds, labels, heat and supported static line/fill effects in complete layer order. This capability supersedes earlier media-combination exclusions only within that shared planar scope. Custom line animation, tilted/globe/terrain views and other unsupported material combinations retain their respective limits. Reordering and compatible diff preserve identities, caches, pending requests and fade clocks; material, geometry and texture budgets are validated before publication. See planar layer composition.

Browser static 3D models (SqsMap extension)

With matching browser SDK capability staticModels=true, MapView.setModel(bytes, placement) loads one self-contained glTF 2.0 GLB through the same Rust decoder and 3D renderer as desktop. The host performs HTTP with fetch and supplies a Uint8Array. This is separate from common Style model layers and the desktop GIS script file API.

map.setProjection('mercator');
map.jumpTo({ center: [102.671647, 24.970096], zoom: 18.5, pitch: 55 });
const response = await fetch(modelUrl);
if (!response.ok) throw new Error('HTTP ' + response.status);
const placement = { lngLat: [102.671647, 24.970096], elevationM: 0,
  verticalReference: 'local-zero', headingDeg: 45, scale: 40, visible: true };
await map.setModel(new Uint8Array(await response.arrayBuffer()), placement);
await map.flush({ render: true });
const state = await map.getModelState();
if (!state?.rendered) throw new Error('Model has not finished drawing');
await map.setModelPlacement({ ...placement, headingDeg: 90 });
// await map.clearModel();

Each map retains at most one such model. Upload replaces only the previous model, preserving style layers and sources. setModelPlacement replaces the complete placement and visibility while reusing decoded geometry/textures. clearModel releases the model. getModelState returns immutable local bounds, primitive/triangle counts, source byte size and the most recently completed perspective draw status; it returns null after clearing. Upload and placement promises confirm command application, not drawing. Use flush({render:true}) and inspect rendered.

Coordinates are WGS84; elevationM and model coordinates use metres. An explicit verticalReference is required. Here local-zero means an illustrative local zero plane, without surveyed elevation or automatic terrain draping. Heading is clockwise and scale is a positive factor. Writes require Mercator and drawing requires a pitched view; globe/planar drawing and terrain sampling are not provided.

Existing decoder limits apply: source and decoded assets each at most 32 MiB, texture sides at most 2048 pixels, and model geometry within 1 km of its anchor. Static assets only: no skins, animation, transparency or advanced material textures. Base-color textures are embedded in the GLB. Invalid input preserves the previous model. Hosts display asset copyright, provenance and placement notes. The complete bridge example uses a CC0 Kenney Hexagon Kit asset with clearly illustrative placement.