WebCodecs
The browser's own video and audio codecs, addressed frame by frame - no media element, no container. Encode what you draw, decode into pixels you own, and decide yourself what happens in between.
@inject Bit.Butil.WebCodecs webCodecsMDN reference
Support depends on the whole configuration - the codec string with its profile and level, the resolution, and whether a hardware encoder exists when one was asked for. Probe before building, because an unsupported configuration comes back as a null handle rather than an exception.
@inject Bit.Butil.WebCodecs webCodecs
var isSupported = await webCodecs.IsSupported();
var ok = await webCodecs.IsConfigSupported(new VideoEncoderConfig
{
Codec = "vp8",
Width = 640,
Height = 480,
Bitrate = 1_000_000,
Framerate = 30,
LatencyMode = "realtime"
});A frame is built here from pixels generated in C#, handed to an encoder, and comes back as compressed chunks on the output callback - the first of which carries the decoder description a matching decoder will need. Nothing writes a container: what you do with the chunks is the point of the API.
var encoder = await webCodecs.CreateVideoEncoder(
new VideoEncoderConfig { Codec = "vp8", Width = 320, Height = 240, Bitrate = 500_000, Framerate = 10 },
onChunk: chunk =>
{
// mux it, send it, store it - the chunk is yours
Console.WriteLine($"{chunk.Type} {chunk.Data.Length} bytes at {chunk.Timestamp}us");
},
onError: message => Console.WriteLine(message));
await using var frame = await webCodecs.CreateFrame(rgbaBytes, "RGBA", 320, 240, timestamp: 0);
await encoder!.Encode(frame!, keyFrame: true);
await encoder.Flush(); // an encoder holds frames back; this is what makes it hand them overThe chunks produced above are decoded straight back, and each frame is drawn to a canvas and disposed. Disposal is not optional here: a frame holds real memory - often a GPU surface - and a decoder whose frames pile up undisposed stalls.
Bit.Butil.WebCodecs webCodecs
<canvas @ref="canvasElement" width="640" height="480"></canvas>
{
private ElementReference canvasElement;
private async Task Play(EncodedVideoChunk firstChunk, IEnumerable<EncodedVideoChunk> chunks)
{
var decoder = await webCodecs.CreateVideoDecoder(
new VideoDecoderConfig { Codec = "vp8", Description = firstChunk.DecoderDescription },
onFrame: async frame =>
{
// onFrame is an Action, so this lambda is async void: a throw would be unobserved
try { await frame.DrawTo(canvasElement); }
finally { await frame.DisposeAsync(); } // required - the decoder stalls without it
});
foreach (var chunk in chunks) await decoder!.Decode(chunk);
await decoder!.Flush();
}
}A frame can also be grabbed from a video, canvas or image element - which is how a camera stream reaches an encoder without a recorder in between. The timestamp is yours to choose: it is the app's timeline, and an encoder paces its output from it.
Bit.Butil.WebCodecs webCodecs
@* A <video> playing a camera stream, a <canvas> you draw into, an <img> that has loaded. *@
<canvas @ref="canvasElement" width="640" height="480"></canvas>
{
private ElementReference canvasElement;
private VideoEncoderHandle? encoder; // from CreateVideoEncoder
// The timestamp is microseconds, and it is what orders the encoded chunks - 33,333 is one frame
// at 30fps.
private async Task Capture(int frameIndex)
{
await using var frame = await webCodecs.CaptureFrame(canvasElement, timestamp: frameIndex * 33_333);
await encoder!.Encode(frame!);
}
}The same shape, one dimension down: build AudioData from PCM samples, encode it to Opus or AAC, and decode it back. Sample rate and channel count have to match the configuration - an encoder does not resample.
var encoder = await webCodecs.CreateAudioEncoder(
new AudioEncoderConfig { Codec = "opus", SampleRate = 48000, NumberOfChannels = 1, Bitrate = 64_000 },
onChunk: chunk => Console.WriteLine($"{chunk.Data.Length} bytes at {chunk.Timestamp}us"));
await using var data = await webCodecs.CreateAudioData(pcmFloatBytes, "f32", 48000,
numberOfFrames: 480, numberOfChannels: 1, timestamp: 0);
await encoder!.Encode(data!);
await encoder.Flush();await using is the shape that gets this right.
GetQueueSize and stop feeding when it climbs:
every queued frame holds its pixels, so a producer that ignores it turns a smooth encode into an
out-of-memory crash.
API reference
ValueTask<bool> IsSupported()ValueTask<bool> IsConfigSupported(VideoEncoderConfig | VideoDecoderConfig | AudioEncoderConfig | AudioDecoderConfig config)ValueTask<VideoEncoderHandle?> CreateVideoEncoder(VideoEncoderConfig config, Action<EncodedVideoChunk> onChunk, Action<string>? onError = null)ValueTask<VideoDecoderHandle?> CreateVideoDecoder(VideoDecoderConfig config, Action<VideoFrameHandle> onFrame, Action<string>? onError = null)ValueTask<AudioEncoderHandle?> CreateAudioEncoder(AudioEncoderConfig config, Action<EncodedAudioChunk> onChunk, Action<string>? onError = null)ValueTask<AudioDecoderHandle?> CreateAudioDecoder(AudioDecoderConfig config, Action<AudioDataHandle> onData, Action<string>? onError = null)ValueTask<VideoFrameHandle?> CaptureFrame(ElementReference source, long timestamp, long? duration = null)ValueTask<VideoFrameHandle?> CreateFrame(byte[] data, string format, int width, int height, long timestamp, long? duration = null)ValueTask<AudioDataHandle?> CreateAudioData(byte[] data, string format, int sampleRate, int numberOfFrames, int numberOfChannels, long timestamp)ValueTask<CodecState> GetState()ValueTask<int> GetQueueSize()ValueTask<bool> Flush()ValueTask Reset()ValueTask DisposeAsync()ValueTask<bool> Encode(VideoFrameHandle frame, bool keyFrame = false)ValueTask<bool> Decode(EncodedVideoChunk chunk)ValueTask<bool> Encode(AudioDataHandle data)ValueTask<bool> Decode(EncodedAudioChunk chunk)ValueTask<bool> DrawTo(ElementReference canvas)ValueTask<byte[]?> CopyToBytes()ValueTask<byte[]?> CopyToBytes(int planeIndex = 0)