Tech
Once streams reach the Media Server, they can be mixed into one. It doesn't matter how each of them was published — WebRTC, RTMP, RTSP, MPEG-TS over SRT, or any other supported protocol.
Mixing happens entirely on the server side. The Media Server decodes the incoming streams, places their pictures on one canvas, mixes their sound and encodes the result as a new stream, the output stream of the mixer.
The mixed stream is like any other on the server: it can be played over WebRTC, HLS, RTSP or RTMP, recorded, and pushed to YouTube or another server. Viewers play one stream instead of several, however many participants there are.
Mixing means transcoding, so every mixer takes server CPU: plan about two cores per mixer.
Every participant's application uses an SDK to publish the camera and microphone as usual, and the name of the stream says which mixer it joins: everything after the # sign is the name of the mixer.
For a stream named alice#room-42 the Media Server creates the room-42 mixer, if there is none yet, and adds the stream to it. The next stream, bob#room-42, joins the same mixer, and viewers play the mixed stream room-42.
Now let's see how to publish a stream into a mixer from each SDK.
The code looks a little different on each platform, but in every SDK the mixer is just a part of the published stream’s name.
In a browser, the Web SDK publishes the stream, and its name tells the Media Server which mixer to add it to:
session
.createStream({
name: 'alice#room-42',
display: localPreview,
})
.publish();Everything after # is the name of the mixer: the Media Server creates room-42 and adds the stream to it.
On Android the mixer is named in the stream options before publishing:
StreamOptions options = new StreamOptions("alice#room-42");
Stream stream = session.createStream(options);
stream.publish();The stream joins the mixer by its name, alice#room-42; the rest is ordinary publishing.
On iOS the mixer is named in the stream options too:
let options = FPWCSApi2StreamOptions()
options.name = "alice#room-42"
options.display = localView
let stream = try session.createStream(options)
try stream.publish()The stream joins the mixer by its name, alice#room-42; the rest is ordinary publishing.
The publishers do not have to know about the mixer. Your backend can mix any streams published on the Media Server through the REST API, whatever protocol they came in with: WebRTC, RTMP, RTSP, MPEG-TS or any other supported protocol.
The backend creates a mixer with /mixer/startup, giving it an identifier like mixer://room-42 and the name of the output stream, and adds streams to it by their names with /mixer/add. /mixer/remove takes a stream out when a participant leaves, and /mixer/terminate stops the mixer.
One mixer, a stream added for every participant:
const api = 'https://media.example.com:8444/rest-api/v3';
const post = (method, body) =>
fetch(`${api}/mixer/${method}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
const uri = 'mixer://room-42';
await post('startup', { uri, localStreamName: 'room-42' });
await post('add', { uri, remoteStreamName: 'alice' });
await post('add', { uri, remoteStreamName: 'bob' });
// when the meeting is over
await post('terminate', { uri });localStreamName names the mixed stream that viewers play; an optional streamLabel in add puts a caption over the participant’s picture.
The mixer places the pictures on its canvas by a layout. By default it is a grid: every participant gets a cell of the same size, and the grid rebuilds itself as streams join and leave. A zero padding grid fills the frame edge to edge, and one more variant crops every picture around its centre, where a speaker’s face usually is.
A layout of your own is described in XML, with fixed positions that streams take by name, or written as a Java class.
An XML layout is a folder of .mix files, one for each number of participants, named after it. Here is the grid above as four such files: a video element is a participant’s picture, and row and div elements place them on the canvas.
One picture in the middle of the canvas:
<?xml version="1.0" encoding="utf-8"?>
<body>
<row height="80%" align="CENTER">
<video width="100%" align="CENTER"/>
</row>
</body>Saved as 1_participants.mix.
Two pictures side by side, half the width each:
<?xml version="1.0" encoding="utf-8"?>
<body>
<row height="80%" align="CENTER">
<video width="50%"
align="INLINE_HORIZONTAL_CENTER"/>
<video width="50%"
align="INLINE_HORIZONTAL_CENTER"/>
</row>
</body>Saved as 2_participants.mix.
Two pictures in the top row, the third in the middle of the bottom one:
<?xml version="1.0" encoding="utf-8"?>
<body>
<row height="50%" align="INLINE_VERTICAL_CENTER">
<div width="50%" height="100%"
align="INLINE_HORIZONTAL_CENTER">
<video width="100%" align="CENTER"/>
</div>
<div width="50%" height="100%"
align="INLINE_HORIZONTAL_CENTER">
<video width="100%" align="CENTER"/>
</div>
</row>
<row height="50%" align="INLINE_VERTICAL_CENTER">
<video width="100%" align="CENTER"/>
</row>
</body>Saved as 3_participants.mix.
Two rows of two:
<?xml version="1.0" encoding="utf-8"?>
<body>
<row height="50%" align="INLINE_VERTICAL_CENTER">
<div width="50%" height="100%"
align="INLINE_HORIZONTAL_CENTER">
<video width="100%" align="CENTER"/>
</div>
<div width="50%" height="100%"
align="INLINE_HORIZONTAL_CENTER">
<video width="100%" align="CENTER"/>
</div>
</row>
<row height="50%" align="INLINE_VERTICAL_CENTER">
<div width="50%" height="100%"
align="INLINE_HORIZONTAL_CENTER">
<video width="100%" align="CENTER"/>
</div>
<div width="50%" height="100%"
align="INLINE_HORIZONTAL_CENTER">
<video width="100%" align="CENTER"/>
</div>
</row>
</body>Saved as 4_participants.mix. The folder is given to a mixer as mixerLayoutDir in /mixer/startup, or to every mixer with mixer_layout_dir.
When one of the streams is a screen share, it takes the main place: the other participants line up above, below, beside or around it, or float over it as picture in picture.
In a video conference everyone should hear the others, but not themselves. With MCU turned on, mixer_mcu_audio, the mixer makes, besides the common mixed stream, a stream for every participant with the sound of everyone in the room except them, so nobody hears the echo of their own voice.
Only the sound is mixed for every participant. The video mixer stays one for the whole room and shows all the participants in a grid.
With mixer_mcu_video turned on, every participant’s stream also carries the room’s video: the same grid for everyone, so a participant sees themselves in it too, but hears only the others.
Each participant publishes into the room and plays the stream the mixer makes for them, from any SDK:
In a browser, the Web SDK publishes the camera into the room and plays the stream made for the participant:
const room = 'room-42';
const login = 'alice';
session
.createStream({
name: `${login}#${room}`,
display: localPreview,
})
.publish();
session
.createStream({
name: `${room}-${login}${room}`,
display: conference,
})
.play();The mixer names the stream for Alice after the room and her stream, without the #: room-42-aliceroom-42, with the sound of everyone in the room except her.
On Android, a stream to publish and a stream to play, each named in its options:
String room = "room-42";
String login = "alice";
StreamOptions publishOptions =
new StreamOptions(login + "#" + room);
session.createStream(publishOptions).publish();
StreamOptions playOptions =
new StreamOptions(room + "-" + login + room);
session.createStream(playOptions).play();The same names as in the browser: alice#room-42 to publish, room-42-aliceroom-42 to play.
On iOS the same two streams, named in their options too:
let room = "room-42"
let login = "alice"
let publishOptions = FPWCSApi2StreamOptions()
publishOptions.name = "\(login)#\(room)"
publishOptions.display = localView
try session.createStream(publishOptions).publish()
let playOptions = FPWCSApi2StreamOptions()
playOptions.name = "\(room)-\(login)\(room)"
playOptions.display = remoteView
try session.createStream(playOptions).play()The same names as in the browser: alice#room-42 to publish, room-42-aliceroom-42 to play.
A mixer is created with /mixer/startup, and most of what it does is set there: the size and frame rate of its canvas, the layout, where a screen share goes and the voice activity frame.
All of it but the canvas size and frame rate can be changed later with /mixer/set_parameter, without stopping the mixer. The video bitrate and MCU are set on the server, for every mixer.
A participant’s sound and picture in the mix are changed on the fly with /mixer/setAudioVideo.
Let’s take a closer look at the available controls.
Every control above is a REST API parameter, or a server setting for the bitrate and MCU. With the post helper from the REST API example:
The canvas, the layout, the screen share, the stream names and the voice activity frame:
const layouts =
'com.flashphoner.media.mixer.video.presentation';
await post('startup', {
uri: 'mixer://room-42',
localStreamName: 'room-42',
mixerVideoWidth: 1280,
mixerVideoHeight: 720,
mixerVideoFps: 30,
mixerLayoutClass: `${layouts}.GridLayout`,
mixerDesktopAlign: 'TOP',
mixerVideoDesktopFullscreen: false,
mixerDisplayStreamName: false,
mixerVoiceActivity: true,
mixerAudioSilenceThreshold: -50,
});The same settings for a running mixer, the canvas size and frame rate aside, with the layouts package from the card before:
await post('set_parameter', {
uri: 'mixer://room-42',
mixerLayoutClass:
`${layouts}.CropNoPaddingGridLayout`,
mixerDesktopAlign: 'LEFT',
mixerVideoDesktopFullscreen: true,
mixerDisplayStreamName: true,
mixerVoiceActivity: false,
mixerAudioSilenceThreshold: -40,
});Turn Bob’s sound down to half and black out his picture:
await post('setAudioVideo', {
uri: 'mixer://room-42',
query: { streamsQuery: ['bob'] },
settings: { audioLevel: 50, videoMuted: true },
});query picks the streams by their names or by a pattern; the sound level is set in percent, from 0 to 100.
The video bitrate and MCU are not REST API parameters: they are set on the server, for every mixer:
# flashphoner.properties
mixer_video_bitrate_kbps=2000
mixer_mcu_audio=true
mixer_mcu_video=trueNo setup, no installation. Open the demo in two or more browser tabs, enter the same room name and different user names, and press Join: every participant publishes into the mixer, sees all of the room in one mixed stream and hears everyone but themselves.
The stream could not be played.
LLcast will deploy a dedicated Media Server just for you, ready to mix your streams. Publish over WebRTC, RTMP, SRT or RTSP, mix them into one stream by name or over the REST API, and play, record or push the result.
