LMixer Reference Manual & Multi-Layer Composition Guide

The LMixer class is the multi-layer video compositor and audio mixer engine of the now2sdk framework, derived from the LObject class. It allows developers to composit multiple video layers (from sources like LFile or LLive), static image overlays, and dynamic transparent vector overlays (LCharacter) into a single output stream.

Under the hood, LMixer utilizes a headless offscreen OpenGL context (QOpenGLContext and QOffscreenSurface) to render composite frames on the GPU using GLSL shaders. This architecture ensures high-performance blending, scaling, cropping, border styling, and rounded corners without consuming CPU cycles, with a software CPU fallback available if OpenGL initialization fails.

1. Multi-Layer Composition Architecture

graph TD
    A[LMixer Composer Loop] --> B{Headless OpenGL Available?}
    B -->|Yes| C[Offscreen OpenGL FBO Renderer]
    B -->|No| D[CPU Software Compositor Fallback]
    
    C --> E[GLSL Fragment Shader Processing]
    E -->|1. Alpha Opacity Blending| F[Composition Outputs]
    E -->|2. Crop Masking| F
    E -->|3. Border Borders Rendering| F
    E -->|4. Rounded Corner Masking| F
    
    H[MixerSink per Layer] -->|Resample to 48kHz S16 Stereo| I[Synchronized Audio Mixer]
    I -->|Summed Audio Bus| F
    
    F --> J[distributeVideoFrame & distributeAudioFrame]

Key Composition Features:

• GLSL Shader Effects: Dynamic alpha transparency blending, border overlays, rectangular crop limits, and custom corner rounding are calculated per pixel inside the fragment shader for high rendering performance.
• Z-Index Layer Sorting: Layers are sorted and composited sequentially based on their zIndex attribute. Higher indices are rendered on top of lower ones.
• Audio Bus Mixing: Each layer has a built-in MixerSink that captures incoming audio, resamples it to 48kHz Stereo S16 PCM, adjusts channel gain, and mixes the channels using a synchronous PTS-aligned sum-bus before pushing it to downstream audio devices.

2. API Reference & Public Methods

void setVideoFormat(const videoFormatProps& props);
void getVideoFormat(videoFormatProps& props);

Sets or retrieves the target output video dimensions, standard frame rate, and color space format for the mixed stream.

void setAudioFormat(const audioFormatProps& props);
void getAudioFormat(audioFormatProps& props);

Sets or retrieves the target output audio properties (bits per sample, sampling rate, and channels).

LLayerItem* addLayer(const std::string& layerID, LObject* obj, 
                     int zIndex = 0,
                     float posX = 0.0f, float posY = 0.0f, 
                     int alpha = 255, 
                     float width = -1.0f, float height = -1.0f,
                     float cropTop = 0.0f, float cropBottom = 0.0f, float cropLeft = 0.0f, float cropRight = 0.0f,
                     int borderSize = 0, const std::string& borderColor = "#FFFFFF",
                     bool audio = false, int gain = 100);

Adds a dynamic streaming source layer (e.g. LFile, LLive, or LCharacter).

• Returns a pointer to the newly created or updated LLayerItem descriptor.
• If a layer with layerID already exists, it updates its properties without interrupting the stream.

LLayerItem* addImage(const std::string& layerID, const std::string& filePath,
                     int zIndex = 0,
                     float posX = 0.0f, float posY = 0.0f,
                     int alpha = 255,
                     float width = -1.0f, float height = -1.0f,
                     float cropTop = 0.0f, float cropBottom = 0.0f, float cropLeft = 0.0f, float cropRight = 0.0f,
                     int borderSize = 0, const std::string& borderColor = "#FFFFFF");

Loads a static image from filePath via FFmpeg, converts it into a transparent YUVA420P buffer, and overlays it as a layer.

LLayerItem* getLayer(const std::string& layerID);

Returns a pointer to the LLayerItem configuration matching layerID. Returns nullptr if not found.

void removeLayer(const std::string& layerID);

Removes the specified layer from the composer and releases its texture and resampler resources.

void setLayerBorderRadius(const std::string& layerID, int radiusPercent);

Applies corner rounding to the target layer. radiusPercent ranges from 0 (sharp rectangle) to 100 (circular mask).

int getLayerCount() const;

Returns the total number of layers registered in the mixer.

LLayerItem* getLayerByIndex(int index);

Iterates and returns a layer descriptor by its internal array index.

3. Layer Properties & Configurations

Every layer added to LMixer is described by the LLayerItem class handle. Modifying these fields directly updates the composite layout on the very next render cycle:

4. Qt Picture-in-Picture (PiP) Composition Example

This implementation (adapted from the Mixer Example mainwindow.cpp) shows how to configure a multi-layer composition in Qt. It sets up:

1. Layer 0: A full-screen player stream (LFile1) with audio enabled.
2. Layer 1: A Picture-in-Picture player stream (LFile2) with audio muted and a custom white border.
3. Layer 2: A transparent character overlay (LCharacter) displaying tickers and logos.
4. Dynamic Binding: Connecting Qt UI sliders to update layer coordinates and opacity in real time.

#include "mainwindow.h"
#include "./ui_mainwindow.h"
#include "LMixer.h"
#include "LFile.h"
#include "LCharacter.h"
#include "LFormat.h"

MainWindow::MainWindow(QWidget *parent)
    : QMainWindow(parent),
      ui(new Ui::MainWindow),
      m_lFile1(new LFile()),
      m_lFile2(new LFile()),
      m_lMixer(new LMixer()),
      m_lCG(new LCharacter()),
      m_lPreviewMixer(new LPreview()),
      m_uiTimer(new QTimer(this))
{
    ui->setupUi(this);

    // 1. Configure the Composer Resolution (1080p50 Broadcast Standard)
    videoFormatProps mixerVideoTarget;
    mixerVideoTarget.setVideoFormat = vF::HD1080_50p;
    m_lMixer->setVideoFormat(mixerVideoTarget);

    audioFormatProps mixerAudioTarget;
    mixerAudioTarget.setAudioFormat = aF::_16B_48K_2CH;
    m_lMixer->setAudioFormat(mixerAudioTarget);

    // 2. Add Layer 0: Full Screen Background video
    m_file1Layer = m_lMixer->addLayer("file1_background", m_lFile1, 0, 0.0f, 0.0f, 255, 1920.0f, 1080.0f);
    if (m_file1Layer) {
        m_file1Layer->audio = true;  // Keep background audio active
        m_file1Layer->gain = 90;     // Set audio gain to 90%
    }

    // 3. Add Layer 1: Overlay Picture-in-Picture (PiP) video
    m_file2Layer = m_lMixer->addLayer("file2_pip", m_lFile2, 1, 1200.0f, 100.0f, 255, 640.0f, 360.0f);
    if (m_file2Layer) {
        m_file2Layer->audio = false;            // Mute PiP to avoid audio clash
        m_file2Layer->borderColor = "#FFFFFF";  // Draw a high-contrast white border
        m_file2Layer->borderSize = 6;           // 6px border thickness
        m_file2Layer->borderAlpha = 230;        // Slightly transparent border
        m_lMixer->setLayerBorderRadius("file2_pip", 15); // Apply 15% rounded corners
    }

    // 4. Add Layer 2: Cairo Graphics & CG Overlay
    m_lMixer->addLayer("cg_overlay", m_lCG, 2, 0.0f, 0.0f, 255, 1920.0f, 1080.0f);

    // 5. Connect Mixer output to preview widget
    m_lPreviewMixer->setProps("ui_framework", "qt");
    m_lPreviewMixer->previewEnable(ui->mixerPreviewWidget, true, true);
    m_lPreviewMixer->previewObject(m_lMixer);

    // 6. Connect Qt sliders to layout controls
    connect(ui->pipXSlider, &QSlider::valueChanged, this, &MainWindow::onPipLayoutChanged);
    connect(ui->pipYSlider, &QSlider::valueChanged, this, &MainWindow::onPipLayoutChanged);
    connect(ui->pipWSlider, &QSlider::valueChanged, this, &MainWindow::onPipLayoutChanged);
    connect(ui->pipHSlider, &QSlider::valueChanged, this, &MainWindow::onPipLayoutChanged);
    connect(ui->pipAlphaSlider, &QSlider::valueChanged, this, &MainWindow::onPipLayoutChanged);

    // Start UI update timer (50ms)
    connect(m_uiTimer, &QTimer::timeout, this, &MainWindow::updateUI);
    m_uiTimer->start(50);
}

MainWindow::~MainWindow() {
    m_lFile1->stop();
    m_lFile2->stop();
    m_lMixer->removeLayer("file1_background");
    m_lMixer->removeLayer("file2_pip");
    m_lMixer->removeLayer("cg_overlay");
    
    delete m_lPreviewMixer;
    delete m_lFile1;
    delete m_lFile2;
    delete m_lMixer;
    delete m_lCG;
    delete ui;
}

// ─── Reactive PiP Layout Adjustment ───
void MainWindow::onPipLayoutChanged() {
    int x = ui->pipXSlider->value();
    int y = ui->pipYSlider->value();
    int w = ui->pipWSlider->value();
    int h = ui->pipHSlider->value();
    int alpha = ui->pipAlphaSlider->value();

    // Directly modify LLayerItem properties on-the-fly (updates on next render)
    if (m_file2Layer) {
        m_file2Layer->posX = static_cast<float>(x);
        m_file2Layer->posY = static_cast<float>(y);
        m_file2Layer->width = static_cast<float>(w);
        m_file2Layer->height = static_cast<float>(h);
        m_file2Layer->alpha = alpha;
    }
}

void MainWindow::updateUI() {
    ui->mixerPreviewWidget->update();
}

5. Chain Pull-Rendering & Thread Synchronization

In now2sdk v0.0.2, LMixer supports Chain Pull-Rendering directly from pull-capable layers (like LSwitcher).

When a pull-capable source is added using addLayer(...):

• MixerSink registers itself as a consumer on the source object via addPullConsumer().
• During the compositor rendering loop, instead of returning the last cached frame, MixerSink::getLatestFrame() calls renderVideoFrame() on the source object to trigger synchronized rendering.
• This completely removes the need for asynchronous thread rendering on the sub-component side, eliminating judder and phase lag.
• When in pull mode, video clones are bypassed at the sink level, dramatically lowering CPU memory copy overhead.