LFrame Reference Manual & CPU Frame Extraction Guide

The LFrame class is the raw frame extraction and CPU memory grabber engine of the now2sdk framework, derived from the LSink base class. It is designed to capture, format-convert, and copy real-time audio and video frames directly into host system memory (CPU RAM) for custom application-level processing.

It is commonly used for:

1. Computer Vision & AI: Injecting video frames into OpenCV, TensorFlow, or PyTorch pipelines.
2. Custom Overlays & Effects: Modifying raw pixel buffers or audio sample blocks in real time.
3. Third-Party Networking Integration: Forwarding raw frames to custom WebRTC, WebSockets, or proprietary streaming protocols.

1. Frame Extraction Architecture

graph TD
    A[Program Feed Source - LObject] -->|pushVideoFrame / pushAudioFrame| B(LFrame Sink)
    
    subgraph Video Extraction Pipeline
        B -->|Convert to RGBA/BGRA/YUV420p| C[Internal SwScale Normalization]
        C -->|Lock-Free Ring Buffer| D[Video Frame Queue]
        D -->|popVideoFrame API| E[User C++ Application Thread]
    end
    
    subgraph Audio Extraction Pipeline
        B -->|Asynchronous Queue| F[Audio Buffer Queue]
        F -->|popAudioFrame API| G[User C++ Application Thread]
    end

Ingestion & Queue Details:

• Format Conversion SwScale: Raw incoming GPU textures or hardware formats (like UYVY or NV12) are converted to accessible host layouts (such as RGBA, BGRA, or planar YUV420p) using optimized SwScale routines.
• Lock-Free Queue: To ensure the main playout or rendering thread is never blocked, LFrame uses separate lock-free async queues for audio and video frames.
• Auto-Drop Policy: If the application thread falls behind and doesn't pop frames fast enough, LFrame drops old frames automatically to keep latency low. The queue depth and dropped frame counts can be queried dynamically.

2. API Reference & Public Methods

📌 Sink Configuration & Ingestion

void setSource(LObject* source);

Registers LFrame as a sink to the program stream source (e.g. LMixer or LLive). If another source was previously registered, it detaches automatically first.

void setProps(const std::string& key, const std::string& value);

Sets dynamic configuration properties at runtime. Parameter options are detailed in Section 3.

void setEnabled(bool enabled);

Starts or stops the frame extraction queue.

📌 Frame Popping & Queue Access

struct VideoFrame {
    uint8_t* data;
    int width;
    int height;
    int rowBytes;
    int64_t pts; // Presentation timestamp in milliseconds
};

bool popVideoFrame(VideoFrame& frame, int timeoutMs = 0);

Pops the oldest video frame from the queue. Returns true if a frame was successfully retrieved. If timeoutMs > 0, blocks up to that duration waiting for a frame.

struct AudioFrame {
    uint8_t* data;
    int sampleRate;
    int channels;
    int sampleCount;
    int64_t pts; // Presentation timestamp in milliseconds
};

bool popAudioFrame(AudioFrame& frame, int timeoutMs = 0);

Pops the oldest audio frame from the queue. Returns true if a frame was successfully retrieved.

📌 Queue Stats

struct FrameStats {
    int videoQueueSize;
    int audioQueueSize;
    int droppedVideoFrames;
    int droppedAudioFrames;
};

void getStats(FrameStats& stats);

Queries current queue statuses, pending frame counts, and total dropped frames.

3. Properties & Configuration Options

All options listed below are configured at runtime via the setProps(const std::string& key, const std::string& value) interface:

4. C++ Frame Extraction Example

This example demonstrates how to attach an LFrame sink to a playout source, configure its format to BGRA, and extract raw pixel data in a processing thread.

#include "LFile.h"
#include "LFrame.h"
#include <thread>
#include <iostream>

void processingThread(LFrame* grabber) {
    while (true) {
        LFrame::VideoFrame vFrame;
        // Pop frame with 100ms timeout
        if (grabber->popVideoFrame(vFrame, 100)) {
            // Process raw pixels in vFrame.data (vFrame.width x vFrame.height)
            std::cout << "Grabbed Video Frame PTS: " << vFrame.pts 
                      << " Size: " << vFrame.width << "x" << vFrame.height << std::endl;
        }

        LFrame::AudioFrame aFrame;
        if (grabber->popAudioFrame(aFrame, 10)) {
            // Process raw audio samples in aFrame.data
        }
    }
}

int main() {
    LFile* player = new LFile();
    LFrame* grabber = new LFrame();

    // Attach frame grabber to file player
    grabber->setSource(player);

    // Configure properties
    grabber->setProps("video_format", "BGRA");
    grabber->setProps("max_video_queue", "5");
    grabber->setEnabled(true);

    // Start playback
    player->setProps("file", "test_broadcast.mp4");
    player->play();

    // Start application processing thread
    std::thread worker(processingThread, grabber);
    worker.join();

    delete grabber;
    delete player;
    return 0;
}