obs-studio/docs/sphinx/backend-design.rst

OBS Studio Backend Design
=========================
The OBS Studio backend is powered by the library libobs.  Libobs
provides the main pipeline, the video/audio subsystems, and the general
framework for all plugins.


Libobs Plugin Objects
---------------------
Libobs is designed to be modular, where adding modules will add custom
functionality.  There are four libobs objects that you can make plugins
for:

- :ref:`plugins_sources` -- Sources are used to render video and/or
  audio on stream.  Things such as capturing displays/games/audio,
  playing a video, showing an image, or playing audio.  Sources can also
  be used to implement audio and video filters.

- :ref:`plugins_outputs` -- Outputs allow the ability to output the
  currently rendering audio/video.  Streaming and recording are two
  common examples of outputs, but not the only types of outputs.
  Outputs can receive the raw data or receive encoded data.

- :ref:`plugins_encoders` -- Encoders are OBS-specific implementations
  of video/audio encoders, which are used with outputs that use
  encoders.  x264, NVENC, Quicksync are examples of encoder
  implementations.

- :ref:`plugins_services` -- Services are custom implementations of
  streaming services, which are used with outputs that stream.  For
  example, you could have a custom implementation for streaming to
  Twitch, and another for YouTube to allow the ability to log in and use
  their APIs to do things such as get the RTMP servers or control the
  channel.

*(Author's note: the service API is incomplete as of this writing)*


Libobs Threads
--------------
There are three primary threads spawned by libobs on initialization:

- The obs_graphics_thread_ function used exclusively for rendering in
  `libobs/obs-video.c`_

- The video_thread_ function used exclusively for video encoding/output
  in `libobs/media-io/video-io.c`_

- The audio_thread_ function used for all audio
  processing/encoding/output in `libobs/media-io/audio-io.c`_

*(Author's note: obs_graphics_thread was originally named
obs_video_thread; it was renamed as of this writing to prevent confusion
with video_thread)*


.. _output_channels:

Output Channels
---------------
Rendering video or audio starts from output channels.  You assign a
source to an output channel via the :c:func:`obs_set_output_source()`
function.  The *channel* parameter can be any number from
0..(MAX_CHANNELS_-1).  You may initially think that this is how you
display multiple sources at once; however, sources are hierarchical.
Sources such as scenes or transitions can have multiple sub-sources, and
those sub-sources in turn can have sub-sources and so on (see
:ref:`displaying_sources` for more information).  Typically, you would
use scenes to draw multiple sources as a group with specific transforms
for each source, as a scene is just another type of source.  The
"channel" design allows for highly complex video presentation setups.
The OBS Studio front-end has yet to even fully utilize this back-end
design for its rendering, and currently only uses one output channel to
render one scene at a time.  It does however utilize additional channels
for things such as global audio sources which are set in audio settings.

*(Author's note: "Output channels" are not to be confused with output
objects or audio channels.  Output channels are used to set the sources
you want to output, and output objects are used for actually
streaming/recording/etc.)*


General Video Pipeline Overview
-------------------------------
The video graphics pipeline is run from two threads: a dedicated
graphics thread that renders preview displays as well as the final mix
(the obs_graphics_thread_ function in `libobs/obs-video.c`_), and a
dedicated thread specific to video encoding/output (the video_thread_
function in `libobs/media-io/video-io.c`_).

Sources assigned to output channels will be drawn from channels
0..(MAX_CHANNELS_-1).  They are drawn on to the final texture which will
be used for output `[1]`_.  Once all sources are drawn, the final
texture is converted to whatever format that libobs is set to (typically
a YUV format).  After being converted to the back-end video format, it's
then sent along with its timestamp to the current video handler,
`obs_core_video::video`_.

It then puts that raw frame in a queue of MAX_CACHE_SIZE_ in the `video
output handler`_.  A semaphore is posted, then the video-io thread will
process frames as it's able.  If the video frame queue is full, it will
duplicate the last frame in the queue in an attempt to reduce video
encoding complexity (and thus CPU usage) `[2]`_.  This is why you may
see frame skipping when the encoder can't keep up.  Frames are sent to
any raw outputs or video encoders that are currently active `[3]`_.

If it's sent to a video encoder object (`libobs/obs-encoder.c`_), it
encodes the frame and sends the encoded packet off to the outputs that
encoder is connected to (which can be multiple).  If the output takes
both encoded video/audio, it puts the packets in an interleave queue to
ensure encoded packets are sent in monotonic timestamp order `[4]`_.

The encoded packet or raw frame is then sent to the output.


General Audio Pipeline Overview
-------------------------------
The audio pipeline is run from a dedicated audio thread in the audio
handler (the `audio_thread`_ function in `libobs/media-io/audio-io.c`_);
assuming that AUDIO_OUTPUT_FRAMES_ is set to 1024, the audio thread
"ticks" (processes audio data) once every 1024 audio samples (around
every 21 millisecond intervals at 48khz), and calls the audio_callback_
function in `libobs/obs-audio.c`_ where most of the audio processing is
accomplished.

A source with audio will output its audio via the
obs_source_output_audio_ function, and that audio data will be appended
or inserted in to the circular buffer `obs_source::audio_input_buf`_.
If the sample rate or channel count does not match what the back-end is
set to, the audio is automatically remixed/resampled via swresample
`[5]`_.  Before insertion, audio data is also run through any audio
filters attached to the source `[6]`_.

Each audio tick, the audio thread takes a reference snapshot of the
audio source tree (stores references of all sources that output/process
audio) `[7]`_.  On each audio leaf (audio source), it takes the closest
audio (relative to the current audio thread timestamp) stored in the
circular buffer `obs_source::audio_input_buf`_, and puts it in
`obs_source::audio_output_buf`_.

Then, the audio samples stored in `obs_source::audio_output_buf`_ of the
leaves get sent through their parents in the source tree snapshot for
mixing or processing at each source node in the hierarchy `[8]`_.
Sources with multiple children such as scenes or transitions will
mix/process their children's audio themselves via the
`obs_source_info::audio_render`_ callback.  This allows, for example,
transitions to fade in the audio of one source and fade in the audio of
a new source when they're transitioning between two sources.  The mix or
processed audio data is then stored in `obs_source::audio_output_buf`_
of that node similarly, and the process is repeated until the audio
reaches the root nodes of the tree.

Finally, when the audio has reached the base of the snapshot tree, the
audio of all the sources in each output channel are mixed together for a
final mix `[9]`_.  That final mix is then sent to any raw outputs or
audio encoders that are currently active `[10]`_.

If it's sent to an audio encoder object (`libobs/obs-encoder.c`_), it
encodes the audio data and sends the encoded packet off to the outputs
that encoder is connected to (which can be multiple).  If the output
takes both encoded video/audio, it puts the packets in an interleave
queue to ensure encoded packets are sent in monotonic timestamp order
`[4]`_.

The encoded packet or raw audio data is then sent to the output.

.. _obs_graphics_thread: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-video.c#L588-L651
.. _libobs/obs-audio.c: https://github.com/obsproject/obs-studio/blob/master/libobs/obs-audio.c
.. _libobs/obs-video.c: https://github.com/obsproject/obs-studio/blob/master/libobs/obs-video.c
.. _video_thread: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L169-L195
.. _libobs/media-io/video-io.c: https://github.com/obsproject/obs-studio/blob/master/libobs/media-io/video-io.c
.. _video output handler: https://github.com/obsproject/obs-studio/blob/master/libobs/media-io/video-io.c
.. _audio_thread: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.c#L241-L282
.. _libobs/media-io/audio-io.c: https://github.com/obsproject/obs-studio/blob/master/libobs/media-io/audio-io.c
.. _MAX_CHANNELS: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-defs.h#L20-L21
.. _[1]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-video.c#L99-L129
.. _obs_core_video::video: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-internal.h#L250
.. _MAX_CACHE_SIZE: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L34
.. _[2]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L431-L434
.. _[3]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L115-L167
.. _libobs/obs-encoder.c: https://github.com/obsproject/obs-studio/blob/master/libobs/obs-encoder.c
.. _[4]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-output.c#L1382-L1439
.. _AUDIO_OUTPUT_FRAMES: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.h#L30
.. _audio_callback: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L367-L485
.. _obs_source_output_audio: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2578-L2608
.. _obs_source::audio_input_buf: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L1280-L1283
.. _[5]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2561-L2563
.. _[6]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2591
.. _[7]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L393-L415
.. _obs_source::audio_output_buf: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-internal.h#L580
.. _[8]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L417-L423
.. _obs_source_info::audio_render: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.h#L410-L412
.. _[9]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L436-L453
.. _[10]: https://github.com/obsproject/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.c#L144-L165