123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194 |
- OBS Studio Backend Design
- =========================
- The OBS Studio backend is powered by the library libobs. Libobs
- provides the main pipeline, the video/audio subsystems, and the general
- framework for all plugins.
- Libobs Plugin Objects
- ---------------------
- Libobs is designed to be modular, where adding modules will add custom
- functionality. There are four libobs objects that you can make plugins
- for:
- - :ref:`plugins_sources` -- Sources are used to render video and/or
- audio on stream. Things such as capturing displays/games/audio,
- playing a video, showing an image, or playing audio. Sources can also
- be used to implement audio and video filters.
- - :ref:`plugins_outputs` -- Outputs allow the ability to output the
- currently rendering audio/video. Streaming and recording are two
- common examples of outputs, but not the only types of outputs.
- Outputs can receive the raw data or receive encoded data.
- - :ref:`plugins_encoders` -- Encoders are OBS-specific implementations
- of video/audio encoders, which are used with outputs that use
- encoders. x264, NVENC, Quicksync are examples of encoder
- implementations.
- - :ref:`plugins_services` -- Services are custom implementations of
- streaming services, which are used with outputs that stream. For
- example, you could have a custom implementation for streaming to
- Twitch, and another for YouTube to allow the ability to log in and use
- their APIs to do things such as get the RTMP servers or control the
- channel.
-
- *(Author's note: the service API is incomplete as of this writing)*
- Libobs Threads
- --------------
- There are three primary threads spawned by libobs on initialization:
- - The obs_graphics_thread_ function used exclusively for rendering in
- `libobs/obs-video.c`_
- - The video_thread_ function used exclusively for video encoding/output
- in `libobs/media-io/video-io.c`_
- - The audio_thread_ function used for all audio
- processing/encoding/output in `libobs/media-io/audio-io.c`_
- *(Author's note: obs_graphics_thread was originally named
- obs_video_thread; it was renamed as of this writing to prevent confusion
- with video_thread)*
- .. _output_channels:
- Output Channels
- ---------------
- Rendering video or audio starts from output channels. You assign a
- source to an output channel via the :c:func:`obs_set_output_source()`
- function. The *channel* parameter can be any number from
- 0..(MAX_CHANNELS_-1). You may initially think that this is how you
- display multiple sources at once; however, sources are hierarchical.
- Sources such as scenes or transitions can have multiple sub-sources, and
- those sub-sources in turn can have sub-sources and so on (see
- :ref:`displaying_sources` for more information). Typically, you would
- use scenes to draw multiple sources as a group with specific transforms
- for each source, as a scene is just another type of source. The
- "channel" design allows for highly complex video presentation setups.
- The OBS Studio front-end has yet to even fully utilize this back-end
- design for its rendering, and currently only uses one output channel to
- render one scene at a time. It does however utilize additional channels
- for things such as global audio sources which are set in audio settings.
- *(Author's note: "Output channels" are not to be confused with output
- objects or audio channels. Output channels are used to set the sources
- you want to output, and output objects are used for actually
- streaming/recording/etc.)*
- General Video Pipeline Overview
- -------------------------------
- The video graphics pipeline is run from two threads: a dedicated
- graphics thread that renders preview displays as well as the final mix
- (the obs_graphics_thread_ function in `libobs/obs-video.c`_), and a
- dedicated thread specific to video encoding/output (the video_thread_
- function in `libobs/media-io/video-io.c`_).
- Sources assigned to output channels will be drawn from channels
- 0..(MAX_CHANNELS_-1). They are drawn on to the final texture which will
- be used for output `[1]`_. Once all sources are drawn, the final
- texture is converted to whatever format that libobs is set to (typically
- a YUV format). After being converted to the back-end video format, it's
- then sent along with its timestamp to the current video handler,
- `obs_core_video::video`_.
- It then puts that raw frame in a queue of MAX_CACHE_SIZE_ in the `video
- output handler`_. A semaphore is posted, then the video-io thread will
- process frames as it's able. If the video frame queue is full, it will
- duplicate the last frame in the queue in an attempt to reduce video
- encoding complexity (and thus CPU usage) `[2]`_. This is why you may
- see frame skipping when the encoder can't keep up. Frames are sent to
- any raw outputs or video encoders that are currently active `[3]`_.
- If it's sent to a video encoder object (`libobs/obs-encoder.c`_), it
- encodes the frame and sends the encoded packet off to the outputs that
- encoder is connected to (which can be multiple). If the output takes
- both encoded video/audio, it puts the packets in an interleave queue to
- ensure encoded packets are sent in monotonic timestamp order `[4]`_.
- The encoded packet or raw frame is then sent to the output.
- General Audio Pipeline Overview
- -------------------------------
- The audio pipeline is run from a dedicated audio thread in the audio
- handler (the `audio_thread`_ function in `libobs/media-io/audio-io.c`_);
- assuming that AUDIO_OUTPUT_FRAMES_ is set to 1024, the audio thread
- "ticks" (processes audio data) once every 1024 audio samples (around
- every 21 millisecond intervals at 48khz), and calls the audio_callback_
- function in `libobs/obs-audio.c`_ where most of the audio processing is
- accomplished.
- A source with audio will output its audio via the
- obs_source_output_audio_ function, and that audio data will be appended
- or inserted in to the circular buffer `obs_source::audio_input_buf`_.
- If the sample rate or channel count does not match what the back-end is
- set to, the audio is automatically remixed/resampled via swresample
- `[5]`_. Before insertion, audio data is also run through any audio
- filters attached to the source `[6]`_.
- Each audio tick, the audio thread takes a reference snapshot of the
- audio source tree (stores references of all sources that output/process
- audio) `[7]`_. On each audio leaf (audio source), it takes the closest
- audio (relative to the current audio thread timestamp) stored in the
- circular buffer `obs_source::audio_input_buf`_, and puts it in
- `obs_source::audio_output_buf`_.
- Then, the audio samples stored in `obs_source::audio_output_buf`_ of the
- leaves get sent through their parents in the source tree snapshot for
- mixing or processing at each source node in the hierarchy `[8]`_.
- Sources with multiple children such as scenes or transitions will
- mix/process their children's audio themselves via the
- `obs_source_info::audio_render`_ callback. This allows, for example,
- transitions to fade in the audio of one source and fade in the audio of
- a new source when they're transitioning between two sources. The mix or
- processed audio data is then stored in `obs_source::audio_output_buf`_
- of that node similarly, and the process is repeated until the audio
- reaches the root nodes of the tree.
- Finally, when the audio has reached the base of the snapshot tree, the
- audio of all the sources in each output channel are mixed together for a
- final mix `[9]`_. That final mix is then sent to any raw outputs or
- audio encoders that are currently active `[10]`_.
- If it's sent to an audio encoder object (`libobs/obs-encoder.c`_), it
- encodes the audio data and sends the encoded packet off to the outputs
- that encoder is connected to (which can be multiple). If the output
- takes both encoded video/audio, it puts the packets in an interleave
- queue to ensure encoded packets are sent in monotonic timestamp order
- `[4]`_.
- The encoded packet or raw audio data is then sent to the output.
- .. _obs_graphics_thread: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-video.c#L588-L651
- .. _libobs/obs-audio.c: https://github.com/jp9000/obs-studio/blob/master/libobs/obs-audio.c
- .. _libobs/obs-video.c: https://github.com/jp9000/obs-studio/blob/master/libobs/obs-video.c
- .. _video_thread: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L169-L195
- .. _libobs/media-io/video-io.c: https://github.com/jp9000/obs-studio/blob/master/libobs/media-io/video-io.c
- .. _video output handler: https://github.com/jp9000/obs-studio/blob/master/libobs/media-io/video-io.c
- .. _audio_thread: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.c#L241-L282
- .. _libobs/media-io/audio-io.c: https://github.com/jp9000/obs-studio/blob/master/libobs/media-io/audio-io.c
- .. _MAX_CHANNELS: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-defs.h#L20-L21
- .. _[1]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-video.c#L99-L129
- .. _obs_core_video::video: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-internal.h#L250
- .. _MAX_CACHE_SIZE: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L34
- .. _[2]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L431-L434
- .. _[3]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L115-L167
- .. _libobs/obs-encoder.c: https://github.com/jp9000/obs-studio/blob/master/libobs/obs-encoder.c
- .. _[4]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-output.c#L1382-L1439
- .. _AUDIO_OUTPUT_FRAMES: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.h#L30
- .. _audio_callback: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L367-L485
- .. _obs_source_output_audio: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2578-L2608
- .. _obs_source::audio_input_buf: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L1280-L1283
- .. _[5]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2561-L2563
- .. _[6]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2591
- .. _[7]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L393-L415
- .. _obs_source::audio_output_buf: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-internal.h#L580
- .. _[8]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L417-L423
- .. _obs_source_info::audio_render: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.h#L410-L412
- .. _[9]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L436-L453
- .. _[10]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.c#L144-L165
|