backend-design.rst 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194
  1. OBS Studio Backend Design
  2. =========================
  3. The OBS Studio backend is powered by the library libobs. Libobs
  4. provides the main pipeline, the video/audio subsystems, and the general
  5. framework for all plugins.
  6. Libobs Plugin Objects
  7. ---------------------
  8. Libobs is designed to be modular, where adding modules will add custom
  9. functionality. There are four libobs objects that you can make plugins
  10. for:
  11. - :ref:`plugins_sources` -- Sources are used to render video and/or
  12. audio on stream. Things such as capturing displays/games/audio,
  13. playing a video, showing an image, or playing audio. Sources can also
  14. be used to implement audio and video filters.
  15. - :ref:`plugins_outputs` -- Outputs allow the ability to output the
  16. currently rendering audio/video. Streaming and recording are two
  17. common examples of outputs, but not the only types of outputs.
  18. Outputs can receive the raw data or receive encoded data.
  19. - :ref:`plugins_encoders` -- Encoders are OBS-specific implementations
  20. of video/audio encoders, which are used with outputs that use
  21. encoders. x264, NVENC, Quicksync are examples of encoder
  22. implementations.
  23. - :ref:`plugins_services` -- Services are custom implementations of
  24. streaming services, which are used with outputs that stream. For
  25. example, you could have a custom implementation for streaming to
  26. Twitch, and another for YouTube to allow the ability to log in and use
  27. their APIs to do things such as get the RTMP servers or control the
  28. channel.
  29. *(Author's note: the service API is incomplete as of this writing)*
  30. Libobs Threads
  31. --------------
  32. There are three primary threads spawned by libobs on initialization:
  33. - The obs_graphics_thread_ function used exclusively for rendering in
  34. `libobs/obs-video.c`_
  35. - The video_thread_ function used exclusively for video encoding/output
  36. in `libobs/media-io/video-io.c`_
  37. - The audio_thread_ function used for all audio
  38. processing/encoding/output in `libobs/media-io/audio-io.c`_
  39. *(Author's note: obs_graphics_thread was originally named
  40. obs_video_thread; it was renamed as of this writing to prevent confusion
  41. with video_thread)*
  42. .. _output_channels:
  43. Output Channels
  44. ---------------
  45. Rendering video or audio starts from output channels. You assign a
  46. source to an output channel via the :c:func:`obs_set_output_source()`
  47. function. The *channel* parameter can be any number from
  48. 0..(MAX_CHANNELS_-1). You may initially think that this is how you
  49. display multiple sources at once; however, sources are hierarchical.
  50. Sources such as scenes or transitions can have multiple sub-sources, and
  51. those sub-sources in turn can have sub-sources and so on (see
  52. :ref:`displaying_sources` for more information). Typically, you would
  53. use scenes to draw multiple sources as a group with specific transforms
  54. for each source, as a scene is just another type of source. The
  55. "channel" design allows for highly complex video presentation setups.
  56. The OBS Studio front-end has yet to even fully utilize this back-end
  57. design for its rendering, and currently only uses one output channel to
  58. render one scene at a time. It does however utilize additional channels
  59. for things such as global audio sources which are set in audio settings.
  60. *(Author's note: "Output channels" are not to be confused with output
  61. objects or audio channels. Output channels are used to set the sources
  62. you want to output, and output objects are used for actually
  63. streaming/recording/etc.)*
  64. General Video Pipeline Overview
  65. -------------------------------
  66. The video graphics pipeline is run from two threads: a dedicated
  67. graphics thread that renders preview displays as well as the final mix
  68. (the obs_graphics_thread_ function in `libobs/obs-video.c`_), and a
  69. dedicated thread specific to video encoding/output (the video_thread_
  70. function in `libobs/media-io/video-io.c`_).
  71. Sources assigned to output channels will be drawn from channels
  72. 0..(MAX_CHANNELS_-1). They are drawn on to the final texture which will
  73. be used for output `[1]`_. Once all sources are drawn, the final
  74. texture is converted to whatever format that libobs is set to (typically
  75. a YUV format). After being converted to the back-end video format, it's
  76. then sent along with its timestamp to the current video handler,
  77. `obs_core_video::video`_.
  78. It then puts that raw frame in a queue of MAX_CACHE_SIZE_ in the `video
  79. output handler`_. A semaphore is posted, then the video-io thread will
  80. process frames as it's able. If the video frame queue is full, it will
  81. duplicate the last frame in the queue in an attempt to reduce video
  82. encoding complexity (and thus CPU usage) `[2]`_. This is why you may
  83. see frame skipping when the encoder can't keep up. Frames are sent to
  84. any raw outputs or video encoders that are currently active `[3]`_.
  85. If it's sent to a video encoder object (`libobs/obs-encoder.c`_), it
  86. encodes the frame and sends the encoded packet off to the outputs that
  87. encoder is connected to (which can be multiple). If the output takes
  88. both encoded video/audio, it puts the packets in an interleave queue to
  89. ensure encoded packets are sent in monotonic timestamp order `[4]`_.
  90. The encoded packet or raw frame is then sent to the output.
  91. General Audio Pipeline Overview
  92. -------------------------------
  93. The audio pipeline is run from a dedicated audio thread in the audio
  94. handler (the `audio_thread`_ function in `libobs/media-io/audio-io.c`_);
  95. assuming that AUDIO_OUTPUT_FRAMES_ is set to 1024, the audio thread
  96. "ticks" (processes audio data) once every 1024 audio samples (around
  97. every 21 millisecond intervals at 48khz), and calls the audio_callback_
  98. function in `libobs/obs-audio.c`_ where most of the audio processing is
  99. accomplished.
  100. A source with audio will output its audio via the
  101. obs_source_output_audio_ function, and that audio data will be appended
  102. or inserted in to the circular buffer `obs_source::audio_input_buf`_.
  103. If the sample rate or channel count does not match what the back-end is
  104. set to, the audio is automatically remixed/resampled via swresample
  105. `[5]`_. Before insertion, audio data is also run through any audio
  106. filters attached to the source `[6]`_.
  107. Each audio tick, the audio thread takes a reference snapshot of the
  108. audio source tree (stores references of all sources that output/process
  109. audio) `[7]`_. On each audio leaf (audio source), it takes the closest
  110. audio (relative to the current audio thread timestamp) stored in the
  111. circular buffer `obs_source::audio_input_buf`_, and puts it in
  112. `obs_source::audio_output_buf`_.
  113. Then, the audio samples stored in `obs_source::audio_output_buf`_ of the
  114. leaves get sent through their parents in the source tree snapshot for
  115. mixing or processing at each source node in the hierarchy `[8]`_.
  116. Sources with multiple children such as scenes or transitions will
  117. mix/process their children's audio themselves via the
  118. `obs_source_info::audio_render`_ callback. This allows, for example,
  119. transitions to fade in the audio of one source and fade in the audio of
  120. a new source when they're transitioning between two sources. The mix or
  121. processed audio data is then stored in `obs_source::audio_output_buf`_
  122. of that node similarly, and the process is repeated until the audio
  123. reaches the root nodes of the tree.
  124. Finally, when the audio has reached the base of the snapshot tree, the
  125. audio of all the sources in each output channel are mixed together for a
  126. final mix `[9]`_. That final mix is then sent to any raw outputs or
  127. audio encoders that are currently active `[10]`_.
  128. If it's sent to an audio encoder object (`libobs/obs-encoder.c`_), it
  129. encodes the audio data and sends the encoded packet off to the outputs
  130. that encoder is connected to (which can be multiple). If the output
  131. takes both encoded video/audio, it puts the packets in an interleave
  132. queue to ensure encoded packets are sent in monotonic timestamp order
  133. `[4]`_.
  134. The encoded packet or raw audio data is then sent to the output.
  135. .. _obs_graphics_thread: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-video.c#L588-L651
  136. .. _libobs/obs-audio.c: https://github.com/jp9000/obs-studio/blob/master/libobs/obs-audio.c
  137. .. _libobs/obs-video.c: https://github.com/jp9000/obs-studio/blob/master/libobs/obs-video.c
  138. .. _video_thread: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L169-L195
  139. .. _libobs/media-io/video-io.c: https://github.com/jp9000/obs-studio/blob/master/libobs/media-io/video-io.c
  140. .. _video output handler: https://github.com/jp9000/obs-studio/blob/master/libobs/media-io/video-io.c
  141. .. _audio_thread: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.c#L241-L282
  142. .. _libobs/media-io/audio-io.c: https://github.com/jp9000/obs-studio/blob/master/libobs/media-io/audio-io.c
  143. .. _MAX_CHANNELS: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-defs.h#L20-L21
  144. .. _[1]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-video.c#L99-L129
  145. .. _obs_core_video::video: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-internal.h#L250
  146. .. _MAX_CACHE_SIZE: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L34
  147. .. _[2]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L431-L434
  148. .. _[3]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L115-L167
  149. .. _libobs/obs-encoder.c: https://github.com/jp9000/obs-studio/blob/master/libobs/obs-encoder.c
  150. .. _[4]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-output.c#L1382-L1439
  151. .. _AUDIO_OUTPUT_FRAMES: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.h#L30
  152. .. _audio_callback: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L367-L485
  153. .. _obs_source_output_audio: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2578-L2608
  154. .. _obs_source::audio_input_buf: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L1280-L1283
  155. .. _[5]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2561-L2563
  156. .. _[6]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2591
  157. .. _[7]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L393-L415
  158. .. _obs_source::audio_output_buf: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-internal.h#L580
  159. .. _[8]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L417-L423
  160. .. _obs_source_info::audio_render: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.h#L410-L412
  161. .. _[9]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L436-L453
  162. .. _[10]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.c#L144-L165