Skip to main content

  • Core Products

  • Add-Ons

  • Extensions Marketplace

How to solve SEI-related issues?

The Agora SEI​

By default, Agora adds the encoding information of the current video to the transcoded H264/H265 SEI (Supplemental Enhancement Information) during Media Push. The encoding information is a JSON string. The following is the sample code:


_32
{
_32
"canvas": {
_32
"w": 640,
_32
"h": 360,
_32
"bgnd": "#000000"
_32
},
_32
"regions": [
_32
{
_32
"uid": 1,
_32
"alpha": 1.0,
_32
"zorder": 1,
_32
"volume": 50,
_32
"x": 0,
_32
"y": 0,
_32
"w": 320,
_32
"h": 360
_32
},
_32
{
_32
"uid": 2,
_32
"alpha": 1.0,
_32
"zOrder": 1,
_32
"volume": 89,
_32
"x": 320,
_32
"y": 0,
_32
"w": 320,
_32
"h": 360
_32
}
_32
],
_32
"ver": "20190611",
_32
"ts": 1535385600000,
_32
"app_data": ""
_32
}

The definition of the parameters:

ParameterDescription
canvasThe canvas information. It contains the following properties:
  • w: Width (pixel) of the canvas. It corresponds to the width member in the LiveTranscoding class.

  • h: Height (pixel) of the canvas. It corresponds to the height member in the LiveTranscoding class.

  • bgnd: The background color (RGB) of the canvas, represented by a hexademical string. It corresponds to the backgoundColor member in the LiveTranscoding class.
    • regionsThe layout information of the host. It corresponds to the transcodingUsers member in the LiveTranscoding class. It contains the following properties:
  • suid: (Optional)The string user account of the host in this region. This parameter applies to scenarios where string user accounts are used to identify the host.

  • uid: UID of the host in this region. It corresponds to the uid member in the TranscodingUser class.

  • alpha: The transparency of the video frame of the host. The value range is [0.0, 1.0]. It corresponds to the alpha member in TranscodingUser .

  • zorder: The layout position of the video frame of the host. The value range is [0, 100]. It corresponds to the zOrder member in TranscodingUser.

  • volume: The volume (dB) of the host. The value range is [0, 100].

  • x: The horizontal position of the video frame of the broadacaster from the top left corner of the Media Push. It corresponds to the x member in TranscodingUser.

  • y: The vertical position of the video frame of the host from the top left corner of the Media Push. It corresponds to the y member in TranscodingUser.

  • w: Width (pixel) of the video frame of the host. It corresponds to the width member in TranscodingUser.

  • h: Height (pixel) of the video frame of the host. It corresponds to the height member in TranscodingUser.
    • verThe version of the SEI protocol. The current version is 20190611.
    tsTimestamp (ms) of the current encoding information.
    app_dataExtra user-defined information. It corresponds to the transcodingExtraInfo member in the LiveTranscoding class.

    The structure of SEI​

    The SEI information is as follows:


    _12
    0000 0664bd7b 22617070 5f646174 61223a22 .d.{"app_data":"
    _12
    0010 222c2263 616e7661 73223a7b 2262676e ","canvas":{"bgn
    _12
    0020 64223a22 23666666 66666622 2c226822 d":"#ffffff","h"
    _12
    0030 3a363430 2c227722 3a333630 7d2c2272 :640,"w":360},"r
    _12
    0040 6567696f 6e73223a 5b7b2261 6c706861 egions":[{"alpha
    _12
    0050 223a3235 352c2268 223a3634 302c2275 ":255,"h":640,"u
    _12
    0060 6964223a 33313031 32373137 39312c22 id":3101271791,"
    _12
    0070 766f6c75 6d65223a 32382c22 77223a33 volume":28,"w":3
    _12
    0080 36302c22 78223a30 2c227922 3a302c22 60,"x":0,"y":0,"
    _12
    0090 7a6f7264 6572223a 317d5d2c 22747322 zorder":1}],"ts"
    _12
    00a0 3a313533 37393630 32333537 38332c22 :1537960235783,"
    _12
    00b0 76657222 3a223230 31383038 3238227d ver":"20190611"}

    In which:

    • 06: The SEI frame.
    • 64: The SEI frame type defined by the user. Here we define it as 100.
    • bd: The length of the SEI frame. The following are some sample calculations rendered in decimal and hexadecimal:
      • If the length is 922, because 922 can be divided by 255 (0xff) three times and the remainder is 157 (0x9d), then bd is ffffff9d.
      • If the length is 572, because 572 can be divided by 255 (0xff) two times and the remainder is 62 (0x3e), then bd is ffff3e.
      • If the length is 234, because 234 divided by 255 (0xff) gives 0 and the remainder is 234 (0xea), then bd is ea.
    • Other digits: Content of the SEI frame.

    FAQ​

    Q: If I use the SEI frame in Media Push, is it necessary to use the signaling system to send the layout information? Do I use either signaling or SEI? A: Adding the H264/H265 SEI information to the video stream during Media Push is different from sending the uplink data. Only the app_data parameter of the SEI information relates to the uplink data.

    Q: Does the signaling system send the layout information? Why does the SEI frame contain the layout information? A: Agora puts the layout information in the SEI frame because the audience members in the LIVE_BROADCASTING profile also need the layout information of the current stream to set the window region of the hosts.