Skip to main content

  • Core Products

  • Add-Ons

  • Extensions Marketplace

Communicate with Video SDK

If your business requires not only high-quality audio and video transmission, but also easy integration, low memory usage, and low power consumption, for applications such as smart cameras and smart doorbells, Agora recommends that you use the Video SDK with the IoT SDK.

The following table shows the data formats supported when the IoT SDK communicates with the Video SDK (v4.x):

Video SDK TypeSupported formats
Native/third-party frameworks:
  • Android
  • iOS/macOS
  • Windows
  • Electron
  • Unity
  • Flutter
  • React Native
  • Audio: G722, G711, Opus, AAC
  • Video: H.264, JPEG
Web (v4.x)
  • Audio: G722, G711, Opus
  • Video: H.264
Embedded systems
  • Audio: G711, Opus
  • Video: H.264, JPEG

The sending and receiving ends can communicate with each other only if the set data format is consistent and complies with the above table.

The following sections describe how to set the audio and video data formats in the SDKs.

Set up IoT SDK​

Audio

IoT SDK only supports mono audio data, and the data sending interval is 20 ms.

To send and receive AAC audio data, set audio_frame_info_t.data_type to AUDIO_DATA_TYPE_AACLC or AUDIO_DATA_TYPE_HEAAC and specify the corresponding data size when calling the agora_rtc_send_audio_data method. The size of AAC audio data needs to be resolved by the user.

Video

In IoT SDK, H.264 video data is sent and received by default.

To send and receive JPEG video data, when calling the agora_rtc_send_video_data method, set video_frame_info_t.data_type to VIDEO_DATA_TYPE_GENERIC_JPEG.

Set up Video SDK for Native/third-party frameworks​

Audio

In Native/third-party framework SDKs (v4.x), Opus or AAC audio data is sent and received by default.

To send and receive G722 or G711 audio data, use the following methods:

Video SDK typeG722G711 (PCMA)G711 (PCMU)
AndroidagoraEngine.setAudioProfile(6) or agoraEngine.setParameters("{\"che.audio. custom_payload_type\":9}")agoraEngine.setParameters("{\"che.audio. custom_payload_type\":8}")agoraEngine.setParameters("{\"che.audio. custom_payload_type\":0}")
iOS/macOSagoraKit.setAudioProfile(6) or agoraKit.setParameters("{\"che.audio. custom_payload_type\":9}")agoraKit.setParameters("{\"che.audio. custom_payload_type\":8}")agoraKit.setParameters("{\"che.audio. custom_payload_type\":0}")
WindowsagoraEngine->setAudioProfile(6) or agora::base::AParameter apm(agoraEngine); apm->setParameters("{\"che.audio. custom_payload_type\":9}")agora::base::AParameter apm(agoraEngine);apm->agoraEngine->setParameters("{\"che.audio. custom_payload_type\":8}")agora::base::AParameter apm(agoraEngine);apm->agoraEngine->setParameters("{\"che.audio. custom_payload_type\":0}")
Electronthis.agoraEngine?.setAudioProfile(6) or this.agoraEngine?.setParameters("{\"che.audio. custom_payload_type\":9}")this.agoraEngine?.setParameters("{\"rtc.audio. custom_payload_type\":8}")this.agoraEngine?.setParameters("{\"rtc.audio. custom_payload_type\":0}")
Unityawait agoraEngine.setAudioProfile(profile: AudioProfileType.audioProfileIot) or await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":9}")await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":8}")await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":0}")
Flutterawait agoraEngine.setAudioProfile(profile: AudioProfileType.audioProfileIot) or await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":9}")await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":8}")await agoraEngine.setParameters("{\"che.audio. custom_payload_type\":0}")
React Nativeawait agoraEngine.current?.setAudioProfile(6) or await agoraEngine.current?.setParameters("{\"che.audio. custom_payload_type\":9}")await agoraEngine.current?.setParameters("{\"che.audio. custom_payload_type\":8}")await agoraEngine.current?.setParameters("{\"che.audio. custom_payload_type\":0}")

Video

In Native/third-party framework SDKs (v4.x), H.264 video data is sent and received by default.

To send and receive JPEG video data, use the following methods:

Video SDK typeMethod
AndroidagoraEngine.setParameters("{\"engine.video.codec_type\": \"20\"}")
iOS/macOSagoraKit.setParameters("{\"engine.video.codec_type\": \"20\"}")
Windowsagora::base::AParameter apm(agoraEngine);
apm->setParameters("{\"engine.video.codec_type\": \"20\"}")
Electronthis.agoraEngine.setParameters("{\"engine.video.codec_type\": \"20\"}");
UnityagoraEngine.SetParameters("{\"engine.video.codec_type\": \"20\"}");
Flutterawait agoraEngine.setParameters("{\"engine.video.codec_type\": \"20\"}");
React Nativethis.engine.setParameters("{\"engine.video.codec_type\": \"20\"}");

Set up Video SDK for Web​

Audio

In Video SDK for Web (v4.x), Opus audio data is sent and received by default.

To send and receive G722, G711 audio data, call the following method:

  • G711 (PCMA): AgoraRTC.createClient({mode: "live", codec: "h264", audioCodec: "pcma"})
  • G711 (PCMU): AgoraRTC.createClient({mode: "live", codec: "h264", audioCodec: "pcmu"})
  • G722: AgoraRTC.createClient({mode: "live", codec: "h264", audioCodec: "g722"})

Only the Web SDK version 4.9.0 or higher supports setting the audio encoding format.

Video

In Web SDK (v4.x), VP8 video data is sent and received by default. You call createClient to set the video data format to "h264".

Set up Video SDK for embedded systems​

In Video SDK for embedded systems, Opus audio data and H.264 video data are sent and received by default. If you want to receive JPEG video data or send and receive G711 audio data, contact sales-us@agora.io for activation.

Video SDK for embedded systems does not support sending JPEG video data.