3D Spatial Audio

3D Spatial Audio brings theater-like effects to Broadcast Streaming, making it seem as if the sound originates from all around the user. Video SDK provides spatial audio effects that give users an immersive audio experience in scenarios such as e-sports competitions, and online conferences.

You can configure the following spatial audio effects:

• Spatial audio effects for users:

  By setting the local and remote user's spatial positions, you create an environment that enables users to experience changes in the distance, position, and orientation of other users in real time. This includes:

  • Audio range: Enable users in the same group and within the range to hear each other.

  • Voice blur: Enable users to hear sounds outside of the selected range and distance, or sounds of specific objects, as blurred whispers.

  • Air attenuation: Create a more natural and realistic audio experience, making it easier for people to have natural conversations. The audio engine models how sound behaves as it travels through the air and adjusts the volume based on the distance of the audio source from the listener.

  • Local sound source playback: Customize the background sound and other audio effects.

• Spatial audio effects for media player:

  Update the spatial position of the media player to add a sense of space to media resources such as background sounds and musical accompaniment. Agora provides local cartesian coordinate system calculation solution for the media player. This solution calculates the relative positions of the local user and the media player through Video SDK. You update the spatial coordinates of the local user and the media player, respectively, so that the local user can hear the spatial audio effect of the media player.

The following figure shows the workflow you need to integrate spatial audio into your app.

In order to follow this procedure you must have:

• Implemented the SDK quickstart project for Broadcast Streaming.

• Android Studio 4.1 or higher.
• Android SDK API Level 24 or higher.
• A mobile device that runs Android 4.1 or higher.

• An Agora account and project.

• A computer with Internet access.

  Ensure that no firewall is blocking your network communication.

• To ensure a true spatial experience, Agora recommends using an audio device that supports true binaural playback.

To create the environment necessary to implement spatial audio into your app, open the SDK quickstart project for Broadcast Streaming you created previously.

This section shows how to use the Video SDK to implement spatial audio into your app, step-by-step.

In a real-word application, you report your local spatial position to a server in your environment and retrieve positions of remote users in the channel from your server. In this simple example, you use a SeekBar to set the spatial position of a remote user.

To add a SeekBar to the UI, in /app/res/layout/activity_main.xml, add the following code before </RelativeLayout>:

_18

<TextView

_18

android:id="@+id/seekBarText"

_18

android:layout_width="wrap_content"

_18

android:layout_height="wrap_content"

_18

android:layout_below="@id/JoinButton"

_18

android:text="Change spatial position"

_18

android:layout_alignStart="@id/JoinButton"/>

_18

_18

<SeekBar

_18

android:id="@+id/distanceSeekBar"

_18

android:layout_width="match_parent"

_18

android:layout_height="wrap_content"

_18

android:layout_below="@id/seekBarText"

_18

android:layout_alignStart="@id/JoinButton"

_18

android:layout_alignEnd="@id/LeaveButton"

_18

android:min="0"

_18

android:max="10"

_18

android:progress="0" />

1. Add the required libraries

   To set up and configure an instance of the spatial audio engine and use the SeekBar, import the corresponding Agora and Android libraries. In /app/java/com.example.<projectname>/MainActivity, add the following to the list of import statements:

   
   

   _5

   import io.agora.spatialaudio.ILocalSpatialAudioEngine;

   _5

   import io.agora.spatialaudio.LocalSpatialAudioConfig;

   _5

   import io.agora.spatialaudio.RemoteVoicePositionInfo;

   _5

   _5

   import android.widget.SeekBar;

   
   

To implement spatial audio features in your app, take the following steps:

1. Declare the variables you need

   You create an instance of ILocalSpatialAudioEngine to configure spatial audio parameters and set self and remote user positions. In /app/java/com.example.<projectname>/MainActivity, add the following declarations to the MainActivity class:

   
   

   _3

   private ILocalSpatialAudioEngine localSpatial;

   _3

   private int remoteUid = 0; // Stores the uid of the remote user

   _3

   private SeekBar distanceSeekBar;

   
   

2. Instantiate and configure the spatial audio engine

   To create an instance of ILocalSpatialAudioEngine at startup, take the following steps:

   1. When a user launches the app, you create an instance of ILocalSpatialAudioEngine, configure it and update the user's self position. To do this, add the following method to the MainActivity class:

      
      

      _24

      private void configureSpatialAudioEngine() {

      _24

      // Enable spatial audio

      _24

      agoraEngine.enableSpatialAudio(true);

      _24

      _24

      // Create and initialize the spatial audio engine

      _24

      LocalSpatialAudioConfig localSpatialAudioConfig = new LocalSpatialAudioConfig();

      _24

      localSpatialAudioConfig.mRtcEngine = agoraEngine;

      _24

      localSpatial = ILocalSpatialAudioEngine.create();

      _24

      localSpatial.initialize(localSpatialAudioConfig);

      _24

      _24

      // Set the audio reception range of the local user in meters

      _24

      localSpatial.setAudioRecvRange(50);

      _24

      _24

      // Set the length of unit distance in meters

      _24

      localSpatial.setDistanceUnit(1);

      _24

      _24

      // Define the position of the local user

      _24

      float[] pos = new float[]{0.0F, 0.0F, 0.0F};

      _24

      float[] forward = new float[]{1.0F, 0.0F, 0.0F};

      _24

      float[] right = new float[]{0.0F, 1.0F, 0.0F};

      _24

      float[] up = new float[]{0.0F, 0.0F, 1.0F};

      _24

      // Set the position of the local user

      _24

      localSpatial.updateSelfPosition(pos, forward, right, up);

      _24

      }

      
      

   2. To execute this method at startup, add the following line to the onCreate method after setupVideoSDKEngine();:

      
      

      _1

      configureSpatialAudioEngine();

      
      

3. Set the spatial position of a remote user

   To update the spatial position of a remote user:

   1. Define the RemoteVoicePositionInfo and call updateRemotePosition.

      
      

      _12

      public void updateSpatialAudioPosition(float sourceDistance){ //View view){

      _12

      // Define a remote user's spatial position

      _12

      RemoteVoicePositionInfo positionInfo = new RemoteVoicePositionInfo();

      _12

      // The three values represent the front, right, and top coordinates

      _12

      positionInfo.position = new float[]{sourceDistance, 0.0F, 0.0F};

      _12

      positionInfo.forward = new float[]{sourceDistance, 0.0F, 0.0F};

      _12

      _12

      // Update the spatial position of the specified remote user

      _12

      localSpatial.updateRemotePosition(remoteUid, positionInfo);

      _12

      _12

      showMessage("Spatial position updated");

      _12

      }

      
      

   2. To call updateSpatialAudioPosition when a user moves the slider, add the following code to onCreate:

      
      

      _15

      distanceSeekBar = (SeekBar)findViewById(R.id.distanceSeekBar);

      _15

      _15

      distanceSeekBar.setOnSeekBarChangeListener(new SeekBar.OnSeekBarChangeListener() {

      _15

      public void onProgressChanged(SeekBar seekBar, int progress, boolean fromUser) {

      _15

      //Required to implement OnSeekBarChangeListener

      _15

      }

      _15

      _15

      public void onStartTrackingTouch(SeekBar seekBar) {

      _15

      //Required to implement OnSeekBarChangeListener

      _15

      }

      _15

      _15

      public void onStopTrackingTouch(SeekBar seekBar) {

      _15

      updateSpatialAudioPosition((float) seekBar.getProgress());

      _15

      }

      _15

      });

      
      

   3. To update the spatial position of a specific remote user, you need the uid of that user. Add the following line to the onUserJoined callback, to store the remoteUid:

      
      

      _1

      remoteUid = uid;

      
      

To ensure that you have implemented 3D Spatial Audio features into your app:

1. Generate a temporary token in Agora Console .

2. In your browser, navigate to the Agora web demo and update App ID, Channel, and Token with the values for your temporary token, then click Join.

3. In Android Studio, open app/java/com.example.<projectname>/MainActivity and update appId, channelName and token with the values from Agora Console.

4. Connect an Android device to your development device.

5. In Android Studio, click Run app. A moment later, you see the project installed on your device.

   If this is the first time you run your app, grant camera and microphone permissions.

6. Press Join to connect to the same channel as your web demo.

7. Test spatial audio effects for remote users

   1. Put on earphones connected to the Android test device.

   2. Speak into the microphone connected to the web demo app. Note the quality of audio playing through your earphones.

   3. Move the slider to the right. Your app updates the position of the remote user in the spatial audio engine by increasing the distance.

   4. Speak into the microphone again and listen to the audio through your earphones. You feel that the remote user has moved away from you.

8. Test spatial audio effects for media player

   1. To setup spatial audio position of your media player, add Media Playing to your app.

   2. Replace the call to localSpatial.updateRemotePosition in updateSpatialAudioPosition with the following:

      
      

      _2

      // Update spatial position of the media player

      _2

      localSpatial.updatePlayerPositionInfo(mediaPlayer.getMediaPlayerId(), positionInfo);

      
      

   3. Play the media file in your Android app.

   4. Connect your earphones to the web demo and note the quality of audio playing.

   5. Move the slider to the right. Your app updates the position of the media player in the spatial audio engine.

      Note the change in quality of media playing through your earphones. You feel that the position of the media player has moved away from you.

This section completes the information on this page, or points you to documentation that explains other aspects about this product.

• ILocalSpatialAudioEngine
• updateSelfPosition
• updateRemotePosition
• removeRemotePosition
• clearRemotePositions
• RemoteVoicePositionInfo