Emulating 3D Sound

3D sound, also called directional hearing, creates a richer audio experience for players by positioning virtual sound sources in 3D space. Sounds are more realistic and give the game an extra dimension-you might be able to hear a bad guy sneaking up on you from behind, or hear a door opening far off in the distance. Many different effects are used to create 3D sound. Here are some of the most common ones:

Make sound diminish with distance so the farther away a sound source is, the quieter it is.
Pan sounds to the appropriate speaker so a sound source on the left of the player is played in the left speaker, and a sound source on the right of the player is played in the right speaker. This can also be extended to four-speaker surround sound.
Apply room effects so sound waves bounce off walls, creating echoes and reverberation.
Apply the Doppler effect so a sound source's movement affects its pitch. For instance, a sound source quickly moving toward the player has a higher pitch than a stationary one. If you've ever heard a fire engine or train whiz past you, you've experienced this effect.

"3D sound? But we haven't even discussed 3D graphics yet!" Don't worry; 3D sound doesn't apply only to 3D space. All of the effects used to create 3D sound can also be applied to a 2D game.

The Idea Behind Creating a 3D Filter

So now we move on to creating a 3D filter. We already discussed a way to create echoes, so we won't focus on room effects. Instead, we focus on creating a sound that diminishes with distance. But first, how can we change the volume of a sound? Using Java Sound's Controls, you can dynamically change the volume and pan of a sound. Unfortunately, the Controls don't have the best quality for real-time use, and using them for such can create clicks or "wobbly" sounds. Therefore, you'll do the volume change yourself instead of using Controls. This means it is your task to avoid clicks and pops. "How do clicks and pops occur?" you ask. Good question! Clicks and pops can happen when the volume of a sound abruptly changes, as in Screenshot.

**Abruptly changing the volume of a sound can result in "pops."**

To avoid this in your 3D filter, whenever you need to change the volume of a sound, be sure to gradually change the volume over time, as in Screenshot.

**Gradually changing the volume of a sound creates a more natural listening experience.**

Because your samples are stored in 16-bit signed format, to change the volume, just multiply each sample by a factor. For example:

sample = (short)(sample * volume);

If you want to play the sound at half the volume, multiply each sample by 0.5.

Implementing a 3D Filter

The Filter3d class, in Listing 4.7, modifies the volume of a sound to make it get quieter with distance. It keeps track of two Sprite objects: one as a sound source and one as a listener. The farther away the source Sprite is from the listener Sprite, the quieter the sound is. Distance is measured using the Pythagorean Theorem:

A²+B²=C²

Also, the Filter3d class has a maximum distance that sound can be heard. If the listener is more than the maximum distance from the sound source, the sound isn't heard at all. The sound's volume is scaled linearly from 0 to the maximum distance. The Filter3d class is designed so that the sound is changed whenever the sprites' positions change. However, the change in the sound might fall a split second behind the sprites' change in position. Internally, Lines use a buffer to store sound samples before they're actually sent to the native sound system. Then the native sound system might have its own buffer. After that, the sound card might have its own internal buffer. All these buffers create a small delay. To minimize the delay, we set up a short 100ms buffer in the SimpleSoundPlayer class. Although this minimizes the effect, it doesn't eliminate it.

Listing 4.7 Filter3d.java

package com.brackeen.javagamebook.sound;
import com.brackeen.javagamebook.graphics.Sprite;
/**
 The Filter3d class is a SoundFilter that creates a 3D sound
 effect. The sound is filtered so that it is quieter the farther
 away the sound source is from the listener.
 <p>Possible ideas to extend this class:
 <ul><li>pan the sound to the left and right speakers
 </ul>
 @see FilteredSoundStream
*/
public class Filter3d extends SoundFilter {
 // number of samples to shift when changing the volume.
 private static final int NUM_SHIFTING_SAMPLES = 500;
 private Sprite source;
 private Sprite listener;
 private int maxDistance;
 private float lastVolume;
 /**
 Creates a new Filter3d object with the specified source
 and listener Sprites. The Sprite's position can be
 changed while this filter is running.
 <p> The maxDistance parameter is the maximum distance
 that the sound can be heard.
 */
 public Filter3d(Sprite source, Sprite listener,
 int maxDistance)
 {
 this.source = source;
 this.listener = listener;
 this.maxDistance = maxDistance;
 this.lastVolume = 0.0f;
 }
 /**
 Filters the sound so that it gets more quiet with
 distance.
 */
 public void filter(byte[] samples, int offset, int length) {
 if (source == null || listener == null) {
 // nothing to filter - return
 return;
 }
 // calculate the listener's distance from the sound source
 float dx = (source.getX() - listener.getX());
 float dy = (source.getY() - listener.getY());
 float distance = (float)Math.sqrt(dx * dx + dy * dy);
 // set volume from 0 (no sound) to 1
 float newVolume = (maxDistance - distance) / maxDistance;
 if (newVolume <= 0) {
 newVolume = 0;
 }
 // set the volume of the sample
 int shift = 0;
 for (int i=offset; i<offset+length; i+=2) {
 float volume = newVolume;
 // shift from the last volume to the new volume
 if (shift < NUM_SHIFTING_SAMPLES) {
 volume = lastVolume + (newVolume - lastVolume) *
 shift / NUM_SHIFTING_SAMPLES;
 shift++;
 }
 // change the volume of the sample
 short oldSample = getSample(samples, i);
 short newSample = (short)(oldSample * volume);
 setSample(samples, i, newSample);
 }
 lastVolume = newVolume;
 }
}

In Filter3d you create a diminishing-with-distance effect-the minimum amount necessary for a 3D effect. If you wanted to add panning, you would need to update the SoundFilter architecture to enable you to turn a mono sound into a stereo sound. Then you could calculate the pan you wanted from -1 to 1, where -1 is the left speaker, 1 is the right speaker, and 0 is in the middle. Finally, the samples for each speaker could be calculated as follows:

short sampleLeft = (short)(sample * (1-pan));
short sampleRight = (short)(sample * (1+pan));

For a 2D game, the pan could be determined by the position where a sound source is on the screen. A sound source on the left side of the screen would play in the left speaker, and so on.

Trying Out the 3D Filter

Although you don't have panning in the Filter3d class, the diminishing-with-distance effect is cool enough for many games; you can try it out by creating the Filter3dTest class, in Listing 4.8. The Filter3dTest class is a graphical demo that enables you to move a fly around on the screen using the mouse, as shown in the screenshot in Screenshot. Along with the fly is a virtual "ear" in the middle of the screen. The fly makes a buzzing sound, and the closer the fly is to the ear, the louder the buzzing sound is.

Screenshot The `Filter3dTest` class plays the fly's buzzing sound louder the closer it is to the ear.

Java graphics 04fig05

The fly is a simple three-frame animation that is drawn wherever the mouse is located. As usual, press Escape to exit the demo.

Listing 4.8 Filter3dTest.java

import java.awt.*;
import java.awt.event.KeyEvent;
import java.io.InputStream;
import java.io.IOException;
import javax.sound.sampled.*;
import com.brackeen.javagamebook.graphics.*;
import com.brackeen.javagamebook.input.*;
import com.brackeen.javagamebook.sound.*;
import com.brackeen.javagamebook.test.GameCore;
import com.brackeen.javagamebook.util.LoopingByteInputStream;
/**
 The Filter3dTest class demonstrates the Filter3d
 functionality. A fly buzzes around the listener, and the
 closer the fly is, the louder it's heard.
 @see Filter3d
 @see SimpleSoundPlayer
*/
public class Filter3dTest extends GameCore {
 public static void main(String[] args) {
 new Filter3dTest().run();
 }
 private Sprite fly;
 private Sprite listener;
 private InputManager inputManager;
 private GameAction exit;
 private SimpleSoundPlayer bzzSound;
 private InputStream bzzSoundStream;
 public void init() {
 super.init();
 // set up input manager
 exit = new GameAction("exit",
 GameAction.DETECT_INITAL_PRESS_ONLY);
 inputManager = new InputManager(
 screen.getFullScreenWindow());
 inputManager.mapToKey(exit, KeyEvent.VK_ESCAPE);
 inputManager.setCursor(InputManager.INVISIBLE_CURSOR);
 createSprites();
 // load the sound
 bzzSound = new SimpleSoundPlayer("../sounds/fly-bzz.wav");
 // create the 3D filter
 Filter3d filter =
 new Filter3d(fly, listener, screen.getHeight());
 // create the filtered sound stream
 bzzSoundStream = new FilteredSoundStream(
 new LoopingByteInputStream(bzzSound.getSamples()),
 filter);
 // play the sound in a separate thread
 new Thread() {
 public void run() {
 bzzSound.play(bzzSoundStream);
 }
 }.start();
 }
 /**
 Loads images and creates sprites.
 */
 private void createSprites() {
 // load images
 Image fly1 = loadImage("../images/fly1.png");
 Image fly2 = loadImage("../images/fly2.png");
 Image fly3 = loadImage("../images/fly3.png");
 Image ear = loadImage("../images/ear.png");
 // create "fly" sprite
 Animation anim = new Animation();
 anim.addFrame(fly1, 50);
 anim.addFrame(fly2, 50);
 anim.addFrame(fly3, 50);
 anim.addFrame(fly2, 50);
 fly = new Sprite(anim);
 // create the listener sprite
 anim = new Animation();
 anim.addFrame(ear, 0);
 listener = new Sprite(anim);
 listener.setX(
 (screen.getWidth() - listener.getWidth()) / 2);
 listener.setY(
 (screen.getHeight() - listener.getHeight()) / 2);
 }
 public void update(long elapsedTime) {
 if (exit.isPressed()) {
 stop();
 }
 else {
 listener.update(elapsedTime);
 fly.update(elapsedTime);
 fly.setX(inputManager.getMouseX());
 fly.setY(inputManager.getMouseY());
 }
 }
 public void stop() {
 super.stop();
 // stop the bzz sound
 try {
 bzzSoundStream.close();
 }
 catch (IOException ex) { }
 }
 public void draw(Graphics2D g) {
 // draw background
 g.setColor(new Color(0x33cc33));
 g.fillRect(0, 0, screen.getWidth(), screen.getHeight());
 // draw listener
 g.drawImage(listener.getImage(),
 Math.round(listener.getX()),
 Math.round(listener.getY()),
 null);
 // draw fly
 g.drawImage(fly.getImage(),
 Math.round(fly.getX()),
 Math.round(fly.getY()),
 null);
 }
}

One thing to note about Filter3dTest is that a separate thread is created to play the sound. The play() method of SimpleSoundPlayer blocks until the sound is done playing, and because you're using a looping sound, this method could theoretically block forever. To keep the program from getting stuck, a new thread is created. Also, don't forget that you need to call System.exit(0) to exit Java programs that use Java Sound. To take care of this, you can add an extra method, lazilyExit(), to the GameCore class that Filter3dTest extends:

public void lazilyExit() {
 Thread thread = new Thread() {
 public void run() {
 // first, wait for the VM exit on its own.
 try {
 Thread.sleep(2000);
 }
 catch (InterruptedException ex) { }
 // system is still running, so force an exit
 System.exit(0);
 }
 };
 thread.setDaemon(true);
 thread.start();
}

This method creates a new daemon thread that waits two seconds and then exits the VM. Because the VM normally exits when all nondaemon threads are finished, if this thread is running only with other daemon threads, the VM exits cleanly on its own. Otherwise, after two seconds, the System.exit(0) method is called. The lazilyExit() method is called in the GameCore class right after restoring the screen. Now you might notice a problem: Creating a new thread every time you play a sound isn't the cleanest solution. You'll fix this by creating a more advanced sound manager.