Control optical audio volume within OSX
by postbreak
No sooner do I write an article complaining about not having the ability to change the sound volume while using optical audio on OSX do I find an easy fix. By harnessing the application Soundflower you can easily circumvent this lock. First download Soundflower from their google code page. Next change your default audio output to Soundflower (2ch) as seen below.
Next you are going to open the Soundflowerbed located in your Applications folder, it is inside the folder Soundflower. Now you have a menu item that looks like a flower. Click on the Soundflowerbed icon, go down to Soundflower (2ch) and make sure both channels are enabled and check the output to Built-in Output. That’s it, you’re finished. Now you can adjust the volume from your keyboard and from your receiver. Why Apple has this locked is beyond me because this method works fine with no audible difference. Here is an image showing how my settings look.
I had to adjust the volume a couple of times for my output to start working so if you don’t hear sound right away make sure you try to change the volume first. Also this application requires you to reboot so save your documents before following this guide.
iamthekiller 
This only works for stereo audio output via the optical port, although it works well. If, however, one is trying to pass digital surround from a DVD to the optical port, it won’t work with Soundflower and will cause the Mac Mini to crash (in my case, anyway). I had to uninstall Soundflower to get the Mac to put out surround-encoded digital signals to my 5.1 receiver and just accept, unfortunately, that I can’t therefore use the Apple remote to adjust the overall volume of the surround sound audio level. If anyone comes up with a software fix for this, I’d love to see it.
I get it. Indeed you can change the volume of digital output. Problem is that if it is encoded in a surround format (DTS for instance), changing the volume will completely mess up the sound file. Somehow the encoded surround information is lost and the receiver will output only static hiss. Here is a living proof of this:
http://ask.metafilter.com/35539/Why-is-optical-digital-audio-locked-at-full-on-a-Mac
It tried it myself and indeed my amp displayed “DTS” and played the song in all speakers. Note that you MUST have your iTunes volume up to max for this.