How to Encrypt Your USB Flash Drives

Did you know that your modern Linux kernel comes with a built-in encryption framework? I am talking about dm-crypt (device-mapper crypt) and the user-friendly layer on top of it, called LUKS (Linux Unified Key Setup). I just encrypted all of my USB flash drives two weeks ago using the dm-crypt + LUKS method and I am very happy with the results.

The process itself is very simple.

# 1. Find the correct device.

lsblk

# 2. Wipe the device with random data. I prefer to target the disk by its UUID
# because using the /dev/sdX convention is not very reliable (the letters can
# change between boots/hotmounts). NOTE: You might be interested in
# http://frandom.sourceforge.net/ if your device is over 16 GiB or so, because
# using /dev/urandom can be very slow. If using Arch Linux, you can get it from
# the AUR: https://aur.archlinux.org/packages/frandom/.

dd if=/dev/urandom of=/dev/disk/by-uuid/XXX bs=4096

# 3. Create the partition on the device.

cfdisk /dev/disk/by-uuid/XXX

# 4. Encrypt the partition and make it LUKS-compatible. See the manpage for
# cryptsetup(8).
#   -c: cipher type to use
#   -y: LUKS will ask you to input the passphrase; using -y will ask you twice
#       and complain if the two do not match.
#   -s: Key size in bits; the larget the merrier, but limited by the cipher/mode used.

cryptsetup -c aes-xts-plain -y -s 512 luksFormat /dev/disk/by-uuid/XXX

# 5. Open the partition with LUKS.

cryptsetup luksOpen /dev/disk/by-uuid/XXX mycrypteddev

# The partition is now available from /dev/mapper/mycrypteddev as a "regular"
# partition, since LUKS is now handling all block device encryption between the
# user and the device.

# 6. Set up a filesystem on the partition.

mkfs.ext4 /dev/mapper/mycrypteddev

# 7. Close the partition with LUKS.

cryptsetup luksClose /dev/mapper/mycrypteddev

# Encryption setup complete! Now every time you want to access the partition,
# you must first open it with LUKS and then mount it. Then when you're done, do
# the reverse: unmount and close it with LUKS.

# To mount and open with LUKS:

cryptsetup luksOpen /dev/disk/by-uuid/XXX mycrypteddev
mount -t ext4 /dev/mapper/mycrypteddev /mnt/mount_point

# To unmount and close with LUKS:

umount /mnt/mount_point
cryptsetup luksClose mycrypteddev

The mount/open and unmount/close steps necessary for using the device is laborious. That’s why you should write a bash script to run them. I’ve written the following bash script called cmount.sh to access my 3 USB drives this way:

#!/bin/zsh
# LICENSE: PUBLIC DOMAIN
# mount/unmount encrypted flash drives

mp=$3
uuid=""

case $2 in
    "0")
        uuid="11e102cd-dea1-46a8-ae9b-b3f74b536e64" # my red USB drive
        ;;
    "1")
        uuid="cf169437-b937-4a39-86cb-7ca82bd9fe94" # my green one
        ;;
    "2")
        uuid="57a0b7d5-d2a6-47e0-a0e3-adf69501d0cd" # my blue one
        ;;
    *)
        ;;
esac

if [[ $uuid == "" ]]; then
    echo "No predefined device specified."
    exit 0
fi

case $1 in
    "m")
        echo "Authorizing encrypted partition /dev/mapper/$mp..."
        sudo cryptsetup luksOpen /dev/disk/by-uuid/$uuid $mp
        echo -n "Mounting partition on /mnt/$mp..."
        sudo mount -t ext4 /dev/mapper/$mp /mnt/$mp && echo "done."
        ;;
    "u")
        echo -n "Unmounting /mnt/$mp..."
        sudo umount /mnt/$mp && echo "done."
        echo -n "Closing encrypted partition /dev/mapper/$mp..."
        sudo cryptsetup luksClose $mp && echo "done."
        ;;
    *)
        ;;
esac

To mount the green USB to /mnt/ef0 (“ef0” is just an arbitrary folder name):

./cmount.sh m 1 ef0

Then to unmount:

./cmount.sh u 1 ef0

Simple, eh? Go forth and encrypt all of your USB drives, so that when they get lost, they can’t be read by curious strangers. You can use the above steps to create and encrypt multiple partitions in the same device, or to only encrypt one partition while leaving other partitions unencrypted (i.e., steps 4 through 7 are partition-specific). The choice is yours. I prefer partition-level (aka “block device”) encryption over file/folder encryption because I don’t have to mentally think every time “hey, do I want to encrypt this?” for every file/folder I create.

If you want to look into encrypting your hard drives and swap partitions, read through this disk encryption page, and particularly, this section. There are many “levels” of encryption, and you should consider the many options available to you.

Advertisements

Intro to QuickCheck and Hpc

And you thought my Haskell honeymoon was over.

So the other day, I finally got around to using the legendary QuickCheck testing library for a semi-serious project of mine. And I must say that I am very impressed with QuickCheck — it caught two subtle (but serious) errors in my core functions — in day one! I’m going to explain some basic concepts to help you get started with QuickCheck immediately; afterwards I’ll throw in a brief discussion about the Haskell Program Coverage (“Hpc”) tool, too, since it goes well with QuickCheck.

QuickCheck

The point of QuickCheck is to generate tons of random, junk values (the nastier-looking, the better) to feed into your functions. Well, they’re not really junk values, from a type system perspective, but the point is to generate lots of edge-case values that we forget to consider when writing the code. QuickCheck supports testing of pure functions as well as monadic ones (ST monad), but I’ll just focus on the pure functions because that’s what I have experience with.

So the next question is, how does QuickCheck generate random values? Well, if your function uses any of the basic types like Int, [Integer], Double, Word64, etc., QuickCheck knows how to generate the random values already. But if you use a custom type, or a unique combination of ordinary types (e.g., (Maybe Int, Maybe Int)), you have to instruct QuickCheck how to generate it. You do this by writing an instance for QuickCheck’s Arbitrary typeclass.

So let’s say my custom type is (Maybe Int, Maybe Int), and that it has a special requirement: if it is (Just a, Just b), then b must be greater than a. No problem.

newtype MPair = MPair (Maybe Int, Maybe Int)
    deriving (Show)

instance Arbitrary MPair where
    arbitrary = do
        -- First, generate a random Int.
        a <- arbitrary :: Gen Int
        -- Now generate another random Int, but such that it is greater than *a*.
        b <- suchThat (arbitrary :: Gen Int) (>a)
        -- Now choose between either (Just a), or Nothing.
        a' <- elements [Just a, Nothing]
        -- Repeat the same coin flip for *b*.
        b' <- elements [Just b, Nothing]
        -- Return the random result.
        return $ MPair (a', b')

There is a method to the madness! There are other useful combinators besides suchThat and elements, of course.

So now you’ve told QuickCheck how to generate the random values to stress-test your functions. The next step is to define the stress-tests. In the QuickCheck world, such testing functions are used to test very specific properties of your functions that must hold true no matter what you throw at it. By convention, these functions are named “prop_[function name]”.

So here’s an example, prop_isTouching, which, appropriately enough, tests the isTouching function..

prop_isTouching :: MPair -> MPair -> Bool
prop_isTouching x y =
    isTouching x' y' == isTouching y' x'
    where
    x' = fromMPair x
    y' = fromMPair y

isTouching :: (Ord a) => (Maybe a, Maybe a) -> (Maybe a, Maybe a) -> Bool
isTouching p@(pStart, pEnd) q@(qStart, qEnd)
    -- if no pair has both of its values defined, then just return False;
    -- technically this notion is incorrect, but the greater codebase behind this
    -- interprets it this way, so it's OK
    | not (isBoth p) || not (isBoth q) = False
    -- if one or both pairs are empty, then return False
    | isBlank p || isBlank q = False
    | isLeft q = inP qStart'
    | isRight q = inP qEnd'
    | isLeft p = inQ pStart'
    | isRight p = inQ pEnd'
    | otherwise = inP qStart'
        || inP qEnd'
        || inQ pStart'
        || inQ pEnd'
    where
    pStart' = fromJust pStart
    pEnd' = fromJust pEnd
    qStart' = fromJust qStart
    qEnd' = fromJust qEnd
    inP = isInside (pStart', pEnd')
    inQ = isInside (qStart', qEnd')

isLeft
    , isRight
    , isBoth
    , isBlank :: (Maybe a, Maybe a) -> Bool
isLeft (a, b) = isJust a && isNothing b
isRight (a, b) = isNothing a && isJust b
isBoth (a, b) = isJust a && isJust b
isBlank (a, b) = isNothing a && isNothing b

fromMPair :: MPair -> (Maybe Int, Maybe Int)
fromMPair (MPair a) = a

isInside :: (Ord a) => (a, a) -> a -> Bool
isInside (a, b) c = a <= c && c <= b

You might be wondering, “Hey, I thought you were going to test a function that takes (Maybe Int, Maybe Int), not (Maybe a, Maybe a)!” Well, believe me, it pays off a lot to write higher-order functions like this that work on multiple types. The fact that we can test it using Maybe Ints (a very simple type) is just one benefit. Notice how I’ve made sure to restrict isTouching‘s arguments to the Ord typeclass, since we expect the right hand value in the pair to be greater than the one on the left (if it exists). The fancy Arbitrary instance up above was not in vain.

Anyway, the isTouching function merely checks to see if one given pair “falls in” or “touches” the other pair of values. It’s a pretty mundane function, but such functions often form the backbone of the rest of your code, so it’s really important to get these 100% right. The various helper functions like isLeft, isRight, fromMPair, etc. may seem annoying, as if they get in the way of the example to test isTouching itself. But think about it: all of these auxiliary functions will, by virtue of their necessity, be tested by calling prop_isTouching! And if prop_isTouching keeps failing (and your custom type or function is too complex for a 10-minute debugging session), you can always add more prop_ functions to test these auxiliary functions in isolation. Haskell embraces small, cute functions, and so should you!

The prop_isTouching function itself is straightforward enough: it tests the “commutative” property of isTouching — that the order of the arguments does not matter.

So far so good. Now we just need to run prop_isTouching hundreds (or thousands) of times to see if it holds. QuickCheck defines some basic test running functions, for use in the IO monad. The simplest one is aptly named quickCheck. So, we can run the function above like so:

import Test.QuickCheck

import [Your testing module, where prop_isTouching resides]

main :: IO ()
main = quickCheck prop_isTouching

This will run prop_isTouching 100 times with the default testing settings. But default settings are usually not desirable (for me, I only caught errors when I “upped” the number of runs to 1000, among other things). Oh, and quickCheck will not tell you the name of the function it is testing. So here is a more useful version:

{-# LANGUAGE RecordWildCards #-}
import System.Exit
import Test.QuickCheck

import [Your testing module, where prop_isTouching resides]

-- Rigorous test arguments.
rigorous :: Args
rigorous = Args
    { replay = Nothing
    , maxSuccess = 1000 -- tests to run
    , maxDiscard = 1000 -- the number of tests that are thrown out and ignored b/c of "==>" conditions, before "giving up" and failing due to too many discarded tests
    , maxSize = 1000 -- if a prop_ function uses a list ([]) type, maxSize is the max length of the randomly generated list
    , chatty = True
    }

-- Quick test arguments.
quick :: Args
quick = Args
    { replay = Nothing
    , maxSuccess = 100
    , maxDiscard = 100
    , maxSize = 100
    , chatty = True
    }

runTests :: [String] -> IO ()
runTests as = case as of
    [] -> runTests' quick
    a -> case head a of
        "1" -> runTests' quick
        "2" -> runTests' rigorous
        _ -> runTests' quick
    where
    runTests' :: Args -> IO ()
    runTests' testArgs = do
        -- if all of your prop_ functions are of the same type, you can put
        -- them in a list and use mapM_ instead
        f prop_isTouching "prop_isTouching"
        f prop_someOtherFunc1 "someOtherFunc1"
        f prop_someOtherFunc2 "someOtherFunc2"
        f prop_someOtherFunc3 "someOtherFunc3"
        f prop_someOtherFunc4 "someOtherFunc4"
        where
        f prop str = do
            putStrLn str
            quitOnFail =<< quickCheckWithResult testArgs prop
        quitOnFail r = case r of
            -- pattern match with just two dots with RecordWildCards because I'm lazy
            Success{..} -> return ()
            _ -> exitFailure

main :: IO ()
main = getArgs >>= runTests

If you compile the above as “test”, then running “./test 2” will use the “rigorous” test settings. The key difference is that instead of quickCheck, we use quickCheckWithResult. With it, we can provide our choice of test settings (the Args type), and also get some feedback on what the test results were. For simplicity’s sake, we only check if the test was a complete success; if it’s anything else, we abort immediately.

Here’s a quick note about the term “shrink” that you might encounter: if QuickCheck spots a failure, it will first try to shrink the size of the random input repeatedly while maintaining the “failure” result (and QuickCheck will tell you about how many shrinks were performed). This is to help you work with reasonably small values (esp. useful if you have list arguments with hundreds of items each, like in prop_foo above).)

The careful reader would have wondered what the “==>” was about in the comments. Well, the “==>” function is called the “implication” function, and is used to throw out invalid values before running the prop_ function. It’s another way to customize the random value, sort of like how we defined a custom Arbitrary instance up above for the MPair type synonym. For example,

prop_foo :: [Int] -> [Int] -> Prop
prop_foo xs ys =
    not (null xs) ==> -- xs must not be null
    (length ys > 3) ==> -- ys must have at least 4 elements
    foo xs ys == foo ys xs

and it only gets to the “foo xs ys == bar xs ys” part if the two statements above are true. The only difference is that we have to end up with a Prop type instead of Bool, as was the case in prop_isTouching. No other part of your testing code needs to change. The advantage in using (==>) is its ease of use — you can trivially write any rare edge-case condition that you know does not conform to the spec, without bothering to write an Arbitrary instance. However, the disadvantage is that QuickCheck will waste time generating invalid inputs before it gets to the test the function in question. Because of this, you should first try to create an instance of Arbitrary before going with (==>).

Here is a version without using (==>) just for illustrative purposes:

import Data.List
import Test.QuickCheck

main = verboseCheck prop_foo

newtype Xs = Xs [Int]
    deriving (Show)
newtype Ys = Ys [Int]
    deriving (Show)

prop_foo :: Xs -> Ys -> Bool
prop_foo (Xs xs) (Ys ys) =
    foo xs ys == foo ys xs

foo :: [Int] -> [Int] -> [Int]
foo xs ys = sort $ xs ++ ys

instance Arbitrary Xs where
    arbitrary = do
    -- xs must not be null
    xs <- suchThat (arbitrary :: Gen [Int]) (not . null)
    return $ Xs xs

instance Arbitrary Ys where
    arbitrary = do
    -- ys must have at least 4 elements
    ys <- suchThat (arbitrary :: Gen [Int]) ((>3) . length)
    return $ Ys ys

That’s it! You now know enough to get started with QuickCheck today. No excuses! Make it a New Year’s resolution for 2012, if you have to!

QuickCheck has taught me to be skeptical of the “if it compiles, it is OK” Haskell attitude — it has really opened my eyes. I now want QuickCheck as a bare minimum testing setup for all of my Haskell code. No tests = poor design (Medieval II: Total War, I’m looking at you…).

QuickCheck can even be used to test C code with the Foreign Function Interface (FFI), so I’m forever tempted to go back to my other C projects and test them with QuickCheck as well (can you imagine the millions of bugs that lie hidden in C code everywhere — that could be exposed with just a little bit of Haskell knowledge and QuickCheck by the everyday developer? Suddenly, I am overcome with optimism… yet I fear that it is really pessimism underneath… hmm.)

Hpc

I said I would mention Hpc at the beginning. So, what does it do?

Simply put, Hpc tells you which codepaths were used (or not used) after running a binary executable. This is a perfect match for the binary you used for running QuickCheck tests, because during the course of its run QuickCheck will have fed tens of thousands of random values into your functions and have visited most, if not all, parts of your code. Thankfully, Hpc is included with GHC, so you already have it! Just compile your binary with the -fhpc flag. Here is the process:

  1. Compile your program with -fhpc. (ghc -fhpc –make foo.hs)
  2. Run your program. (./foo)
  3. Your program should have generated a *.tix metadata file, as well as a .hpc folder with *.mix files for hpc to analyze.
  4. Run hpc report foo for a quick plaintext report on your codepaths.
  5. Run hpc markup foo to generate prettified HTML files for a more comprehensive overview.
  6. Rinse and repeat. Remove the old *.tix files if you don’t want combined results.

One big caveat: make sure to delete any existing object files (*.o, *.hi) that you had earlier which were produced without the -fhpc flag! Otherwise, those portions of code will not generate hpc metadata! Alternatively, instead of deleting every *.o or *.hi file you have down your project’s folder tree, you could also pass the -fforce-recompile flag to ghc (this is the new name for the deprecated -no-recomp flag, which was used (sans explanation!) in Real World Haskell, Chapter 11).

Go forth, fellow Haskell newbies — embrace testing and code coverage analysis today!

Simple Haskell Data Pretty-printing

A couple years ago when I was knee-deep in Ruby, there was this really neat module called ‘pp’ that you could use to print out, in prettified format, any data structure.

#!/usr/bin/ruby

require 'pp'

a = {0 => [1, 2, 3, [4, 5], 6], 1 => "hello", 2 => ["foo", "bar", "quux", "toto", "tata"], 3 => "this is a long sentence"}

pp(a)

Would result in something like this (depending on your window width):

{0=>[1, 2, 3, [4, 5], 6],
 1=>"hello",
 2=>["foo", "bar", "quux", "toto", "tata"],
 3=>"this is a long sentence"}

Today I discovered that you can do basically the same thing in Haskell with minimum fuss. All you need is the Text.Show.Pretty module (fellow Archers can grab the haskell-pretty-show package on AUR). The github page for the module is here. Here is how you’d use it:

import IO
import qualified Text.Show.Pretty as Pr

data VeryComplexNestedType = VeryComplexNestedType
    { foo :: Foo
    , bar :: Bar
    , quux :: Quux
    }

someData = VeryComplexNestedType ......

main = putStrLn $ Pr.ppShow someData 

It took me quite a while to figure out that the module I needed was Text.Show.Pretty. All the web searches I did for pretty-printing led me to gigantic/industrial libraries like Text.PrettyPrint.HughesPJ or Language.Haskell.Pretty. Those libraries are there to let you customize your own pretty printing function on some arbitrary data type of your choice. But for super-simple, generic pretty-printing like Ruby’s ‘pp’ module, there’s Text.Show.Pretty. For some reason, it remains relatively unknown (the AUR page shows just 1 vote for it as of this writing).

Spread the word!

UPDATE March 13, 2011: In the comments, Neill pointed out that you can also optionally use the “ppsh” executable, which comes with the Text.Show.Pretty module (/usr/bin/ppsh on Arch Linux). Thus, you don’t have to change all of your existing code that uses “show”; all you have to do is pipe the output into ppsh, like “./myprogram | ppsh”. This method is arguably superior because now you don’t have to include Text.Show.Pretty as a dependency in your project.

Update October 30, 2012: To install Text.Show.Pretty (aka pretty-show on Hackage) on Arch Linux, just use cblrepo. The all AUR Haskell packages are pretty much dead at this point.

MPD – A Brief Guide

The Music Player Daemon, aka MPD, is a great little application to handle music on your linux computer. Until recently (when Linux really started to take off and gain popularity), MPD has been notoriously difficult to get to work properly. I decided to write this small intro to MPD after having used it for about 6 months. This is a little guide to it to get you started on using MPD as your music manager.

The World of MPD

First, what MPD is:

  • MPD keeps track of your music library. It does this by defining a music directory, to which you can easily add symlinks to all the various places where you store your music on your computer.
  • MPD tracks changes to your music library (any time an mp3 is removed, or a new album added, you simply update the MPD database file — which takes three or four seconds at most).
  • MPD can store playlists.
  • MPD can play songs, and output this information to various devices, such as your physical speakers (usually via your alsa sound drivers, but it can be any audio driver on your system), or even a http port for remote web listening (via icecast).

What MPD is NOT (what most people get confused about):

  • MPD cannot by itself determine: which songs are played and in what order. Nor does it come with any controls such as PAUSE, STOP, NEXT TRACK, etc. MPD, quite simply, can either: play a song, or do nothing while it is running. MPD’s behavior is controlled by clients, such as GMPC, Sonata, etc (my personal favorite is ncmpcpp).

The downside of this dichotomy between what MPD is and what it is not, is that you must install two applications to get the same functionality as any standard multimedia player like iTunes. However, there are many, many advantages of splitting up music-playing from music-controls. First, mpd will hardly crash and will be error-free due to its simple nature. Second, you gain the ability to access your large music collection from other computers in your home network without much fuss. This means that you can do things like: make your laptop act as a remote control for your desktop’s music output (the desktop’s speakers), or even create multiple MPD instances for specific purposes simultaneously (e.g., alsa and also icecast output).

Typical Setup

For most people, they only need one instance of MPD running and have it configured so that it plays songs directly to their computer speakers. So their MPD configuration file will look something like this (comment lines removed to save space):

music_directory "/home/shinobu/.mpd-untracked/music"
playlist_directory "/home/shinobu/.mpd-untracked/playlist"
db_file "/home/shinobu/.mpd-untracked/mpd.db"
log_file "/home/shinobu/.mpd-untracked/mpd.log"
pid_file "/home/shinobu/.mpd-untracked/mpd.pid"
state_file "/home/shinobu/.mpd-untracked/mpdstate"
sticker_file "/home/shinobu/.mpd-untracked/sticker.sql"
user "shinobu"
bind_to_address		"192.168.0.110" # bind it to this computer's IP address if someone on the LAN wants to use this mpd
port				"6600"
save_absolute_paths_in_playlists "yes"
follow_outside_symlinks "yes"
follow_inside_symlinks "yes"
audio_output {
    type			"alsa"
    name			"Exelion ALSA output"
    format			"44100:16:2"	# optional
    mixer_device	"default"
    mixer_control	"PCM"
}
replaygain			"track"
volume_normalization "yes"
filesystem_charset		"UTF-8"
id3v1_encoding			"ISO-8859-1"

It’s very straightforward. All of the essential files and folders are customized and defined in the config file. (My mpd directory is .mpd-untracked, since these files/folders are not tracked by my git folder containing all of my dotfiles. See my previous two posts for more info on that setup.) The config file itself resides at /home/shinobu/syscfg/mpd/cfg-alsa — a customized path and filename, which is referenced each time MPD starts on bootup (in my .xinitrc file), with the command “mpd /home/shinobu/syscfg/mpd/cfg-alsa”. MPD will daemonize itself when it is called, so calling it from your .xinitrc file without the “&” argument at the end of the line is OK. Notice how the above configuration avoids running into permission problems, since everything resides under a normal user’s directory.

I then start up ncmpcpp, which by default looks to the mpd host as localhost and port as 6600 — matching the (default) values in my MPD config file above. And it all works rather beautifully.

Multi-MPD Setup for Music Sharing

Back when I still used iTunes (about 3 years ago), I remember it had this neat feature to share your music with anyone on your local network. You can do the same thing with MPD. The trick is to add a separate MPD instance for this feature. I myself have this multi-MPD setup on my desktop, with one MPD for alsa-only output (as seen above), and a second MPD for icecast-only output. Icecast, by the way, is a more general solution not wholly related to MPD for sharing music over the internet — we can use icecast since it plays rather well with MPD (UNIX philosophy FTW!). Since my desktop has a second MPD instance so that it works with icecast, my laptop can now take advantage of the huge music collection that I have on it, remotely (well, as long as I’m on the home network in my case).

Here’s my MPD config file for icecast output:

music_directory "/home/shinobu/.mpd-untracked/music"
playlist_directory "/home/shinobu/.mpd-untracked/playlist"
db_file "/home/shinobu/.mpd-untracked/mpd.db"
log_file "/home/shinobu/.mpd-untracked/mpd-icecast.log"
pid_file "/home/shinobu/.mpd-untracked/mpd-icecast.pid"
state_file "/home/shinobu/.mpd-untracked/mpdstate-icecast"
sticker_file "/home/shinobu/.mpd-untracked/sticker.sql"
user "shinobu"
bind_to_address		"192.168.0.110" # bind it to this computer's IP address if someone on the LAN wants to use this mpd
port				"6601"
save_absolute_paths_in_playlists "yes"
follow_outside_symlinks "yes"
follow_inside_symlinks "yes"
audio_output {
    type			"shout"
    name			"exelion HD mpd stream"
    encoding		"ogg"			# optional
    host			"localhost"
    port			"8000"
    mount			"/mpd.ogg" # i.e., "http://192.168.0.110/mpd.ogg" is the live stream
                                           # (provided that MPD is actually playing a song)
#--------------------------------------------------------------------------------------------------#
# See 'source password' in /etc/icecast.xml for password                                           #
#--------------------------------------------------------------------------------------------------#
    password		"hackme"
#--------------------------------------------------------------------------------------------------#
# Use 'bitrate' or 'quality'. Use 'bitrate' for static bitrate, or 'quality' for variable.         #
# Use one or the other -- not both!                                                                #
# bitrate                     "128"                                                                #
#--------------------------------------------------------------------------------------------------#
    quality			"10"
#--------------------------------------------------------------------------------------------------#
# 'format' is the audio format of the stream. The first number is the sample rate in Hertz (Hz);   #
# 44100 Hz is equal to cd quality. The second number is the number of bits per sample; again 16    #
# bits is the quality used on cd's. The third number is the number of channels; 1 channel is       #
# mono, 2 channels is stereo.                                                                      #
#--------------------------------------------------------------------------------------------------#
    format			"44100:16:2"
    protocol		"icecast2"		# optional
    user			"source"		# optional
    #description		"My Stream Description"	# optional
    #genre			"jazz"			# optional
    #public			"no"			# optional
    #timeout			"2"			# optional
}
replaygain			"track"
volume_normalization "yes"
filesystem_charset		"UTF-8"
id3v1_encoding			"ISO-8859-1"

As you can see, much of the configuration is exactly the same as my first MPD config file. You’d want it like this, since you’d want it to use the same music collection and list of playlists. (Well, if you had a private stash of music you didn’t want anyone in your LAN to know about, you’d point to a different music folder.) So the music directory, playlist directory, and database file are all the same. However, you’d want a different set of logfiles for this second MPD instance, so those values are changed. The same goes to the pid and state files. The port number is 6601, so that it doesn’t conflict with my first MPD. The final difference is the audio output, which is set to “shout” and not “alsa”. The “shout” output works with icecast, and the values in here, like port 8000 and “hackme” as the password, are taken straight from the default icecast intallation’s xml file (/etc/icecast.xml — of course, you can call icecast with a different config file location than the default one).

It’s important that you make this second MPD output solely to icecast. If you put in alsa output here, you’ll end up with two different songs playing simultaneously on your desktop’s speakers! (Aside: MPD is perfectly capable of outputting to multiple audio outputs, so you could, if you wanted, make one MPD instance output to both alsa and icecast — suitable if you’re the only one at home and you want your music to “follow” you to all the computers connected to the LAN.) As for icecast itself, you can also call it from your .xinitrc file, after your mpd-calling lines, like so: icecast -b -c /home/shinobu/syscfg/icecast/cfg.xml. The -b flag daemonizes icecast (required since you’re calling it from your .xinitrc file), and the -c flag merely points to the correct config file location. My icecast config file looks like this:

<icecast>
    <limits>
        <clients>100</clients>
        <sources>2</sources>
        <threadpool>5</threadpool>
        <queue-size>524288</queue-size>
        <client-timeout>30</client-timeout>
        <header-timeout>15</header-timeout>
        <source-timeout>10</source-timeout>
        <burst-on-connect>1</burst-on-connect>
        <burst-size>65535</burst-size>
    </limits>

    <authentication>
        <source-password>hackme(~</source-password>
        <relay-password>hackme(~</relay-password>
        <admin-user>admin</admin-user>
        <admin-password>hackme(~</admin-password>
    </authentication>
    <hostname>localhost</hostname>
    <listen-socket>
        <port>8000</port>
    </listen-socket>
    <fileserve>1</fileserve>

    <paths>
        <basedir>/usr/share/icecast</basedir>
             <logdir>/home/shinobu/.icecast-untracked/log</logdir>
             <webroot>/home/shinobu/.icecast-untracked/icecast/web</webroot>
             <adminroot>/home/shinobu/.icecast-untracked/icecast/admin</adminroot>
        <alias source="/" dest="/status.xsl"/>
    </paths>

    <logging>
        <accesslog>access.log</accesslog>
        <errorlog>error.log</errorlog>
      	<loglevel>1</loglevel>
      	<logsize>10000</logsize>
    </logging>
    <security>
        <chroot>0</chroot>
    </security>
</icecast>

Here, the value for the <source-password> tag must match the password specified in your icecast-MPD’s config file. I’ve left it to the default “hackme” here for demonstration purposes — you should change it to something more secure (better safe than sorry!). My password is like @#$k23lsdf9. Make sure the directory in <logdir> exists. As for the <webroot> and <adminroot> directories, copy them straight out of /usr/share/icecast/web and /usr/share/icecast/admin, and place them into your custom location. This will take care of those annoying permission errors that users often face.

Now, on your remote computer, such as your laptop, you need (1) an MPD client, and (2) a music player, like mplayer, to intercept the raw .ogg music stream from your desktop with the icecast-MPD setup. In my ncmpcpp config file on my laptop, I have custom values for the mpd_host and mpd_port fields to be my desktop’s IP address 192.168.0.110 and 6601, respectively (matching the values in the second MPD config file above). So when I fire up ncmpcpp on my laptop, it connects over the LAN network into the music collection residing in my desktop, all with the magic of MPD. Once I activate the ogg stream by playing a file, I can then access this stream with mplayer — specifically, with mplayer -ao alsa -softvol -volume 10 -volstep 1 -prefer-ipv4 http://192.168.0.110:8000/mpd.ogg. I have this long command aliased, of course, to a hotkey in my XMonad configuration file. And within a couple seconds, I get a top-quality, ~500 kbps ogg stream right on my laptop (the quality setting in my MPD config file, set to “10”, is the maximum quality supported by the ogg format; 500 kbps, or a little over 60 KBps, should be nothing on your home ethernet LAN).

So here’s what’s happening with this second MPD instance on my desktop, configured with icecast output: MPD keeps tabs on what my music collection consists of from port 6601, and “plays” to port 8000 — converting my current mp3/flac file into an ogg stream, which is then served by icecast. From my laptop, I then connect to this MPD by the desktop’s IP address and MPD port of 6601 with ncmpcpp. Then, once I tell MPD to start playing, I can then access the ogg stream with mplayer on my laptop. Mplayer simply looks to http://192.168.0.110:8000/mpd.ogg. When icecast on my desktop receives this request from my laptop’s mplayer, it gives the go-ahead and essentially acts as a bridge between my desktop’s icecast-output MPD and my laptop’s mplayer. Mplayer then takes this ogg stream, and outputs it onto my laptop’s speakers using the alsa drivers.

Here’s a picture to describe what’s going on. (The latest LibreOffice Drawing app is really a pleasure to use!)

So I hope this post proves useful to people who are new to Linux and are struggling to get MPD (and especially with Icecast) to work.

EDIT July 17, 2010: Prettier/simpler graphic uploaded.

UPDATE April 7, 2011: Cleaner source code format; fixed some typos; updated graphic.

UPDATE November 1, 2011: Grammar fixes and some rewordings. I’m delighted to say that even after 2 years of using MPD, I still use the setup described in this post. Talk about configuration stability!