Use your /etc/hosts file to save time

I was installing Arch Linux on yet another machine (new laptop), and was pleasantly surprised by a little tidbit in the Arch Wiki pages — specifically the Beginner’s Guide (which is continually improved and updated fanatically by the Arch community). In short, you can alias host addresses in your /etc/hosts file. This can be for web addresses (www.google.com) or addresses on your LAN. Here is the example and explanation given by the wiki page:

“Tip: For convenience, you may also use /etc/hosts aliases for hosts on your network, and/or on the Web, e.g.:

64.233.169.103   www.google.com   g
192.168.1.90   media
192.168.1.88   data

The above example would allow you to access google simply by typing ‘g’ into your browser, and access to a media and data server on your network by name and without the need for typing out their respective IP addresses.”

Putting your hostname aliases in your /etc/hosts file is great, since all of your programs use it. Firefox, your terminal/shell, git, etc. They all reference this file. So if you’re using git, you can do git clone ssh://username@hostname-alias/… to clone a repo from your LAN, as I described in my post here, instead of ssh://username@192.168.0.2/…. So now all of your hosts on your LAN will be much easier to remember. Here’s my /etc/hosts file:

#
# /etc/hosts: static lookup table for host names
#
#<ip-address>    <hostname.domain.org>    <hostname>
127.0.0.1        aether.localdomain    aether
# aliases
192.168.0.20   exelion
192.168.0. 21 luxion.e
192.168.0.22   luxion.w
# End of file

I’ve aliased luxion.e to my laptop’s ethernet adapter, and luxion.w to its wireless adapter. After setting up your aliases in /etc/hosts, you should get in the habit of writing out your (sane) hostnames for all your LAN computers, instead of painfully writing out their numeric IP addresses. (And I’m sure you’d want to alias your hostnames even more with the advent of IPv6 — and its rather long IP address format.)

Clean Japanese and Korean Characters in URXVT

I have files and directories using Japanese and Korean. Unfortunately, urxvt by default does not display these characters nicely (the hangul looks especially ugly) by default. The way to fix this is to have a good set of TrueType fonts for both Japanese and Korean, and to have URXVT use these as defaults.

If you’re on Arch Linux like me, you can install from the AUR the ttf-kochi-substitute and ttf-baekmuk fonts for Japanese and Korean fonts, respectively. (There is also ttf-sazanami and ttf-unfonts-core for more Japanese and Korean fonts, respectively — but here I’m going to use Kochi Gothic and Baekmuk Gulim in URXVT). Now, in your ~/.Xdefaults file, put this in:

urxvt*font: xft:Terminus:pixelsize=14,\
            xft:Kochi Gothic:antialias=false,\
            xft:Baekmuk Gulim

This makes it so that Terminus is used, then Kochi Gothic, then Baekmuk Gulim. It’s good to have Kochi Gothic on a higher priority than Baekmuk Gulim, since Kochi Gothic’s kanji glyphs look much better than Baekmuk’s (and since Japanese words often have kanji in them, whereas Korean files almost always have just hangul in them). Also, the pixelsize defined with Terminus is used for all succeeding fonts below. Now my URXVT looks really nice!

Adding Replay Gain in Linux Automatically

UPDATE February 1, 2010: This post is now somewhat obsolete, as mp3gain supports writing to ID3v2 tags instead of APE tags. Read the warning below.

UPDATE July 17, 2010: WARNING: mp3gain version 1.5.1 (latest stable command line version) seems to corrupt ID3v2 tag information with the “-s i” option (write info into ID3v2 tags instead of default APE tags). For me, it seems to corrupt the JPEG image data inside the ID3v2 tag for some reason. If you care about the existing ID3v2 tags in your mp3 files, do NOT use mp3gain on them with the “-s i” option (the “-s a” option which is the default and only writes to APE tags, does not do any harm if you only care about APE tags). A good tool to check your tag info (ID3v2, APE, etc.) is with MP3 Diags (if you’re on Arch Linux, get it here). So for now, the method below is still my preferred way of doing things (mp3gain with APE and then ape2id3.py).

Replay gain tags in mp3, flac, and ogg files lets the player adjust the volume accordingly to make the song sound not too loud and not too soft. The Music Player Daemon (aka MPD), my favorite music player, recognizes replay gain tags if present. However, for mp3 files, the popular APE format for replay gain tags are not supported; MPD can only read ID3 tags for replay gain for mp3’s. Luckily, an mp3 file can have both APE and ID3 tags. This means that we can use mp3gain (a cute, simple command-line app available in pretty much every Linux distro) to add APE tags into our mp3s with:

mp3gain [file(s)]

This will add APE replay gain tags into the mp3 file(s) chosen. Then, we can use a simple script to read these APE replay gain tags, convert them into ID3 tags, and put these ID3 tags into the same mp3s. Such a script, thankfully, already exists here. Now, the mp3 file will have both APE and ID3 tags for replay gain values! And MPD will happily use the ID3 values.

If you’re in a hurry, you’d automate this process for entire directories, recursively. That’s what this most excellent Linux page on replay gain page suggests. (You should REALLY bookmark that page, as it has the best information hands down about replay gain in Linux.) I’ve modified the scripts for metaflac there to make it work with mp3’s instead. Here are the two scripts:

Script A:

#!/bin/bash
# Define error codes
ARGUMENT_NOT_DIRECTORY=10
FILE_NOT_FOUND=11
# Check that the argument passed to this script is a directory.
# If it's not, then exit with an error code.
if [ ! -d "$1" ]
then
    echo -e "33[1;37;44m Arg "$1" is NOT a directory!33[0m"
    exit $ARGUMENT_NOT_DIRECTORY
fi
echo -e "33[1;37m********************************************************33[0m"
echo -e "33[1;37mCalling tag-mp3-with-rg.sh on each directory in:33[0m"
echo -e "33[1;36m"$1"33[0m"
echo ""
find "$1" -type d -exec ~/syscfg/shellscripts/replaygain/mp3/tag-mp3-with-rg.sh '{}' \;

Script B (the ‘tag-mp3-with-rg.sh’ script referenced above in Script A):

#!/bin/bash
# Error codes
ARGUMENT_NOT_DIRECTORY=10
FILE_NOT_FOUND=11
# Check that the argument passed to this script is a directory.
# If it's not, then exit with an error code.
if [ ! -d "$1" ]
then
    echo -e "33[1;37mArg "$1" is NOT a directory!33[0m"
    exit $ARGUMENT_NOT_DIRECTORY
fi
# Count the number of mp3 files in this directory.
mp3num=`ls "$1" | grep -c \\.mp3`
# If no mp3 files are found in this directory,
# then exit without error.
if [ $mp3num -lt 1 ]
then
    echo -e "33[1;33m"$1" 33[1;37m--> (No mp3 files found)33[0m"
    exit 0
else
    echo -e "33[1;36m"$1" 33[1;37m--> (33[1;32m"$mp3num"33[1;37m mp3 files)33[0m"
fi
# Run mp3gain on the mp3 files in this directory.
echo -e ""
echo -e "33[1;37mForcing (re)calculation of Replay Gain values for mp3 files and adding them as APE2 tags into the mp3 file...33[0m"
echo -e ""
# first delete any APE replay gain tags in the files
mp3gain -s d "$1"/*.mp3
# add fresh APE tags back into the files
mp3gain "$1"/*.mp3
echo -e ""
echo -e "33[1;37mDone.33[0m"
echo -e ""
echo -e "33[1;37mAdding ID3 tags with the same calculated info from above...33[0m"
echo -e ""
# the -d is for debug messages if there are any errors, and the -f is for overwriting any existing ID3 replay gain tags
~/syscfg/shellscripts/replaygain/mp3/ape2id3.py -df "$1"/*.mp3
echo -e ""
echo -e "33[1;37mDone.33[0m"
echo -e ""
echo -e "33[1;37mReplay gain tags (both APE and ID3) successfully added recursively.33[0m"
echo -e ""

And here is the APE to ID3 conversion script from the link above (the ‘ape2id3.py’ script called from Script B):

#! /usr/bin/env python

import sys
from optparse import OptionParser

import mutagen
from mutagen.apev2 import APEv2
from mutagen.id3 import ID3, TXXX

def convert_gain(gain):
   if gain[-3:] == " dB":
       gain = gain[:-3]
   try:
       gain = float(gain)
   except ValueError:
       raise ValueError, "invalid gain value"
   return "%.2f dB" % gain

def convert_peak(peak):
   try:
       peak = float(peak)
   except ValueError:
       raise ValueError, "invalid peak value"
   return "%.6f" % peak

REPLAYGAIN_TAGS = (
   ("mp3gain_album_minmax", None),
   ("mp3gain_minmax", None),
   ("replaygain_album_gain", convert_gain),
   ("replaygain_album_peak", convert_peak),
   ("replaygain_track_gain", convert_gain),
   ("replaygain_track_peak", convert_peak),
)

class Logger(object):
   def __init__(self, log_level, prog_name):
       self.log_level = log_level
       self.prog_name = prog_name
       self.filename = None

   def prefix(self, msg):
       if self.filename is None:
           return msg
       return "%s: %s" % (self.filename, msg)

   def debug(self, msg):
       if self.log_level >= 4:
           print self.prefix(msg)

   def info(self, msg):
       if self.log_level >= 3:
           print self.prefix(msg)

   def warning(self, msg):
       if self.log_level >= 2:
           print self.prefix("WARNING: %s" % msg)

   def error(self, msg):
       if self.log_level >= 1:
           sys.stderr.write("%s: %s\n" % (self.prog_name, msg))

   def critical(self, msg, retval=1):
       self.error(msg)
       sys.exit(retval)

class Ape2Id3(object):
   def __init__(self, logger, force=False):
       self.log = logger
       self.force = force

   def convert_tag(self, name, value):
       pass

   def copy_replaygain_tag(self, apev2, id3, name, converter=None):
       self.log.debug("processing '%s' tag" % name)

       if not apev2.has_key(name):
           self.log.info("no APEv2 '%s' tag found, skipping tag" % name)
           return False
       if not self.force and id3.has_key("TXXX:%s" % name):
           self.log.info("ID3 '%s' tag already exists, skpping tag" % name)
           return False

       value = str(apev2[name])
       if callable(converter):
           self.log.debug("converting APEv2 '%s' tag from '%s'" %
                          (name, value))
           try:
               value = converter(value)
           except ValueError:
               self.log.warning("invalid value for APEv2 '%s' tag" % name)
               return False
           self.log.debug("converted APEv2 '%s' tag to '%s'" % (name, value))

       id3.add(TXXX(encoding=1, desc=name, text=value))
       self.log.info("added ID3 '%s' tag with value '%s'" % (name, value))
       return True

   def copy_replaygain_tags(self, filename):
       self.log.filename = filename
       self.log.debug("begin processing file")

       try:
           apev2 = APEv2(filename)
       except mutagen.apev2.error:
           self.log.info("no APEv2 tag found, skipping file")
           return
       except IOError:
           e = sys.exc_info()
           self.log.error("%s" % e[1])
           return

       try:
           id3 = ID3(filename)
       except mutagen.id3.error:
           self.log.info("no ID3 tag found, creating one")
           id3 = ID3()

       modified = False
       for name, converter in REPLAYGAIN_TAGS:
           copied = self.copy_replaygain_tag(apev2, id3, name, converter)
           if copied:
               modified = True
       if modified:
           self.log.debug("saving modified ID3 tag")
           id3.save(filename)

       self.log.debug("done processing file")
       self.log.filename = None

def main(prog_name, options, args):
   logger = Logger(options.log_level, prog_name)
   ape2id3 = Ape2Id3(logger, force=options.force)
   for filename in args:
       ape2id3.copy_replaygain_tags(filename)

if __name__ == "__main__":
   parser = OptionParser(version="0.1", usage="%prog [OPTION]... FILE...",
                         description="Copy APEv2 ReplayGain tags on "
                                     "FILE(s) to ID3v2.")
   parser.add_option("-q", "--quiet", dest="log_level",
                     action="store_const", const=0, default=1,
                     help="do not output error messages")
   parser.add_option("-v", "--verbose", dest="log_level",
                     action="store_const", const=3,
                     help="output warnings and informational messages")
   parser.add_option("-d", "--debug", dest="log_level",
                     action="store_const", const=4,
                     help="output debug messages")
   parser.add_option("-f", "--force", dest="force",
                     action="store_true", default=False,
                     help="force overwriting of existing ID3v2 "
                          "ReplayGain tags")
   prog_name = parser.get_prog_name()
   options, args = parser.parse_args()

   if len(args) < 1:
       parser.error("no files specified")

   try:
       main(prog_name, options, args)
   except KeyboardInterrupt:
       pass

# vim: set expandtab shiftwidth=4 softtabstop=4 textwidth=79:

It’s pretty simple. Script A just calls Script B recursively on every directory found inside the designated directory. Script B finds mp3 files, and first tags them with APE replay gain tags with mp3gain. Then, it calls the APE to ID3 conversion script above to add equivalent ID3 tags into those same mp3s. I’ve modified Script B so that it first deletes any APE replay gain tags already present in the mp3 file before doing the replay gain calculations — but this is optional. I also added a bunch of ANSI color escape codes to Script A and B so that they look prettier. These three scripts work beautifully well together with mp3 files inside directories. However, the directories MUST be album directories, as all mp3 files found in a directory are treated as having come from the same album for album replay gain tags (track replay gain tags are always independent on a per-file basis).

I should probably rewrite Script A and B in Python to make it easier to maintain — but everything is pretty simple as it is. If you have 1 big folder full of mp3’s from different artists/albums, then you could change the

mp3gain "$1"/*.mp3

in Script B into something like

for file in $mp3files
do
    mp3gain "$file"
done

This way, mp3gain is called separately for each mp3 file (instead of being called once for all mp3 files in the folder). Now you don’t have to worry about the mp3’s in that folder being treated as having come from 1 album for those album gain tags. To top things off, you should edit your shell’s config (e.g., .zshrc), and alias Script A to something easy, like rgmp3, so that you can just do

rgmp3 [directory]

to get this whole thing to work. Now run this command on your master mp3 folder, take a nap, and come back. All your mp3’s will now have both APE and ID3 replay gain tags!

I hope people find this useful. I’ve googled and googled countless times about replay gain in the past, and until I discovered the excellent link mentioned above a couple days ago, I could never really get replay gain tags working for my mp3’s.

MPD – A Brief Guide

The Music Player Daemon, aka MPD, is a great little application to handle music on your linux computer. Until recently (when Linux really started to take off and gain popularity), MPD has been notoriously difficult to get to work properly. I decided to write this small intro to MPD after having used it for about 6 months. This is a little guide to it to get you started on using MPD as your music manager.

The World of MPD

First, what MPD is:

  • MPD keeps track of your music library. It does this by defining a music directory, to which you can easily add symlinks to all the various places where you store your music on your computer.
  • MPD tracks changes to your music library (any time an mp3 is removed, or a new album added, you simply update the MPD database file — which takes three or four seconds at most).
  • MPD can store playlists.
  • MPD can play songs, and output this information to various devices, such as your physical speakers (usually via your alsa sound drivers, but it can be any audio driver on your system), or even a http port for remote web listening (via icecast).

What MPD is NOT (what most people get confused about):

  • MPD cannot by itself determine: which songs are played and in what order. Nor does it come with any controls such as PAUSE, STOP, NEXT TRACK, etc. MPD, quite simply, can either: play a song, or do nothing while it is running. MPD’s behavior is controlled by clients, such as GMPC, Sonata, etc (my personal favorite is ncmpcpp).

The downside of this dichotomy between what MPD is and what it is not, is that you must install two applications to get the same functionality as any standard multimedia player like iTunes. However, there are many, many advantages of splitting up music-playing from music-controls. First, mpd will hardly crash and will be error-free due to its simple nature. Second, you gain the ability to access your large music collection from other computers in your home network without much fuss. This means that you can do things like: make your laptop act as a remote control for your desktop’s music output (the desktop’s speakers), or even create multiple MPD instances for specific purposes simultaneously (e.g., alsa and also icecast output).

Typical Setup

For most people, they only need one instance of MPD running and have it configured so that it plays songs directly to their computer speakers. So their MPD configuration file will look something like this (comment lines removed to save space):

music_directory "/home/shinobu/.mpd-untracked/music"
playlist_directory "/home/shinobu/.mpd-untracked/playlist"
db_file "/home/shinobu/.mpd-untracked/mpd.db"
log_file "/home/shinobu/.mpd-untracked/mpd.log"
pid_file "/home/shinobu/.mpd-untracked/mpd.pid"
state_file "/home/shinobu/.mpd-untracked/mpdstate"
sticker_file "/home/shinobu/.mpd-untracked/sticker.sql"
user "shinobu"
bind_to_address		"192.168.0.110" # bind it to this computer's IP address if someone on the LAN wants to use this mpd
port				"6600"
save_absolute_paths_in_playlists "yes"
follow_outside_symlinks "yes"
follow_inside_symlinks "yes"
audio_output {
    type			"alsa"
    name			"Exelion ALSA output"
    format			"44100:16:2"	# optional
    mixer_device	"default"
    mixer_control	"PCM"
}
replaygain			"track"
volume_normalization "yes"
filesystem_charset		"UTF-8"
id3v1_encoding			"ISO-8859-1"

It’s very straightforward. All of the essential files and folders are customized and defined in the config file. (My mpd directory is .mpd-untracked, since these files/folders are not tracked by my git folder containing all of my dotfiles. See my previous two posts for more info on that setup.) The config file itself resides at /home/shinobu/syscfg/mpd/cfg-alsa — a customized path and filename, which is referenced each time MPD starts on bootup (in my .xinitrc file), with the command “mpd /home/shinobu/syscfg/mpd/cfg-alsa”. MPD will daemonize itself when it is called, so calling it from your .xinitrc file without the “&” argument at the end of the line is OK. Notice how the above configuration avoids running into permission problems, since everything resides under a normal user’s directory.

I then start up ncmpcpp, which by default looks to the mpd host as localhost and port as 6600 — matching the (default) values in my MPD config file above. And it all works rather beautifully.

Multi-MPD Setup for Music Sharing

Back when I still used iTunes (about 3 years ago), I remember it had this neat feature to share your music with anyone on your local network. You can do the same thing with MPD. The trick is to add a separate MPD instance for this feature. I myself have this multi-MPD setup on my desktop, with one MPD for alsa-only output (as seen above), and a second MPD for icecast-only output. Icecast, by the way, is a more general solution not wholly related to MPD for sharing music over the internet — we can use icecast since it plays rather well with MPD (UNIX philosophy FTW!). Since my desktop has a second MPD instance so that it works with icecast, my laptop can now take advantage of the huge music collection that I have on it, remotely (well, as long as I’m on the home network in my case).

Here’s my MPD config file for icecast output:

music_directory "/home/shinobu/.mpd-untracked/music"
playlist_directory "/home/shinobu/.mpd-untracked/playlist"
db_file "/home/shinobu/.mpd-untracked/mpd.db"
log_file "/home/shinobu/.mpd-untracked/mpd-icecast.log"
pid_file "/home/shinobu/.mpd-untracked/mpd-icecast.pid"
state_file "/home/shinobu/.mpd-untracked/mpdstate-icecast"
sticker_file "/home/shinobu/.mpd-untracked/sticker.sql"
user "shinobu"
bind_to_address		"192.168.0.110" # bind it to this computer's IP address if someone on the LAN wants to use this mpd
port				"6601"
save_absolute_paths_in_playlists "yes"
follow_outside_symlinks "yes"
follow_inside_symlinks "yes"
audio_output {
    type			"shout"
    name			"exelion HD mpd stream"
    encoding		"ogg"			# optional
    host			"localhost"
    port			"8000"
    mount			"/mpd.ogg" # i.e., "http://192.168.0.110/mpd.ogg" is the live stream
                                           # (provided that MPD is actually playing a song)
#--------------------------------------------------------------------------------------------------#
# See 'source password' in /etc/icecast.xml for password                                           #
#--------------------------------------------------------------------------------------------------#
    password		"hackme"
#--------------------------------------------------------------------------------------------------#
# Use 'bitrate' or 'quality'. Use 'bitrate' for static bitrate, or 'quality' for variable.         #
# Use one or the other -- not both!                                                                #
# bitrate                     "128"                                                                #
#--------------------------------------------------------------------------------------------------#
    quality			"10"
#--------------------------------------------------------------------------------------------------#
# 'format' is the audio format of the stream. The first number is the sample rate in Hertz (Hz);   #
# 44100 Hz is equal to cd quality. The second number is the number of bits per sample; again 16    #
# bits is the quality used on cd's. The third number is the number of channels; 1 channel is       #
# mono, 2 channels is stereo.                                                                      #
#--------------------------------------------------------------------------------------------------#
    format			"44100:16:2"
    protocol		"icecast2"		# optional
    user			"source"		# optional
    #description		"My Stream Description"	# optional
    #genre			"jazz"			# optional
    #public			"no"			# optional
    #timeout			"2"			# optional
}
replaygain			"track"
volume_normalization "yes"
filesystem_charset		"UTF-8"
id3v1_encoding			"ISO-8859-1"

As you can see, much of the configuration is exactly the same as my first MPD config file. You’d want it like this, since you’d want it to use the same music collection and list of playlists. (Well, if you had a private stash of music you didn’t want anyone in your LAN to know about, you’d point to a different music folder.) So the music directory, playlist directory, and database file are all the same. However, you’d want a different set of logfiles for this second MPD instance, so those values are changed. The same goes to the pid and state files. The port number is 6601, so that it doesn’t conflict with my first MPD. The final difference is the audio output, which is set to “shout” and not “alsa”. The “shout” output works with icecast, and the values in here, like port 8000 and “hackme” as the password, are taken straight from the default icecast intallation’s xml file (/etc/icecast.xml — of course, you can call icecast with a different config file location than the default one).

It’s important that you make this second MPD output solely to icecast. If you put in alsa output here, you’ll end up with two different songs playing simultaneously on your desktop’s speakers! (Aside: MPD is perfectly capable of outputting to multiple audio outputs, so you could, if you wanted, make one MPD instance output to both alsa and icecast — suitable if you’re the only one at home and you want your music to “follow” you to all the computers connected to the LAN.) As for icecast itself, you can also call it from your .xinitrc file, after your mpd-calling lines, like so: icecast -b -c /home/shinobu/syscfg/icecast/cfg.xml. The -b flag daemonizes icecast (required since you’re calling it from your .xinitrc file), and the -c flag merely points to the correct config file location. My icecast config file looks like this:

<icecast>
    <limits>
        <clients>100</clients>
        <sources>2</sources>
        <threadpool>5</threadpool>
        <queue-size>524288</queue-size>
        <client-timeout>30</client-timeout>
        <header-timeout>15</header-timeout>
        <source-timeout>10</source-timeout>
        <burst-on-connect>1</burst-on-connect>
        <burst-size>65535</burst-size>
    </limits>

    <authentication>
        <source-password>hackme(~</source-password>
        <relay-password>hackme(~</relay-password>
        <admin-user>admin</admin-user>
        <admin-password>hackme(~</admin-password>
    </authentication>
    <hostname>localhost</hostname>
    <listen-socket>
        <port>8000</port>
    </listen-socket>
    <fileserve>1</fileserve>

    <paths>
        <basedir>/usr/share/icecast</basedir>
             <logdir>/home/shinobu/.icecast-untracked/log</logdir>
             <webroot>/home/shinobu/.icecast-untracked/icecast/web</webroot>
             <adminroot>/home/shinobu/.icecast-untracked/icecast/admin</adminroot>
        <alias source="/" dest="/status.xsl"/>
    </paths>

    <logging>
        <accesslog>access.log</accesslog>
        <errorlog>error.log</errorlog>
      	<loglevel>1</loglevel>
      	<logsize>10000</logsize>
    </logging>
    <security>
        <chroot>0</chroot>
    </security>
</icecast>

Here, the value for the <source-password> tag must match the password specified in your icecast-MPD’s config file. I’ve left it to the default “hackme” here for demonstration purposes — you should change it to something more secure (better safe than sorry!). My password is like @#$k23lsdf9. Make sure the directory in <logdir> exists. As for the <webroot> and <adminroot> directories, copy them straight out of /usr/share/icecast/web and /usr/share/icecast/admin, and place them into your custom location. This will take care of those annoying permission errors that users often face.

Now, on your remote computer, such as your laptop, you need (1) an MPD client, and (2) a music player, like mplayer, to intercept the raw .ogg music stream from your desktop with the icecast-MPD setup. In my ncmpcpp config file on my laptop, I have custom values for the mpd_host and mpd_port fields to be my desktop’s IP address 192.168.0.110 and 6601, respectively (matching the values in the second MPD config file above). So when I fire up ncmpcpp on my laptop, it connects over the LAN network into the music collection residing in my desktop, all with the magic of MPD. Once I activate the ogg stream by playing a file, I can then access this stream with mplayer — specifically, with mplayer -ao alsa -softvol -volume 10 -volstep 1 -prefer-ipv4 http://192.168.0.110:8000/mpd.ogg. I have this long command aliased, of course, to a hotkey in my XMonad configuration file. And within a couple seconds, I get a top-quality, ~500 kbps ogg stream right on my laptop (the quality setting in my MPD config file, set to “10”, is the maximum quality supported by the ogg format; 500 kbps, or a little over 60 KBps, should be nothing on your home ethernet LAN).

So here’s what’s happening with this second MPD instance on my desktop, configured with icecast output: MPD keeps tabs on what my music collection consists of from port 6601, and “plays” to port 8000 — converting my current mp3/flac file into an ogg stream, which is then served by icecast. From my laptop, I then connect to this MPD by the desktop’s IP address and MPD port of 6601 with ncmpcpp. Then, once I tell MPD to start playing, I can then access the ogg stream with mplayer on my laptop. Mplayer simply looks to http://192.168.0.110:8000/mpd.ogg. When icecast on my desktop receives this request from my laptop’s mplayer, it gives the go-ahead and essentially acts as a bridge between my desktop’s icecast-output MPD and my laptop’s mplayer. Mplayer then takes this ogg stream, and outputs it onto my laptop’s speakers using the alsa drivers.

Here’s a picture to describe what’s going on. (The latest LibreOffice Drawing app is really a pleasure to use!)

So I hope this post proves useful to people who are new to Linux and are struggling to get MPD (and especially with Icecast) to work.

EDIT July 17, 2010: Prettier/simpler graphic uploaded.

UPDATE April 7, 2011: Cleaner source code format; fixed some typos; updated graphic.

UPDATE November 1, 2011: Grammar fixes and some rewordings. I’m delighted to say that even after 2 years of using MPD, I still use the setup described in this post. Talk about configuration stability!

Unified Configuration File Setup Across Multiple Machines – Revisited

My last post discussed how to get your config files, known as “dotfiles,” synchronized across multiple machines using a rudimentary makefile and git. I said that I hoped of achieving the “one folder for all config files” dream. I have achieved it, and it is pretty simple. Also, I will discuss how to handle files with passwords in them, and some other thoughts on this setup.

Keep track of system files, too

Essentially, my last post left out all but the system configuration files, such as /etc/fstab and the like. The /etc folder is owned by root, as well as the /boot folder. My first approach was to simply replace all such files with more symlinks, which would point to files owned by the normal user of the system. This approach had its drawbacks: (1) not all system files are symlinkable (e.g., /etc/sudoers is particularly security-conscious), and (2) the idea of deleting system files and replacing them with symlinks, on its face, sounded like I was setting myself up for a big grand screw-up.

So I thought: “Well, since system files are seldom ever edited anyway, why not just back them up periodically?” And that’s what I did instead. Now my makefile, as discussed in the previous post, has a section like this:

# copy contents of system files to keep track of them
syscopy:
ifeq ('$(HOSTNAME)','exelion')
	cat /boot/grub/menu.lst >       /home/shinobu/syscfg/sys/boot-grub-menu.lst-exelion
	cat /etc/X11/xorg.conf >        /home/shinobu/syscfg/sys/etc-X11-xorg.conf-exelion
	cat /etc/fstab >                /home/shinobu/syscfg/sys/etc-fstab-exelion
	cat /etc/hosts >                /home/shinobu/syscfg/sys/etc-hosts-exelion
	cat /etc/inittab >              /home/shinobu/syscfg/sys/etc-inittab-exelion
	cat /etc/makepkg.conf >         /home/shinobu/syscfg/sys/etc-makepkg.conf-exelion
	cat /etc/rc.conf >              /home/shinobu/syscfg/sys/etc-rc.conf-exelion
	cat /etc/rc.local >             /home/shinobu/syscfg/sys/etc-rc.local-exelion
	cat /etc/rc.local.shutdown >    /home/shinobu/syscfg/sys/etc-rc.local.shutdown-exelion
	cat /etc/yaourtrc >             /home/shinobu/syscfg/sys/etc-yaourtrc-exelion
	cat /etc/sudoers >              /home/shinobu/syscfg/sys/etc-sudoers-exelion # requires superuser privileges to read!
else
	cat /boot/grub/menu.lst >       /home/shinobu2/syscfg/sys/boot-grub-menu.lst-luxion
	cat /etc/X11/xorg.conf >        /home/shinobu2/syscfg/sys/etc-X11-xorg.conf-luxion
	cat /etc/fstab >                /home/shinobu2/syscfg/sys/etc-fstab-luxion
	cat /etc/hosts >                /home/shinobu2/syscfg/sys/etc-hosts-luxion
	cat /etc/inittab >              /home/shinobu2/syscfg/sys/etc-inittab-luxion
	cat /etc/makepkg.conf >         /home/shinobu2/syscfg/sys/etc-makepkg.conf-luxion
	cat /etc/network.d/luxion-wired > /home/shinobu2/syscfg/sys/etc-network.d-luxion-wired
	cat /etc/network.d/luxion-wireless-home-nopassword > /home/shinobu2/syscfg/sys/etc-network.d-luxion-wireless-home-nopassword
	cat /etc/rc.conf >              /home/shinobu2/syscfg/sys/etc-rc.conf-luxion
	cat /etc/rc.local >             /home/shinobu2/syscfg/sys/etc-rc.local-luxion
	cat /etc/rc.local.shutdown >    /home/shinobu2/syscfg/sys/etc-rc.local.shutdown-luxion
	cat /etc/yaourtrc >             /home/shinobu2/syscfg/sys/etc-yaourtrc-luxion
	cat /etc/sudoers >              /home/shinobu2/syscfg/sys/etc-sudoers-luxion
endif

I have in my /etc/rc.local the command “make -f /path/to/the/above/makefile -B syscopy“. So every time my system boots up, all of the config files are copied into their backup-equivalents in the syscfg/sys folder. Since git tracks changes in the syscfg folder, only changes in the config files are detected and tracked as changes (i.e., git doesn’t track changes in file modification times, which is a good thing here for our purposes — otherwise git would be saying that every time we boot up all of our system files have changed!). So now all of my system config files are tracked passively (by merely reading off them). Of course, if I manually edit a system file, I can still call make -B syscopy myself manually, and then run git diff in the syscfg folder to track those changes, and then git commit to solidify those changes into the git history.

For config files with passwords in them

DO NOT EVER PUT CONFIG FILES WITH PASSWORDS INTO YOUR TRACKED DOTFILES FOLDER! Not only does this mean that your password, in plain text, is tracked by git, but that should you ever change your password, git will notice the changes and track them as well! This will give anyone who gets access to your git repo a complete, timestamped history of your passwords for your applications (like icecast, irssi, etc.) So to get around this problem, I have it set up so that I have a copy of the password-containing config file, minus the passwords in them. Whenever I make changes to the original password-containing files, I update the changes into the copies, and then track these copies in git, not the originals.

Not all config files need symlinks

In my last post, I discussed how creating symlinks via commands in the makefile was the key to this whole setup. But for some (smarter) applications, symlinks are not needed, since they can intelligently be told which config file to use. Alpine, icecast, irssi, and mpd are like this, so I just have config files for them inside my syscfg folder, and just run these apps (which are all autostarted for me each time on boot) with commandline parameters pointing to the non-default config file locations.

Unified Configuration File Setup Across Multiple Machines

SUMMARY: This post shows you how to sync multiple configuration files across multiple hosts with git and a makefile.

INTRODUCTION

If you have 20 different so-called ‘dotfiles’ like me, they can get difficult to keep track of. It can be even more difficult if you have multiple computers that you use often, and if you want them all to be updated to your latest settings.

For myself, I need to keep track of:

  • .gitconfig
  • .zshrc
  • .zsh (folder)
  • .vim (folder)
  • .vimrc
  • .gvimrc
  • .vimperatorrc
  • .Xdefaults
  • .boxes
  • .xmonad (folder)
  • .xmonad/init.sh (custom init script that XMonad is told to call in the startup hook)
  • .xmonad/xmonad.hs
  • shellscripts (folder which has a growing number of custom shell scripts that I like to use every now and then, or at least keep as a reference on both my desktop and laptop)
  • .xinitrc

Of course, this list will grow over time, as I start to learn more things and begin using more programs. What I want to do is (1) copy these files over to any other host that I use/own automatically and sync back to all the other machines any changes that I make to any one particular machine; and (2) have a unified config file structure, with a directory name for the application/setting, and a simple file called ‘cfg’ for the config file (which will be symlinked to what the application thinks is the true, appropriate location of the config file). The great thing about these two issues is that there is a simple, durable solution for these two precise concerns: git and make. So enough blabbering, let’s get to it!

Step 1: Put all config files into a new directory

It doesn’t matter where your new directory (let’s call it syscfg) is located. Move all of your config files that you want to keep track of into this directory. I suggest you rename all of them to fit some kind of unified naming scheme, and take note of what their former names/destinations used to be. For example, I use syscfg/vim to keep all of my vim things in there (instead of the default .vim, including a file called cfg to act as my .vimrc file).

Per-host (host-specific) settings

I highly suggest that you make all of your config files, such as your .xinitrc (or any other script or some sort) file, include per-host specific settings. Otherwise, you will have the same config settings on all of your systems! I.e., you’d want your .xinitrc to have something like:

do-something-universal-here
HOSTNAME=$(hostname)
case "${HOSTNAME}" in
    hostname1)
    do-something-here
    ;;
    hostname2)
    do-something-else-here
    ;;
    *) # catches all other hostnames
    do-something-else-that's-universal-here
    ;;
esac
do-something-else-that's-universal-here

The above syntax is for bash scripts (files that start with a “#!/bin/bash” at the very first line). If you are using zsh, you could also use this syntax for creating aliases that are specific to a certain host (e.g., my laptop doesn’t have 2 hard drives, so it doesn’t have aliases that point to my mount directory).

If your config file is painful to work with in implementing per-host settings (my xmonad.hs file is like this), you can still achieve per-host settings by making 2 config files, so for example cfg-host1 and cfg-host2, and symlink the correct one to .xmonad/xmonad.hs. You determine the correct config file to symlink to in the makefile. Read on.

Step 2: Create a makefile

Your sysconfig should now have a clean, uniform structure for all of your config files. Now, let’s create a makefile so that the program make can install and uninstall the symlinks as necessary. Here’s what my makefile looks like:

CFGROOT := $(shell pwd)
HOSTNAME := $(shell hostname)
all: boxes git shellscripts vim vimperatorrc xdefaults xinitrc xmonad zsh
boxes:
	ln -fs $(CFGROOT)/boxes/cfg ${HOME}/.boxes
git:
	ln -fs $(CFGROOT)/git/cfg ${HOME}/.gitconfig
shellscripts:
	ln -fs $(CFGROOT)/shellscripts ${HOME}/shellscripts
vim:
	ln -fs $(CFGROOT)/vim ${HOME}/.vim
	ln -fs $(CFGROOT)/vim/cfg ${HOME}/.vimrc
	ln -fs $(CFGROOT)/vim/cfg-gui ${HOME}/.gvimrc
vimperatorrc:
	ln -fs $(CFGROOT)/vimperatorrc/cfg ${HOME}/.vimperatorrc
xdefaults:
	ln -fs $(CFGROOT)/xdefaults/cfg ${HOME}/.Xdefaults
xinitrc:
	ln -fs $(CFGROOT)/xinitrc/cfg ${HOME}/.xinitrc
xmonad:
	ln -fs $(CFGROOT)/xmonad ${HOME}/.xmonad
	ln -fs $(CFGROOT)/xmonad/init ${HOME}/.xmonad/init.sh
#--------------------------------------------------------------------------------------------------#
# Since it's really painful to do a unified config file across multiple hosts in XMonad v. 0.8.1,  #
# I have to do it this way.                                                                        #
#--------------------------------------------------------------------------------------------------#
ifeq ('$(HOSTNAME)','exelion')
	ln -fs $(CFGROOT)/xmonad/cfg ${HOME}/.xmonad/xmonad.hs
else
	ln -fs $(CFGROOT)/xmonad/cfg-luxion ${HOME}/.xmonad/xmonad.hs
endif

zsh:
	ln -fs $(CFGROOT)/zsh ${HOME}/.zsh
	ln -fs $(CFGROOT)/zsh/cfg ${HOME}/.zshrc

uninstall:
	rm ${HOME}/.boxes
	rm ${HOME}/.gitconfig
	rm ${HOME}/.vim
	rm ${HOME}/.vimrc
	rm ${HOME}/.gvimrc
	rm ${HOME}/shellscripts
	rm ${HOME}/.vimperatorrc
	rm ${HOME}/.Xdefaults
	rm ${HOME}/.xinitrc
	rm ${HOME}/.xmonad/init.sh
	rm ${HOME}/.xmonad/xmonad.hs
	rm ${HOME}/.xmonad
	rm ${HOME}/.zsh
	rm ${HOME}/.zshrc

This is where symlinks reveal their beauty. From what I know of Windows XP (and my knowledge is very limited because I hate M$ with a passion), you cannot do something like this. Anyway, the above is fairly obvious and straightforward, isn’t it? All this does is create symlinks, and remove them if desired. Since they are symlinks, you can still do something like “vim ~/.vimrc”, and vim will read (assuming again that our directory is syscfg) syscfg/vim/cfg, with all of the pretty syntax highlighting and so on. The -f flag for the ln command simply makes it create the symlink even if the symlink already exists at the destination. See man ln for more info.

If you run make -B all, it will create all the symlinks defined under the keyword all. (The -B flag, for forcing make to run, is required here given our situation with symlinks.) You could in the alternative select only those config files you wish to install; e.g., make -B xmonad for installing only those symlinks for vim’s config files, or make -B vim zsh for vim and zsh’s config files. Lastly, running make uninstall removes all of the symlinks from your system. Experiment to your delight. (Make sure to have the lines with ln and rm start with a TAB character, as make will otherwise throw an error.)

Also, note how the contents in syscfg do not matter at all in how they are named, since it’s really the with its symlinks that takes care of all the proper “dotfile” namings.

Lastly, since these are all symlinks, you can have in your, say, .gvimrc file, a line that says “source ~/.vim/cfg”, and it will still work since the .vim directory is symlinked to your syscfg/vim (i.e., you don’t have to refer to symlinks once you make the symlinks). This is just a long-winded way of saying that using this symlink approach preserves all of your old config file paths from within your config files.

Step 3: Fire up git

Now, fire up git, add your config files, and sync it across all your computers! Use my post here to do this. The only thing to note here for our purposes is that the makefile is stored under syscfg, and that syscfg is where git should be initialized (with git init). Also, only add the config files and any other files that the config files depend on. An example of a file you should NOT want to add to git is any sort of history file, such as zsh’s history file (specified with the HISTFILE option in zsh’s config file — in our case, syscfg/zsh/cfg), since you’d want different session history files for different machines. Another example would be vim’s session files for the session manager plugin. If you add such temporary history files (or any other file that the application automatically makes changes to), you will make git track these changes (very doable, but utterly worthless)! On the other hand, you’d probably want to add simply script-like files that your configs depend on, such as vim’s various plugins (mere .vim text files in the syscfg/vim directory), or even irssi’s perl plugins, if you use irssi (I don’t use IRC on my laptop, hence it’s exclusion from my sample syscfg and makefile above).

With git taking care of the syncing, you now have complete revision history, as well as guaranteed config file integrity across all of your systems. It’s only a matter of cloning, then simply pushing and pulling for all of your config file syncing needs. Personally, I have it set up so that I just do “sl” to ssh into my laptop (sl is aliased to the unbearably long “ssh username@192.168.0.102”; no password since I have ssh set up that way; again, see my post above to do this), then “d sy[TAB]” (“d” is aliased in my syscfg/zsh/cfg to mean “cd”, and I only have one directory starting with “sys” so I can use zsh’s TAB completion to do the rest), and then “gpl” (extremely shortened form for “git pull” — again, see my post on git to make git accept this instead of “git pull origin master”). Yes, all I do is sl, d sy[TAB], gpl, and my laptop is synced. No more checking/rechecking manually whether certain symlinks exist on my laptop, and whether certain vim plugins already exist on it. I wish I had thought of this sooner, as it would have saved me a lot of time.

Reminders and other tips

  • Make sure that all of your crucial config files (like .xinitrc) work properly before implementing this setup! (It’s not fun fixing things in the virtual console a la CTRL+ALT+.)
  • Make sure to have per-host settings in each of your config files (or, failing that, have your makefile link intelligently to different config files on a per-host basis)
  • In my vim config files above, you’ll notice that I symlink my cfg-gui to .gvimrc. The actual cfg-gui file looks very simple, with a “source ~/.vim/cfg”, and all the gvim-specific commands following that. Make sure to have a check for any autocommands so that they are loaded only once so that gvim works properly. (I must say, I only use vim now, except when I feel like seeing 16+ million colors (GTK) as opposed to 256 (urxvt).)
  • The makefile, and its contents, can be scripted in a different programming language if you don’t want to use make. I’ve noticed that some people use ruby to do this. But it could also be python, perl, bash, or any other script.
  • Since we’re going to end up putting into syscfg most of our config files, it wouldn’t be a bad idea to add files that don’t actually need syncing (see my note on irssi above). You’d just put an if-statement in your makefile to exclude these files for certain hosts. The benefit to this approach is that, you would end up with ONE git repo for ALL of your config files. Even though I’m not at this stage yet, I feel myself inevitably being pulled toward this path. I want complete revision history for all my /etc/X11/xorg.conf, /etc/fstab, /etc/sudoers, and even /boot/grub/menu.lst files, if it’s possible to do so. It’s probably a security risk to symlink to these destinations (file permissions, which git isn’t good at, at least according to what I heard from Linus’s Google Tech Talk from 2007), but I’m the only human who has access (and cares about) the config files on my desktop/laptop anyway. I’ll update this post if I end up achieving this “one config directory to rule them all” dream.

This guide was prepared with the help of various internet websites (google is your friend), and also especially this site.