Xorg: Switching Keyboard Layouts Consistenly and Reliably from Userspace

For some time now, the XkbLayout option (set to “us, fr, de”) in my /etc/X11/xorg.conf.d/10-keyboard.conf has been broken for me. This probably has something to do with my recent xmodmap hacks in ~/.xinitrc, and possibly the interaction with the /etc/X11/xorg.conf.d/10-evdev.conf, which includes an entry to catch all keyboards. I decided to fix this problem today, and with a healthy dose of serendipity, got everything working smoothly again. Sources for my custom scripts are included, as usual.

By accident, I discovered the setxkbmap userland program. It does everything you could possibly do with keyboard layouts. It is maintained by the Xorg project, so it’s pretty much guaranteed to work. Here’s a sample run (using Arch’s xorg-setxkbmap 1.2.0-2 and xorg-server 1.10.1-1):

$ setxkbmap -help
Usage: setxkbmap [args] [<layout> [<variant> [<option> ... ]]]
Where legal args are:
-?,-help            Print this message
-compat <name>      Specifies compatibility map component name
-config <file>      Specifies configuration file to use
-device <deviceid>  Specifies the device ID to use
-display <dpy>      Specifies display to use
-geometry <name>    Specifies geometry component name
-I[<dir>]           Add <dir> to list of directories to be used
-keycodes <name>    Specifies keycodes component name
-keymap <name>      Specifies name of keymap to load
-layout <name>      Specifies layout used to choose component names
-model <name>       Specifies model used to choose component names
-option <name>      Adds an option used to choose component names
-print              Print a complete xkb_keymap description and exit
-query              Print the current layout settings and exit
-rules <name>       Name of rules file to use
-symbols <name>     Specifies symbols component name
-synch              Synchronize request w/X server
-types <name>       Specifies types component name
-v[erbose] [<lvl>]  Sets verbosity (1..10).  Higher values yield
                    more messages
-variant <name>     Specifies layout variant used to choose component names

And here’s a sample query on my machine:

$ setxkbmap -query
rules:      base
model:      pc105
layout:     us

And here’s how to change layouts.

To switch to the “us” keyboard layout:

$ setxkbmap us

To switch to the “fr” layout:

$ setxkbmap fr

To switch to the “de” layout:

$ setxkbmap de

And so on. Very useful, indeed. Here are some things to keep in mind:

  1. The layout change is X server-wide. This means that your layout change will take effect on all windows. Of course, you can limit your layout change with the various options like -display or -geometry, but to me that’s impractical.
  2. You can run this command from within a script, in a sub-shell (backgrounded), and it will still work! This is a nice feature which allows one to script/automate the whole thing (which is what I did).
  3. Unfortunately, running this command clears any xmodmap settings that were present before. For Xmonad/vim/etc. users who like to change up their Caps Lock key, for example, you need to re-run the xmodmap commands to get them back. I’m in this boat.

So, here’s my new setup, taking into account the xmodmap issue noted above:

  • layout_switch.sh: This script simply changes up the layout. It’s a dead-simple zsh script that you can easily port to Bash.
  • xmodmap.sh: This script simply does all the xmodmap-editing business that I need to do to get my Xmonad setup to work.
  • ~/.xmonad/xmonad.hs: Here, I have set up a hotkey sequence to call the layout_switch.sh script. I have to be careful to choose a hotkey that will remain consistent even in different layouts (e.g., I first tried the “\” backslash key, but this key’s location is different in the French layout). This means that I have to depend on keys like Escape, or the F1-F12 function keys. Since I already have F1-F12 mapped to additional workspaces, I chose the Escape key.
  • ~/.xinitrc: Here, I call the layout_switch.sh script to set my keyboard layout explicitly on startup. You could do fancy things, for example, of selecting a different layout on a different day (maybe on Sundays you want to start typing in French only, so you could just script that in here).

And here are the relevant source files for these scripts:

layout_switch.sh:

#!/bin/zsh
# LICENSE: PUBLIC DOMAIN
# switch between us, fr, and de layouts

# If an explicit layout is provided as an argument, use it. Otherwise, select the next layout from
# the set [us, fr, de].
if [[ -n "$1" ]]; then
    setxkbmap $1
else
    layout=$(setxkbmap -query | awk 'END{print $2}')
    case $layout in
        us)
            setxkbmap fr
            ;;
        fr)
            setxkbmap de
            ;;
        *)
            setxkbmap us
            ;;
    esac
fi

# remap Caps_Lock key to Xmonad's exclusive 'mod' key
~/syscfg/script/sys/xmodmap.sh

xmodmap.sh:

#!/bin/zsh
# LICENSE: PUBLIC DOMAIN
# remap Caps_Lock key to Xmonad's exclusive 'mod' key

xmodmap -e "remove Lock = Caps_Lock"
xmodmap -e "add mod3 = Caps_Lock"
xmodmap ~/.xmodmap

~/.xmodmap:

remove Lock = Caps_Lock
remove mod3 = Super_L
keysym Caps_Lock = Super_L
add mod3 = Super_L

~/.xmonad/xmonad.hs:

myKeys :: String -> XConfig Layout -> M.Map (KeyMask, KeySym) (X ())
myKeys hostname conf@(XConfig {XMonad.modMask = modm}) = M.fromList $
...
    -- change keyboard layouts
    , ((modm              , xK_Escape), spawn "/home/shinobu/syscfg/script/sys/layout_switch.sh")
...

~/.xinitrc:

...
# explicitly choose "us" (English) keyboard layout (and set Caps Lock key to
# xmonad's modmap key)
/home/shinobu/syscfg/script/sys/layout_switch.sh us &
...

There are a couple big advantages to this setup:

  • Because everything is handled in userspace (we didn’t touch any Xorg file!), we no longer have to restart the X server to diagnose any problem we encounter with changing the keyboard layout. Gone are the days of fiddling with a “XkbLayout” option (or any other Xorg option) and restarting X just to see if the changes worked.
  • Because everything is handled in userspace, we can use any number of custom shell scripts, programs, etc. to customize how the layout switch is made. For me, since I use Xmonad, I can use Xmonad’s extensive, consistent hotkey definitions to choose which hotkeys to do the layout switch. To be honest, I disliked the options available in the base file (/usr/share/X11/xkb/rules/base) when choosing which hotkeys to do my keyboard layout switching. And, my choice of modm + Escape (i.e., (dead) Caps Lock key + Escape) was impossible to realize with the base file.

Finally, a caveat: the only flaw with the setup above is that if I execute the xmonad.hs layout-switching hotkey too quickly too many times, there’s a chance of two independent processes of layout_switch.sh being spawned at nearly the same time. This means that the call to xmodmap.sh within each of those processes might get interleaved! For me, this has resulted in my Caps lock key becoming permanently turned on, for instance. However, you can always execute the xmodmap.sh script a second or two afterwards to clear this up.

I strongly recommend everyone to do their keyboard switching this way — not only is everything made explicit, but everything is transparent, too: keyboard layout switching isn’t a mysterious process anymore.

UPDATE August 18, 2011: Cleaned up some typos.

UPDATE August 24, 2011: See this post — I no longer use the fr and de layouts, but rather, just the us layout with variant altgr-intl. It’s much, much simpler this way.

A Summary of Linux vs. Windows XP

As you may have figured out by now, I don’t use Windows XP full-time any more. I only use it for one or two games. Anyway, I don’t think I’ve written a post yet comparing Linux (Arch Linux to be specific) to Windows XP. I keep referring to XP, because that’s the last Windows OS I’ve actually used on my own machines. But the points that follow still pertain to the latest incarnation of Windows (Windows 7), and perhaps all future versions of it.

I just had to get these annoyances about XP out of my system, and let you, my dear reader, know about them. So the following is my very biased point of view as a relatively new, SELF-IMPOSED Linux convert:

Linux pros:

  • Unlimited, sane configurability: Everything you can imagine about Linux can be configured with TEXT FILES (or simple TEXT based interfaces), containing HUMAN-READABLE text. There are no strange “registry” files like in Windows, and hence, no need to invoke strange programs like “regedit.exe” to do your bidding. If you know how to use a text editor (what you Windows folks hideously understand as “Notepad”), then nothing can stand in your way. EVERYTHING is exposed by Linux. An extremely simple root vs. non-root user dichotomy ensures sane, “system files” vs. “user files” separation. Windows is living in the stone age when it comes to user rights/management.
  • Few, if any, viruses: Because the consumer-level Linux ecosystem is so small, evil hackers don’t spend time writing virus programs for it. More on this virus/malware/spyware discussion below.
  • Does not slow down or get unstable with uptime: If you leave your Windows computer on for more than a day, it’s bound to slow down. Linux has never done this to me. Ever. Even with uptimes past 5 days.
  • Simplified, unified, and standardized software installation/upgrade/removal: On Windows, individual programs have to check if a newer version of itself is available. There is no standard for this — some programs do it automatically if you’re connected to the internet, some programs ask for your approval (incessantly), some programs have a “Check for updates…” button that you have to manually click on, and some are just standalone .exe files that you need to manually remove and reinstall with the newer version! This is insanity. On Arch (or any other desktop-user distro), you just type in your shell “sudo pacman -Syu” to do a system-wide upgrade. With a single command, you upgrade every single package that you have installed (after transparently downloading them from Arch’s world-wide network of package “repositories”). Here, “package” means either a program, or system files — even the OS kernel! You can optionally choose to upgrade only certain packages, or to ignore some of them, to customize your upgrade. Some Archers upgrade every day, and others wait a month or more to upgrade. There are no nag screens, popups, or “automatic” upgrades/installs that break behind your back should you choose to shut down your computer.
  • Tons of free, industrial-strength software: Actively-developed open source programs are, quite simply, the most reliable, robust programs around. Take for example the Linux Kernel: this project is one of the finest achievements of software engineering to date. Legions of high-profile companies rely on the Linux Kernel to power their servers (and many thousands of Linux desktop users, like myself, benefit from this same, state-of-the-art reliability). And because free, open source software (“FOSS”) developers are humans too, there are mountains of open source programs for your use. Things like text editors, web browsers, email clients, office suites, image editors, movie editors, etc. all exist for free in FOSS-land. And many of these programs are the very best in their field, because they are extremely actively developed (where new releases come in weeks or months, instead of years). The Arch repositories are packed with FOSS programs. Packed, I say!
  • No malware/spyware: In Arch, there are two types of repositories: official and unofficial. The official repository only has packages that have been examined, tested, and uploaded by sanctioned members of the Arch Linux Team. The unofficial repositories are those repositories that have packages uploaded by anyone. The biggest and most popular unofficial repository is the Arch User Repository (“AUR”), which has thousands of packages. As time progresses, popular packages from the AUR collect votes from users, and the most popular ones receive the blessings of the Arch Linux Team and gets adopted into the official repository, and hence undergo regular, official maintenance. In short, the room for spyware/malware to creep into any of the official repositories is virtually zero. Even if you install an AUR package that has 0 votes for it, the de facto AUR client yaourt will still tell you to examine the contents of all installation files (these are, predictably enough in the Linux world, TEXT files) before executing them. Of course, you could go out of your way to download a suspicious-looking executable file and run it, but such a course of action is rare and remote enough (and downright stupid enough) to ignore.
  • No annoyances: Two things: (1) Because 95+% of software you use will be FOSS in Linux, this means that there will be no annoyances from programs. By “annoyances” I mean things like unrequested popups, tooltips, and screen-real-estate-plundering dialog boxes. This is because FOSS is community-driven — if a majority of users find some feature in a FOSS program annoying, it will get removed soon enough. This feature of FOSS does not get enough visibility, but it is one of the most satisfying: the absence of half-assed work by imminent-deadline-driven, exhausted developers. (2) Because Linux is so configurable, you can configure-away lesser annoyances by editing HUMAN-readable TEXT files 99% of the time. The other 1% of the time, you have to use TEXT-driven interactive sessions, which is just as easy to do.
  • Fast, easy problem resolution: Let’s say you did a system upgrade, and something broke as a result. Because there are hundreds, if not thousands, of users who all run the same system (see “Simplified, unified, and standardized software installation/upgrade/removal” above), your problem will be voiced in your distro’s forum soon enough. Windows users have an instinctive distrust of other Windows users: you have no idea what programs they’ve installed, and which anti-virus programs they have running (if at all!). This is why fixing a Windows problem takes quite a bit of skill and specialized expertise (and why, if you’re a geek, others tend to ask you to fix their Windows problems). Not so with Linux. If you encounter an error, all you have to do is copy/paste the error message (human-readable, SAVE-able error messages are part of FOSS culture) into a search engine and add in the name of your distro and the word “forum”. You are now one or two clicks away from reading the latest word (in the world) on the problem.
  • Gentle, but eventual, path to a true understanding of your operating system’s ecosystem: Here, I define “OS ecosystem” as understanding how and why your operating system behaves the way it does. Windows is a horrible platform to learn how your OS behaves. This is because it actively hides many important concepts from you. For example, do you have any idea how and why a program shows up on the “Add/Remove Programs” list on XP, even though you just removed it? (Answer: NO.) Such mystical voids of system-level confusion are rather rare in the Linux world.

Windows pros:

  • Industry-leading games: Almost all commercial games are released for Windows.

Seriously, after all of the extremely positive things that I’ve said about Linux, and how Windows fails in each of those areas, what can I say?

Misunderstandings about Linux (unfortunate from the Linux/FOSS community’s perspective):

  • It’s only for power users: No, it’s for people who don’t mind learning why their computer behaves a certain way. If you enjoy driving your car blindfolded, you are the right type of person to use Windows (actually, Mac OSX might be a better candidate).
  • It’s 100% secure: Yes, after all the overwhelmingly positive things I’ve said about the rarity of viruses/spyware/malware on Linux, nothing’s perfect. However, I will at least argue that Linux is more secure than Windows, because Linux does not hide things as much. I mean, there are dozens of sites out there designed to answer the question, “Is svchost.exe a safe process? How about wuauclt.exe?”, all because of how Windows likes to hide things from you. In Linux, pretty much everything is exposed. In Arch, specifically, you can do a “pacman -Qo XYZ” to see which package is responsible for that file. If no package is responsible for it, then that means either (1) a program created that file (e.g., an optional configuration file) or (2) you created it yourself!
  • If you use Linux, you’re an evil hacker: Congratulations, you’ve been brainwashed.

All non-brain-damaged people get sick and tired of Windows’ limitations and shortcomings every day. There are no secrets in the Windows world when it comes to user dissatisfaction. I used to be one of these people. Then one day, I started using Linux. And the difference is night and day. It’s just unfortunate that legions of PC users have been brainwashed to accept non-configurability and the “Windows way” to do a simple task: make the computer work for them for their needs. I used to be brainwashed that way. Here are some things that opened my eyes to just how horrible Windows is out of the box (Windows’ default “feature” vs. Linux’s superior equivalent):

  • “shortcuts” vs. symlinks (downright superior in every way)
  • “drives” (C:\, D:\, … Z:\) vs. mount points (unlimited and customizable)
  • “cmd.exe” vs. zsh/bash/etc (it’s like comparing the speed of a clown’s unicycle with a Formula-1 race car)
  • “Notepad” vs. vim/emacs (I’m at a loss for words)
  • “MS Word” vs. latex/xetex (imagine if MS Word did everything transparently: enter latex)
  • “I need Adobe program XYZ to export to PDF documents” vs. “Not in Linux.” (need I say more?)
  • “CPU-Z” vs. “cat /proc/cpuinfo” (no wonder there’s no CPU-Z for Linux; you don’t need it!)
  • none vs. “sleep 10m; echo ‘alarm message’; mplayer alarm.mp3” (super-simple, customizable alarm)
  • none vs. “sleep 1h5m3s; sudo shutdown -t 1 -hP now & disown; exit” (which means, “turn off the computer 1 hour, 5 minutes, 3 seconds from now”; the long command that starts with “sudo …” is actually aliased to just “of” for me, so I actually only type “sleep 1h5m3s; of”)
  • none vs. “sleep 1h5m3s; sudo shutdown -t 1 -r now & disown; exit” (which means “restart the computer 1 hour, 5 minutes, 3 seconds from now”; I use an alias here as well)

And here are some precious programs/tools that I had zero knowledge of until I entered the Linux/FOSS community:

  • vim (see above)
  • the latex toolchain (see above)
  • ssh (it’s zsh/bash/etc, but on a remote machine)
  • scp (transfer files over to a remote machine, but encrypt it during the transmission for state-of-the-art security)
  • rsync (copy files over to a remote machine; if a similar file already exists there, rsync will take less time to complete the copy)
  • git (source code management system; GREAT for backing up configuration files, text files, etc.)
  • mutt (text-based, no-nonsense email client)
  • gnupg (cutting-edge encryption tool to encrypt your most important files)
  • cron daemon (another TEXT based, no-nonsense way to control which programs/commands are regularly run at which intervals (minutes, hours, days… even years!))

Perhaps the most rewarding feature of using Linux is that there is always something to learn from it. In other words: the longer you use it, the more you learn to control and customize your computing experience (versus “the more I use Windows, the better I get fixing it”).

I must reiterate that I am a SELF-IMPOSED Linux convert. I didn’t get pushed into Linux, or had to learn it for some school/work requirement. I did it first as an adventure, and later on, just fell in love with it. No one held hands with me, and the help that I did get came from the internet. Linux reached critical mass years ago, and there are enough forum/blog posts out there to help you if you ever get stuck.

Use Linux!

UPDATE January 3, 2012: I just re-read this post (as I do time to time with my other posts) because Antaku commented on it, and wanted to list some awesome FOSS software that I forgot to mention:

  • diff/colordiff: This is probably the second-most important thing after text editors when it comes to dealing with text files — it tells you the difference between two text files. English teachers: when they turn in their drafts, you can immediately tell what parts they changed in 0.01 seconds. You don’t have to re-read half of it! There are tons of other useful things you can do with diff, any time text files are involved.
  • sha1sum: State-of-the-art file integrity verifier (using the very famous SHA-1 hash algorithm). It generates a 40-byte digital fingerprint, called a “hash”, of any file. You use this hash in a “reverse” fashion to see if your file is bit-perfect (i.e., does it still have the same hash that it had last month? last year?). (BTW, git uses SHA-1 for all of the data it tracks.) You can also use sha1sum as a crude substitute for diff, because two similar files with even just 1 different byte will have different hashes.
  • gcc/clang: Open source compilers for the C programming language — great if you want to learn to program in C and need a compiler. Since C will be around in the lifetime of everyone reading this post, I suggest you learn it. You can interface with Linux in a very, very tight way with C, which is very cool.
  • dd: Great utility whenever you want to move around bytes. Yes, this is a very generic statement, but that’s just how powerful (and useful!) dd can be. If you want to benchmark your new SSD, you can use dd to generate thousands of files filled with random bytes.
  • xxd/hexdump: Hex viewers! Vim uses xxd to turn itself into a hex editor.
  • cat: Short for concatenate. I.e., if you do “cat foo bar quux > ash”, all of the contents of foo, bar, and quux (their bytes) will be joined up into a single file, into ash. It comes in handy lots of times, whenver you’re using a shell.
  • dmesg: Kernel-level log messages (what the Kernel detected, did) for your browsing pleasure. Great for understanding your computer’s behavior at its lowest levels. Does Windows even have an equivalent???
  • iftop: Excellent network traffic viewer. It tells you which IP addresses you are downloading/uploading from. Great for troubleshooting LAN setups in your home.
  • htop: Like Windows’ famous “Task Manager”, but it actually tells you EVERY SINGLE running process, not just what you launched yourself.