Geoffrey Thomas (geofft)

9 Reasons Python Sucks

2018-12-18T00:00:00+00:00

This is a list of reasons why I think Python is a terrible programming language. Naturally, none of them apply to other programming languages.

It's impossible to determine at compile time whether a Python program can terminate.
While strings in Python 3 are significantly better than strings in Python 2, they consist of a series of "code points" instead of characters, so there's no way to reference the third character of a Python string. For ASCII strings, this works fine in C, so I prefer writing in C. Python claims to support Unicode but can't even get this right.
If I compile a Python module on Linux, it doesn't work on a Mac. This is a shocking oversight for a so-called "cross-platform" language.
The os.link function does not let you create a hard link to a directory.
The standard library sorting function takes at least O(n log n) time. I understand dynamic languages are slow, but why is it this slow?
The cryptography module claims to implement public-key cryptography, but mathematicians have so far been unable to prove that one-way functions, a prerequisite of public-key cryptography, even exist. It's surprising that the Python ecosystem is of such low quality and holds to dubious scientific standards.
When you compile NumPy or SciPy from source, you need to build FORTRAN code, and FORTRAN sucks which is clearly Python's fault.
If you're running two Python programs from different users on the same machine, a speculative load in one program might be able to learn information from the other program based on cache timing side channels.
Python is unable to reverse entropy.

Finding a kernel regression in half an hour with git bisect run

2018-03-02T00:00:00+00:00

The git bisect command helps you identify the first commit in a range that broke something. You give it a good commit and a bad one, and it will do a binary search between the two to find the first bad commit. At each step, you say either git bisect good or git bisect bad depending on whether it passes your test, and it will move you halfway through the remaining commits in the range.

There are several guides for using git bisect with the Linux kernel (e.g., upstream, Gentoo, and Ubuntu all have one). Unfortunately, they're pretty time-intensive operations; they all say something to the effect of, "now build the kernel, reboot into it, and test it, then type git bisect good or git bisect bad depending on whether it worked." For a tricky hardware compatibility bug, this might be your only option. But if you're testing something about the kernel's behavior, this is unnecessarily slow and manual, and you might be tempted to do something else, like read commit logs.

At work a few days ago, someone reported that a certain application no longer worked in a new VM. After some initial debugging with strace, we determined that the program was calling the listen system call with a backlog of 0: that is, it was saying it was willing to accept up to zero connections. By the specification, it shouldn't work—and yet it did work on their older VM. A few things were different between the new systems, but one notable one was that the new VM had kernel 4.9 and the old one kernel 4.1. (Another was that it was deployed in a new cloud environment that my team is responsible for, with some networking changes, so we wanted to ensure we had not broken anything!)

I tried reading through git log --grep listen v4.1..v4.9 net/, but there was entirely too much and I couldn't find anything. So I decided to see if bisection could help me, with the use of git bisect run, which enables fully automated bisecting. I wasn't excited about rebooting my machine to do a binary search across eight kernel releases, but if I could get it to run in some other way, I could just leave it running.

For a normal program, it's pretty easy to use git bisect run, which just wants a command that returns success (0) or failure (1): you can usually do something like git bisect run make test. For a kernel regression, though, we'll need a command to boot the kernel and run some code. We can use the qemu virtual machine software for this, which has two properties that make it particularly suitable as such a command: it can boot a Linux kernel directly, instead of emulating a bootloader on a hard disk, and it can run a temporary VM in a single command line without any additional setup.

We'll build ourselves a tiny "initrd" (initial RAM disk), which is what's commonly used to load enough drivers to access your hard drive and completely boot your system. However, our initrd will just contain our one test program, which will possibly print a success message, and shut down the system. We can't meaningfully get a return value out of qemu, so we'll just grep its output for the success message.

The first step is to check out the kernel sources, if we don't have them already, and build a kernel:

$ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux
$ cd linux
$ make defconfig
$ make -j8

which prints this after a while:

Kernel: arch/x86/boot/bzImage is ready  (#21)

Then make sure we can boot our new kernel with qemu, without an initrd:

$ qemu-system-x86_64 -nographic -append console=ttyS0 -kernel arch/x86/boot/bzImage

That is, run it in text mode with the VM's serial console on standard input/output instead of trying to pop up a graphical window, and tell the kernel to use the serial port for console output. If your system supports it, you can add -enable-kvm to make it a faster, although since we want to shut down the VM immediately once we run our test, it doesn't make a huge difference (2 seconds vs. 4 on my machine).

This will panic, because we gave the kernel neither a root filesystem nor a working initrd. (You can kill the VM by typing Ctrl-A and then X.) So let's write an initrd with a single binary, init. It needs to shut down the system, so we get back to our prompt:

$ mkdir initrd
$ cd initrd
$ cat > init.c << EOF
#include <sys/reboot.h>
#include <stdio.h>
#include <unistd.h>

int main(void) {
        printf("Hello world!\n");
        reboot(RB_POWER_OFF);
}
EOF

(Yes, the system call for shutting down the system is named "reboot", because the name "shutdown" was already used for the system call to close a socket. I guess early UNIX computers didn't support initiating a hardware poweroff from software, so the shutdown command would just stop all processes, sync and unmount disks, and print a message asking the operator to cut power.)

Compile this program statically, so it's a single binary, put it in the particular form required for an initrd (a compressed cpio archive, an old but very simple format with a weird command-line tool) and make sure it's named init, and then we can boot it up with qemu:

$ cd initrd
$ cc -static -o init init.c
$ echo init | cpio -H newc -o | gzip > initrd.gz
1621 blocks
$ cd ..
$ qemu-system-x86_64 -nographic -append console=ttyS0 -kernel arch/x86/boot/bzImage -initrd initrd/initrd.gz
...
[    0.502593] ALSA device list:
[    0.502889]   No soundcards found.
[    0.503554] Freeing unused kernel memory: 1088K (ffffffff81f2f000 - ffffffff8203f000)
[    0.504262] Write protecting the kernel read-only data: 14336k
[    0.505004] Freeing unused kernel memory: 1680K (ffff88000185c000 - ffff880001a00000)
[    0.505855] Freeing unused kernel memory: 1340K (ffff880001cb1000 - ffff880001e00000)
Hello world!
[    1.089618] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
[    1.092997] ACPI: Preparing to enter system sleep state S5
[    1.094083] reboot: Power down

Great. We've built our own kernel, passed it a test binary to run, and got it booted in a qemu command that exits. This is turning into something we can pass to git bisect run. Now it's time to write the actual test. Here's what I ultimately ended up with to track down my bug:

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/time.h>
#include <sys/reboot.h>
#include <sys/ioctl.h>
#include <net/if.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

int main(void) {
    /* The problem I was tracing was only reproducible with syncookies
       disabled. While the initrd gets unpacked into a writable temporary
       filesystem, nothing exists yet, so if I need /proc, I need to create
       and mount it myself. */
    if (getpid() == 1) {
        mkdir("/proc");
        mount("proc", "/proc", "proc", 0, NULL);
        char buf[] = "0\n";
        int fd = open("/proc/sys/net/ipv4/tcp_syncookies", O_WRONLY);
        write(fd, buf, 2);
        close(fd);
    }

    int server = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);

    /* Also, while a loopback ethernet device exist, it's not
       enabled, so network tests won't work. This code is equivalent
       to `ifconfig lo up`. */
    struct ifreq ifreq = {
        .ifr_name = "lo",
    };
    ioctl(server, SIOCGIFFLAGS, &ifreq);
    if (!(ifreq.ifr_flags & IFF_UP)) {
        ifreq.ifr_flags |= IFF_UP;
        ioctl(server, SIOCSIFFLAGS, &ifreq);
    }

    struct sockaddr_in addr = {
        .sin_family = AF_INET,
        .sin_port = htons(54321),
        .sin_addr = {htonl(INADDR_LOOPBACK)},
    };
    bind(server, (struct sockaddr *)&addr, sizeof(addr));
    listen(server, 0);

    int client = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
    struct timeval timeout = {3, 0};
    setsockopt(client, SOL_SOCKET, SO_SNDTIMEO, &timeout, sizeof(timeout));
    if (connect(client, (struct sockaddr *)&addr, sizeof(addr)) == 0) {
        printf("Success\n");
    } else {
        perror("connect");
    }
    if (getpid() == 1) {
        reboot(RB_POWER_OFF);
    }
    return 0;
}

Most of it is specific to the thing I was trying to test, but you may also need the code to create and mount /proc or to enable lo. Also, I put a few things conditional on getpid() == 1 so that I could safely test the program on my host system, where it wasn't running as root and where I didn't want it powering anything off. (I ran it a few times under strace to make sure it was doing what I expected it to do, and I didn't want to bother with getting strace inside my initrd.)

So I first made sure this is reproducible on a stock kernel by itself, isolated from any config my workplace might add:

$ qemu-system-x86_64 -nographic -append console=ttyS0 -kernel arch/x86/boot/bzImage -initrd initrd/initrd.gz | grep ^Success
$ git checkout v4.1
$ make defconfig && make -j8
$ qemu-system-x86_64 -nographic -append console=ttyS0 -kernel arch/x86/boot/bzImage -initrd initrd/initrd.gz | grep ^Success
Success

Cool, it's definitely a regression somewhere between those versions. (The set of config options change from kernel version to kernel version, so across this wide of a range, the easiest thing is to just get the current kernel's default config - if you need custom config options, you might want to edit .config after running make defconfig or something.) Now time to let git bisect run do its thing:

$ git bisect start
$ git bisect bad v4.9
$ git bisect good v4.1
$ git bisect run sh -c 'make defconfig && make -j8 && qemu-system-x86_64 -nographic -append console=ttyS0 -kernel arch/x86/boot/bzImage -initrd initrd/initrd.gz | grep ^Success'

It started printing a bunch of build logs and I went to work on something else. About half an hour later (I expected it to take longer!), it prints this out:

ef547f2ac16bd9d77a780a0e7c70857e69e8f23f is the first bad commit
commit ef547f2ac16bd9d77a780a0e7c70857e69e8f23f
Author: Eric Dumazet <edumazet@google.com>
Date:   Fri Oct 2 11:43:37 2015 -0700

    tcp: remove max_qlen_log

    This control variable was set at first listen(fd, backlog)
    call, but not updated if application tried to increase or decrease
    backlog. It made sense at the time listener had a non resizeable
    hash table.

    Also rounding to powers of two was not very friendly.

    Signed-off-by: Eric Dumazet <edumazet@google.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>
$ git describe --contains ef547f2ac16bd9d77a780a0e7c70857e69e8f23f
v4.4-rc1~141^2~238^2~2

which looks awfully relevant—it implies they were previously rounding off the backlog. Looking at commit, we can see what happened: before kernel 4.4, the backlog argument was always capped to at least 8, and also rounded up to the next power of two. So listen(fd, 0) was turning into listen(fd, 8) on older kernels, and the program previously worked despite using listen() incorrectly. This commit was actually somewhere in the git log I was trying to read, but I must have scrolled past it.

git reflog shows that git bisect went through sixteen commits before settling on this one: it found this one on its 11th try, and then spent 5 more commits confirming that all the commits before this one were good. So I'm glad git bisect run found this commit, and I'm especially glad it found it in half an hour unattended, without me having to manually compile and test sixteen kernels by hand.

Hostnames and usernames to reserve

2015-11-26T00:00:00+00:00

If you're setting up a service where people can register their own usernames to be used as a hostname (username.example.com), email address (username@example.com), or URL path (example.com/username) within your domain, there are some common names you should avoid letting the general public register.

Many Internet protocols make the assumption that a domain is manually managed by its owners, and in particular assume that a name like admin must have been registered or approved by the actual owners. Automatic registration breaks this assumption, and has been the source of some attacks. Microsoft Live has fallen victim to this multiple times: in 2008, a researcher signed up for sslcertificates@live.com and used it to get a login.live.com certificate, and as late as this March, the same problem happened to live.fi, the Finnish version of the service, when an IT professional tried registering the email account hostmaster@live.fi as his personal Live account, and then found he could receive a certificate for that domain.

This is a list of all the names I know that should be restricted from registration in automated systems. If you know of others, please let me know and I'll update this page.

tl;dr: Regardless of how you're currently using usernames, restrict them to lowercase letters, digits, and hyphens, starting with a letter and not ending with a hyphen (that is, /^[a-z]([a-z0-9-]*[a-z0-9])?$/ as an extended regex). Ban all the names in this file (last updated 2015-11-21). Get yourself listed as a public suffix: see below for directions and implications.

Hostnames

Most of these problems involve a computer on the domain doing an unqualified lookup: when a computer named a.example.com looks for b, it will usually find b.example.com. If you're running a simple hosting service, or similar, you may not need to block all of these, but these names are extremely unlikely to be used by legitimate users anyway. So you may as well block all of them to allow expanding in the future.

localhost, localdomain, and broadcasthost: these are usually present in /etc/hosts, and applications or scripts might hard-code an assumption about them having their usual value (especially for localhost).
www: Browsers will often prepend this if the domain itself does not resolve as a hostname.
wpad: Web Proxy Auto-Discovery in several browsers; someone who owns this (unqualified) name can act as a proxy for all web traffic.
isatap: IPv6 tunnel autodiscovery, primarily on Windows. Similarly to WPAD, someone who owns this (unqualified) name can act as a proxy for all IPv6-capable traffic. Windows Server has a built-in blacklist of domain names that defaults to WPAD and ISATAP.
autoconfig: Thunderbird's spec for autoconfiguration. Thunderbird will query the website at autoconfig.example.com for settings when attempting to set up example.com email. Good way to harvest passwords.
Along those lines, imap, pop, pop3, smtp, mail, for email clients that make guesses about what your email servers are. (This includes Thunderbird but also many others.)

Note that valid hostnames are restricted in syntax: they must only contain letters, digits, or hyphens, and cannot start or end with a hyphen. DNS is case-insensitive, so make sure there are no case collisions. An older standard prevents hostnames from starting with a digit, which is a straightforward way to prevent all-numeric usernames (which can cause problems with tools that accept either names or UIDs). Dots separate portions of a domain name and cause various problems (wildcard certificates only apply to one level, a.b.example.com can read and write cookies for b.example.com, etc.), so they're usually more trouble than they're worth. DNS records are much more liberal, but names that don't follow these rules will generally not resolve as hostnames: you can look them up with dig/host/etc., but you can't use them in applications. Checking hostname syntax also prevents you from worrying about names like _tcp or _udp, which are used in SRV records.

Become a public suffix

Most parts of the web platform consider two pages with different origins, that is, scheme (http / https), hostname, and port number, to be unrelated websites that cannot interact with each other by default. However, there are a few exceptions, most notably cookies. Web pages at www.example.com and login.example.com are allowed to set cookies with a scope of example.com, despite not sharing the same hostname / origin. The simple rule of allowing parent domains created the problem of supercookies: example.com could set a cookie scoped to .com, which would then be sent to all sites ending in .com. There are two big problems with this: the first is privacy (being tracked across websites), and the second is session-fixation attacks, where an attacker can overwrite your session cookie with their own, and have your actions (including logging in or sending private data) happen within the attacker's session.

The immediate fix was to ban top-level domains, but this still allowed setting cookies for publicly-registrable suffixes like .co.uk that weren't at the top level. So browser vendors created the public suffix list to track which suffixes are open for public registration. The public suffix list now includes not only "ICANN" entries, such as .com and .co.uk, but also "private" entries, such as .herokuapp.com and .github.io, since the same problems exist with allowing users to set cookies for all Heroku or GitHub Pages users.

So, if you are letting users register hostnames in your domain, you should get it listed as a public suffix, which requires just sending a pull request or an email. It takes some time for the update to reach browsers (the list is compiled into browsers, so it's only updated by a browser version update), so you should try to do this as far in advance as possible before launching.

Note that by making example.com a public suffix, nobody, not even code on example.com itself, can set a cookie for example.com. If you have a website of your own that needs cookies (analytics, registration, etc.), you'll need to run it at e.g. www.example.com, and make example.com just a redirect. Alternatively, you can use a completely separate domain for your own site vs. your users' sites, as with the Heroku and GitHub examples: their own websites are heroku.com and github.com.

Email addresses

The CA/Browser Forum Baseline Requirements, section 3.2.2.4 item 4, requires that if a CA is going to validate a domain by coming up with an administrative email address on its own, it may only use admin, administrator, webmaster, hostmaster, or postmaster. Reserve all of those names, regardless of whether they go somewhere useful.

All CAs are supposed to be compliant with that these days, but for safety's sake, also reserve root, info, ssladmin, ssladministrator, sslwebmaster, sysadmin, is, it, and mis (see this 2009 comment on Mozilla's bug tracker).

RFC 2142 defines the names info, marketing, sales, support, abuse, noc, security, postmaster, hostmaster, usenet, news, webmaster, www, uucp, and ftp. You won't need most of these to actually reach a useful mailbox, though you should reserve all of them.

You may want to reserve mailer-daemon, nobody (a default UNIX user account), noreply, no-reply, etc. for automated processes that send email.

Again, as these names are unlikely to be used by legitimate users, it's usually worth blocking them now and keeping your options open, even if you're not currently offering email service. You may add an email service in the future (Amazon launched Send to Kindle by email over a decade after introducing user accounts). As always, you can manually register these names to trusted or internal users.

URLs

For many websites with user-provided content, like Twitter, Facebook, or GitHub, user-chosen usernames become part of the URL at top level (https://twitter.com/geofft, https://github.com/geofft). If you're building a website like this, the easiest approach is to restrict these usernames as if they were hostnames. This has two advantages: the first is that it's easy to launch a hostname-based system later (e.g. GitHub Pages now supports geofft.github.io) if you know that all your usernames are valid hostnames.

The second is that there are several URL paths you need to reserve at top level, and all of them happen to contain dots and are therefore invalid hostnames. If you do permit dots, you need to block the following names:

robots.txt, for the Robots Exclusion Protocol, used to tell well-behaved crawlers how to well-behave.
favicon.ico, for the shortcut icon displayed in the tab bar and other places.
crossdomain.xml, which allows the Flash plugin to make cross-origin requests. Java and Silverlight also look for and trust crossdomain.xml.
clientaccesspolicy.xml, a Silverlight-specific version of crossdomain.xml.
.well-known, specified in RFC 5785 as a place for these sorts of things so they don't keep cluttering the root level. Thunderbird autoconfiguration looks in here, as do ACME, the automatic certificate enrollment spec from Let's Encrypt; BrowserID / Mozilla Persona; and RFC 7711, a new standard for providing certificates for third-party non-HTTP services. So there are a number of security issues with an unauthorized user being able to create files under /.well-known/.

(These are URLs, not filenames. You should of course also disallow users from creating files named e.g. .htaccess if your web server respects those.)

All of these are invalid hostnames, so simply requiring usernames to be valid hostnames avoids having to check for these specific cases. If you're only allowing users to choose some portion of the URL, and inserting other text (e.g., example.com/user/geofft, example.edu/~geofft), then you don't have to worry about this, but again it may still be useful to keep your options open for other URL, hostname, or email schemes in the future.

Do not allow users to publish custom HTML, especially not custom scripts, at these sorts of URLs. https://example.com/user1, https://example.com/user2, and https://example.com/login all share the same origin, so by the same-origin policy, these web pages can freely interact with each other and mess with each other's content. A few JavaScript interfaces, including service workers, make it very easy to attack another site on the same origin. If you want users to be able to publish custom HTML and JS, use separate hostnames within a public suffix. https://user1.example.com and https://user2.example.com are separate origins, and if you have made example.com a public suffix as mentioned earlier, you can safely let them publish custom scripts, since the sites are no more able to interact with each other than two separate .com websites could.

This post was inspired by a GitHub issue for Sandstorm's sandcats.io dynamic DNS service; thanks to Asheesh Laroia for pointing me at that thread and reviewing a draft of this article.

Smashing the heap by overflowing the stack

2015-07-06T00:00:00+00:00

I ran across something strange while learning about Rust's stack overflow and segmentation fault handling.

First, some backstory: in the past, Rust (and Go) used segmented stacks, also known as split stacks. This is a scheme that allows you to start each thread with a small amount of stack space, and dynamically allocate a new, discontiguous chunk of stack space when you run out. Each function starts with a call to __morestack to see if it has enough stack space for that function's variables. If not, __morestack is supposed to allocate more space, quietly switch out the stack pointer, and call the function. When the function returns, __morestack frees the additional space and restores the original stack pointer. For a system like pre-1.0 Rust's tasks or Go's goroutines, allocating lots of tiny, growable stacks makes sense.

However, Rust's __morestack function currently only serves to trigger stack-overflow handling: the only thing that it does, besides aligning the stack properly, is call the rust_stack_exhausted function, which prints an error message and exits. Rust gives each thread plenty of stack space, so if it overflows, it probably means there's unbounded recursion and the thread should be terminated.

I thought this was a little odd. When you overflow the stack in C, or in any other language without __morestack, the next instruction tries to access unallocated memory. This causes a standard segmentation fault, which terminates the program. What's the need for Rust to catch this and terminate the program on its own?

Part of the answer is to provide a better error message. A straightforward stack overflow is not a memory safety violation in the same way an access to out-of-bounds memory is, so a segmentation fault is the wrong way to report a stack overflow. But the real answer turns out to be subtler than that. While accessing an invalid page of memory cannot cause data corruption, it's not guaranteed that the page is in fact invalid! With enough luck, you can overflow the stack far enough and reach into a completely unrelated memory region.

It turns out that this is possible in well-defined C, in a worrisomely straightforward way. The following program generates no warnings:

#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>

void clobber(uintptr_t target_address) {
    int dummy;
    uintptr_t dummy_address = (uintptr_t)&dummy;
    char array[dummy_address - target_address];
    array[50] = 'x';
    (void)array; // suppress warning about unused variable
}

int main(void)
{
    int i;
    int *x = malloc(20 * sizeof(int));
    for (i = 0; i < 20; i++)
        x[i] = 3;
    clobber((uintptr_t)x);
    for (i = 0; i < 20; i++)
        printf("%d ", x[i]);
    printf("\n");
    return 0;
}

and produces this output on my computer (an x86-64 machine running Debian GNU/Linux 8):

geofft@titan:/tmp$ gcc -Wall -Wextra -Wpedantic --std=c99 -o lol-stacks lol-stacks.c
geofft@titan:/tmp$ ./lol-stacks
3 3 3 3 7864323 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

What's going on? We create a variable on the stack called dummy, to figure out where our stack is. Then we figure out how far we are from the target address and create a giant array of exactly that size. Now the end of the stack lines up, more or less, with the target address. The process of "creating an array" doesn't involve any actual changes to memory, just changes to the interpretation of memory (and usually updating the stack pointer register), so while there's a wide swath of invalid memory between &dummy and target, none of it is accessed. Therefore, because the only access is to a valid memory location, there's no segmentation fault generated, and the change to memory goes through. When the program writes to array, it finds valid, writable memory at that location—namely, one of the elements in the target array x. (Remember that on x86 machines, the stack grows downwards, towards lower-valued addresses, so array[0] is the end of the stack. On other architectures, if the stack grows upwards, we'd need to write to the other end of array.)

As you might expect with memory corruption, this is exploitable. In 2010, Rafal Wojtczuk, a researcher at Invisible Things Lab, discovered that this sort of confusion could be used in a privilege-escalation exploit against the X server (PDF). A client can cause the X server to exhaust most of its memory, request a shared-memory region that gets mapped right past the stack, and then trigger a recursive function in the X server. As the associated blog post puts it, it "exploits a bug in... well, actually it doesn't exploit any concrete bug, which makes it so much more interesting."

How can we defend against this? The X server vulnerability (CVE-2010-2240) relied on the stack running right up against mapped pages, so the straightforward solution was to ensure that an unmapped guard page is always just past the end of the stack. But this doesn't help for more complicated cases that span more than a single page—like my example above, or even a less-contrived function with a 5000-byte string buffer.

It turns out that GCC has an option, -fstack-check, that inserts code to check for this problem. If I recompile my program with -fstack-check, I get a segmentation fault. It appears that gcc has inserted code to step the stack down one page at a time, running a logical-OR with zero at each point, which doesn't affect any value on the stack but forces a memory access. Here's the disassembly of that section of code, from GDB:

   0x0000000000400621 <+139>:   mov    %rsp,%rcx
   0x0000000000400624 <+142>:   sub    %rdx,%rcx
   0x0000000000400627 <+145>:   cmp    %rcx,%rsp
   0x000000000040062a <+148>:   je     0x40063a <clobber+164>
   0x000000000040062c <+150>:   sub    $0x1000,%rsp
=> 0x0000000000400633 <+157>:   orq    $0x0,(%rsp)
   0x0000000000400638 <+162>:   jmp    0x400627 <clobber+145>

The size of my variable-length array has already been computed in %rdx. GCC has inserted code to step through the stack one page ($0x1000 bytes) at a time, in a loop. At each step, it ORs the value at the stack pointer with zero, which doesn't change any data but forces a read and write of memory.

Why isn't this the default, like other safety mechanisms like -fstack-protector? I don't know for sure. Part of the answer could be performance, since each somewhat large stack allocation will result in this code being run—with overhead linear in the size of the allocation, instead of just a single change to the stack pointer—but I'm not sure if anyone expects large stack allocations to be performant. Another answer could be lack of interest, since GCC doesn't even enable it by default when compiling Ada code, even though the Ada language requires that stack overflows and segmentation faults be distinct errors.

This technique is common on Windows. Microsoft's rationale for stack checking is less about preventing unexpected changes to memory and more about making sure the stack is actually in place. Windows places a single guard page past the end of the stack, which is reserved in the memory map, but causes a fault when accessed. The kernel takes this as an indication that the process needs more stack space, and will allocate more physical pages, provided that resource limits allow it and there's enough free RAM. Then it moves the guard page past the new end of the stack. So, if your program has an object that is even slightly more than the page size (e.g., a char buffer[5000]), it's important that the guard page get touched, instead of skipped over. (Linux also uses a similar mechanism, but as far as I can tell, the entire possible stack region is treated as guard pages, so the problem of skipping over a single guard page doesn't arise.)

Rust is looking at moving away from the __morestack approach to this approach, under the name of stack probes (the term "probe" is also used in GCC documentation). Apparently the right name for this technique is a bit of a question. "Stack checking," as Microsoft and GCC call it, is a somewhat generic name, and I imagine many developers would interpret it as referring to the more typical direction of stack protection. Indeed, I was very surprised to see that the first C program above "worked", clobbering memory, even with -fstack-protector. It's common to see stack-smashing attacks, where an out-of-bounds pointer overwrites something valuable on the stack, and that's what -fstack-protector (StackGuard, which uses stack canaries) protects against. But it's not as common to see the inverse problem, where an out-of-bounds stack pointer overwrite something elsewhere in memory, which can be just as unsafe and just as much of a security risk.

Thanks to Nelson Elhage, Liz Denys, Alex Dehnert, and Asheesh Laroia for feedback on drafts of this post.

signalfd is useless

2015-05-17T00:00:00+00:00

UNIX (well, POSIX) signals are probably one of the worst parts of the UNIX API, and that’s a relatively high bar. A signal can be sent to your program at any point when it’s running, and you have to deal with it somehow. Traditionally, there are three ways of responding to signals (“dispositions”): you can let the default behavior run, which is often to kill the program; you can ignore it; or you can set up a function to be a signal handler.

If you register a signal handler, it’s called in the middle of whatever code you happen to be running. This sets up some very onerous restrictions on what a signal handler can do: it can’t assume that any locks are unlocked, any complex data structures are in a reliable state, etc. The restrictions are stronger than the restrictions on thread-safe code, since the signal handler interrupts and stops the original code from running. So, for instance, it can’t even wait on a lock, because the code that’s holding the lock is paused until the signal handler completes. This means that a lot of convenient functions, including the stdio functions, malloc, etc., are unusable from a signal handler, because they take locks internally. POSIX defines a set of “Async-Signal-Safe” functions that are guaranteed not to rely on locks or non-atomic data structures (because a signal handler could be called between two instructions that update a non-atomic data type). Linux’s signal(7) manual page documents has a list of these functions. But because of the constrained environment, one common approach is to defer the actual work until you’re no longer running in a signal handler. A standard solution is the “self-pipe trick”: open an unnamed pipe, write a byte to it in your signal handler and return, and read from it in the main loop. This makes signal handling resemble the handling of anything else that’s a file descriptor, like network sockets, which normalizes their handling a bit.

There’s another problem with signal handlers: in addition to interrupting user code, they also need to interrupt kernel code so that it gets delivered. So, if you’re in the middle of reading from a network socket, waiting for a timeout, or even closing a file descriptor, that system call gets aborted so that the signal handler can run. By default, once you exit the signal handler, your system call will appear to return early, with an error exit of errno == EINTR (“Interrupted system call”). The intention is that you can restart the call yourself if you want, but in practice, very few libraries do, which causes confusing errors for application developers. (Extra credit: explain why the linked solution is wrong.) To avoid these sorts of problems, BSD added a mechanism to mark signal handlers as restartable, which was standardized as the SA_RESTART flag to the sigaction system call. But that does not reliably cause system calls to continue instead of returning EINTR: as documented later in signal(7), Linux will return EINTR in some cases even when a system call is interrupted by an SA_RESTART handler. (The rationale for many of these makes sense—e.g., timeouts where restarting the call would start a timer from zero—but understanding the problem doesn’t make it any less of a problem.)

So, in 2007, Linux gained an API called signalfd, which lets you create a file descriptor that notifies on signals. The idea is that you can avoid the complexity of the self-pipe trick, as well as any problems with EINTR, by just asking the kernel to send you signals via a file descriptor in the first place. You don’t need to register a signal handler, so everything works perfectly… except that signalfd doesn’t actually change how signal dispositions work. If you only create a signalfd, signals still get delivered via the the signal-handling mechanism, so in order to actually get the signal to get delivered over the file descriptor, you need to suspend normal processing of the system. There’s another part of the signal API that lets you set a mask to block the signal: the intention is that you can remove the mask later, and then pending signals will get delivered. But this also means that pending signals are readable from the signalfd, and they’re removed from the queue once read.

This would work fine if signal masks applied only to your current process, but they also get inherited to children. So if you’re masking a signal like SIGINT in order to receive it via signalfd, and you run a subprocess, then that child process will start up with SIGINT masked. And while programs could reset their own signal masks when they start up, in practice, no software does. So you have to be very careful to reset any masked signals before starting a child process, and unfortunately you need to do this yourself: most ways of starting child processes, including the standard libc system(3) function, do not reset handlers or masks.

There’s another problem with masking signals, which is that standard UNIX signals are permitted to coalesce when they queue. Namely, if you have a mask that’s blocking a signal, and that signal gets delivered twice before you proces the first one from the queue, it only gets reported to you once. For something like SIGINT, this might be okay: probably you don’t need to handle multiple Ctrl-C keystrokes differently from a single one. For something like SIGCHLD, which notifies you that a child process has terminated, this is quite a bit more unfortunate. It now means that you know that at least one child process has terminated. And while signals can have associated info (via the SA_SIGINFO flag to sigaction, or by default for signalfd), and SIGCHLD’s siginfo tells you which process has terminated, if it gets coalesced, you only get one set of info. That is, you know that the specified child process has terminated, but also one or more other unspecified child processes could have terminated and the information about which one has been lost. Not only does this apply to unmasking signals and letting them get delivered to a normal signal handler, it also applies to reading signals from a signalfd, since signalfd works by dequeuing signals.

It turns out that there is, at least in theory, a better way to handle this. Just go back to the old self-pipe trick, but install the handler with the SA_NODEFER flag, which allows signal handlers to interrupt themselves. ~~This way you reliably get one handler called per signal.~~ [EDIT: This is wrong, see below.] This brings back the EINTR problem, but there’s a simple solution to that: simply dedicate a separate thread to signal handling, and make sure all signals get routed there. Then no system calls on any other thread will get interrupted. The only way to ensure this is to mask signals on every other thread, but that’s no worse than the status quo with signalfd, since we already had to mask signals in order to get them delivered to signalfd alone. As a bonus, this is portable to other operating systems that don’t have signalfd, so this is the solution I’m planning on implementing for the MIO cross-platform event-handling library for Rust.

Can signalfd be fixed? I think it can, if we explicitly tie signalfd into the signal-disposition mechanism. If signalfd could claim responsibility for signal delivery, instead of requiring that signals be masked or ignored in addition to using signalfd, this would solve both problems. For inheritance, just set the close-on-exec flag on the signalfd, and signals will go back to the default behavior in child processes, once the fd no longer exists. For multiple delivery, because signalfd no longer interacts with signal queueing, it can just be specfied to send one message per signal. Alternatively, a mechanism to mask or ignore a signal that applies to the current process only would also be an improvement. These proposals would be a change to POSIX signal semantics, but fundamentally so is signalfd, and the only way to get clean handling of signals is to avoid the POSIX API, not work within it.

EDIT: Greg Hudson pointed out to me that even traditional handlers can coalesce signals. If a signal is sent twice before the signal-handling thread is scheduled, it still gets coalesced, even if it's destined for a signal handler with SA_NODEFER. So while a signal-handling thread can get you separate siginfo most of the time, it still can’t do so reliably. I’ve written an example of this behavior that uses vfork to prevent the process from being scheduled: it starts multiple children, but only prints one notification. So my claim that there’s a better alternative to signalfd is wrong, but it’s still true that signalfd doesn’t solve any of the other troubles with signal handling: it just saves you a separate thread.

Thanks to Nelson Elhage for pointing me at the LWN article, and Alex Chernyakhovsky, David Benjamin, and Yehuda Katz for discussions about all the unfortunate parts of signal handling.

Testing NSS modules in glibc

2015-04-18T00:00:00+00:00

The Name Service Switch (NSS) is the feature of many C libraries, including the standard Linux one (GNU libc) and the standard Solaris one, that allows name lookup routines to be implemented via plugins. "Name lookup" here refers to things like usernames, host names, etc. When you run ls -l, and the ls command maps numeric user IDs to usernames, the names are provided by any module listed in the Name Service Switch's configuration. This could be the files module that looks at local files like /etc/passwd, the ldap module that talks to a corporate LDAP server, etc.

One of my half-forgotten side projects involves writing a new NSS module. NSS is configured via the file /etc/nsswitch.conf, which is systemwide configuration. If I want to test my NSS module, I could install it globally and reconfigure my system. But I started wondering if there was a way to load an NSS module just for a single process under my control. Since the process is running in my user account, I should be able to reconfigure it, without affecting the rest of the system.

Turns out that it's possible in a somewhat hackish way. There's an internal glibc function called __nss_configure_lookup that overrides NSS settings. If you're writing your own test program, you can just call, e.g., __nss_configure_lookup("passwd", "files") to force all user lookups to go through libnss_files. If you're using an existing program, you can shoehorn this in by use of an LD_PRELOAD:

#include <nss.h>
#include <stdlib.h>

static void __attribute__((constructor))
nsstest_ctor(void)
{
        const char *db = getenv("NSSTEST_DB"), *config = getenv("NSSTEST_CONFIG");
        if (db && config)
                __nss_configure_lookup(db, config);
}

Compile with gcc -fPIC -shared -o nsstest.so nsstest.c. Then you can do things like this:

howe-and-ser-moving:/tmp geofft$ ls -ld ~
drwxr-xr-x 355 geofft root 34816 Apr 17 21:25 /afs/athena.mit.edu/user/g/e/geofft
howe-and-ser-moving:/tmp geofft$ LD_PRELOAD=./nsstest.so NSSTEST_DB=passwd NSSTEST_CONFIG=files ls -ld ~
drwxr-xr-x 355 40490 root 34816 Apr 17 21:25 /afs/athena.mit.edu/user/g/e/geofft

Since my account isn't in the files database on this machine (it's in hesiod), my user ID can no longer be looked up if I restrict passwd lookups to the files database. A more straightforward way of testing is using the getent command that ships with glibc, which lets you ask for a specific entry in a specific NSS database. For instance, both files and hesiod have entries for the root user:

howe-and-ser-moving:/tmp geofft$ LD_PRELOAD=./nsstest.so NSSTEST_DB=passwd NSSTEST_CONFIG=files getent passwd root
root:x:0:0:root:/root:/bin/bash
howe-and-ser-moving:/tmp geofft$ LD_PRELOAD=./nsstest.so NSSTEST_DB=passwd NSSTEST_CONFIG=hesiod getent passwd root
root:*:0:101:Wizard A Root,,,:/mit/root:/bin/csh

If you're writing your own NSS library, you'll also need to set LD_LIBRARY_PATH to point to the directory where it lives, since NSS configuration just takes names, not full paths.

A proposal for /usr/bin/python between Python 2 and 3

2015-04-17T00:00:00+00:00

I just got back from PyCon, where a few debian-python team members met to discuss some goals for Python packaging in Debian over the next release cycle. One item of interest is moving away from installing Python 2 by default (expecting it to be desupported in 2020), which raises questions about what /usr/bin/python should mean. At the moment, it very strongly means Python 2 specifically, so there's a question about what it should be on a system with only Python 3 installed—should it become Python 3? Should it be nonexistent?

PEP 0394 recommends that Python 2 should be installed as python2 and Python 3 as python3, and that python should mean Python 2 for now. I think it's important that python continue to mean Python 2, for the reasons described in the "Migration Notes" section of that PEP. In particular, I think it's very important that you should be able to install Python 2 (for the near future), even if your system did not ship with Python 2, and that the system version of Python 3 should not prevent you from using python to run Python 2 applications.

However, not shipping /usr/bin/python by default also has its downsides. I made a suggestion on the debian-python list for handling this: this blog post is a more detailed version of that proposal.

The basic motivation for this proposal is that third-party script authors love Python in part because it's just about universally available. #!/usr/bin/env python, today, is sort of like #!/bin/sh: you don't necessarily get nice things (Python 3-only features and bash-only features, respectively), but it works. If the python command stops existing, this is inconvenient for end users, who will need to manually edit scripts or install a version of Python themselves, and authors, who may just decide to use a worse language like #!/bin/sh. It's also bad for the Python language community, since it removes an incentive to use Python.

So it would be nice to keep the python command working usefully on both Python 2-only systems and Python 3-only systems. Fortunately, it's pretty doable to port code to work on both Python 2 and 3 with the same source. Especially for third-party scripts with the goal of working out-of-the-box in as many places as possible, writing code to these restrictions isn't particularly onerous, since they needed to stay Python 2-compatible anyway. I've been writing a bunch of Python code recently as polyglot Python 2/3, and the biggest problem has been limiting myself to features in Python 2.7, not getting things to work in both versions of the language.

So here's the proposal: we install a wrapper binary as python, that defaults to launching Python 2 (or reporting an error if Python 2 is not installed). However, Python 2/3-compatible scripts can include a specific marker indicating they can be run on Python 3, which makes these scripts able to run on both Python 2-only systems and Python 3-only systems.

This marker is based on the existing coding: syntax from PEP 0263: it's a "magic comment" on the first or second line of the form pyversions=2.7+,3.3+, indicating which Python major and minor versions are supported. A script compatible with both Python 2 and 3 should include one of these magic comments, and also include a shebang line launching just python. Here's an example based on one from PEP 0263:

#!/usr/bin/env python
# -*- coding=utf-8 -*- pyversions=2.6+,3.3+

from __future__ import print_function

print(u"Hello world!")

On new systems, the python command itself is a wrapper binary that knows what versions of the Python interpreter are installed. If it detects a pyversions comment, it will exec the newest Python interpreter compatible with the comment. For this example, if Python 3.3 is installed, it will launch that: if only Python 3.2 and 2.7 are installed, it will use Python 2.7, since Python 3.2 does not support the u"" literal syntax. Otherwise, it will assume that code is Python 2-only, and exec the newest Python 2 version installed. In either case, if a compatible interpreter cannot be found, it will print an error and exit.

On legacy systems, the python command refers to Python 2. So Python 2/3-compatible scripts will still be able to find an interpreter they are compatible with.

For legacy scripts that use a shebang stating just python, on both new and legacy systems, they will only ever run on Python 2, or not at all. This preserves the existing API, and avoids the poor user experience of running Python 2 scripts with the Python 3 interpreter, as PEP 0394 warns against doing.

However, the python wrapper supports running Python 2/3-compatible scripts with the Python 3 interpreter, which is useful for Python 3-only systems. A future version of Debian can choose to ship the Python 3 interpreter only, and remain compatible with third-party scripts that include the pyversions magic comment. Meanwhile, these third-party scripts can state #!/usr/bin/env python in their shebang, and remain compatible with legacy Python 2-only systems, including those (like Mac OS X) that do not include the python2 symbolic link.

The python wrapper can run in three possible modes:

As an interpreter, it follows the rules previously discussed. The "interpreter" mode covers both invocation via the shebang of a Python script, and invocation of the form python script.py.
For interactive use, it will just run the newest version of Python on the system. This allows, for instance, teachers of the language to have their students just run python, without the pedagogical speed bump of having to explain versions of the language. Running the latest major version of Python is safe, since the interpreter will print out its own version at startup.
Uses such as python -c are considered scripted use, since in this context, the python command is also serving as an API, and existing users of the API expect Python 2 just as much as scripts do. (Imagine, for instance, a shell script with KEY=$(python -c "import local_settings; print SECRET_KEY")), which will fail if python means Python 3.) In this mode, the wrapper will instead look for an environment variable named PYVERSIONS. If this is set, it will be parsed as the value of the pyversions magic comment. So shell scripts that include Python 2/3-compatible python -c commands can e.g. export PYVERSIONS=2.7+,3.3+ at the top of the script, and work on systems with either just Python 2 or just Python 3.

The end result here is that third-party script authors can continue writing polyglot Python 2/3 code for the indefinite future, and be compatible with both existing Python 2-only distributions and newer Python 3-only distributions. Meanwhile, distributions can move towards installing Python 3 only by default and avoid installing Python 2 as far as possible, without closing off the ability to install Python 2 when needed, or breaking legacy Python 2-only code.

This is a good transition story for distributions like Debian, Ubuntu, and Fedora that are currently trying to move to Python 3 only, but don't want to break existing code. It's also a good transition story for distributions like Arch that have already started moving /usr/bin/python to Python 3: interactive and scripted use of the python command can continue to launch Python 3, but compatibility with third-party Python 2 scripts is regained. And most importantly, it's good for third-party script authors, who want to write one script that works on Debian, Ubuntu, Fedora, Arch, and Mac OS X alike, and for their end users.

I'm interested in initial feedback on this idea: if it generally seems like a good plan, I'll firm up the details and write it up as a formal PEP. There are a few details to work out, like how this interacts with the py.exe Windows launcher described in PEP 0397 (if at all), but my focus is making /usr/bin/python useful and backwards-compatible on UNIX-like platforms.

Edit 3 May 2015: I've uploaded a proof-of-concept implementation of this launcher to GitHub to see what the overhead and complexity of this approach looks like.

SSH ControlMaster and ControlPath

2014-04-07T00:00:00+00:00

One of my favorite SSH tricks is using the ControlMaster and ControlPath options for speeding up connections. The idea is simple: if you're already logged into a server and you need to log in again, just reuse the connection you already have instead of redoing the entire connection handshake. In addition to making connections much faster, if you're logging in with a password, you don't have to re-type your password for each connection. This can be really handy if you're doing a lot of copies with scp, checking out files with Subversion's svn+ssh://, etc., and don't want (SSH keys can also solve this problem in many cases, but requires you to set up keys on the server, whereas this just requires client-side configuration.)

Setting this up is easy. We'll designate one ssh process as a master, and then when another ssh process wants to connect to the same host, it will connect to the master process instead and request a shared connection. In order to make this work, we first have to set up a place for this master process to listen for requests.

To do this, add the line

ControlPath ~/.ssh/control-%h-%p-%r

to your ~/.ssh/config file. You can read more about the syntax in the ssh_config man page; this particular setting puts the shared sockets in your ~/.ssh directory, which is usually a good place for it, and makes sure that the path is unique for each hostname, port, and remote username, as recommended by the man page.

Then create the master ssh process by running it with the -M option. You'll get a normal login, which you can use or just keep around in a minimized window. You have to keep this connection around until you're done with all your other connections. If for some reason you want to avoid opening a shell — for instance, you're using this with GitHub, which doesn't let you open a shell — you'll also want the -N option.

Once you've done this, every new ssh process that's connecting to the same host (with the same username and port) will reuse the master connection instead of making its own. You don't need to do anything special; as long as ControlPath is set, each new ssh process will check for a master connection at that path.

As mentioned, this can make SSH significantly faster for repeated operations. For instance, git fetch against GitHub becomes over twice as fast for me, compared to re-doing SSH key authentication:

geofft@cactuar:~/src/vireo$ time git fetch

real    0m0.305s
user    0m0.005s
sys     0m0.011s
geofft@cactuar:~/src/vireo$ ssh -M -N -f git@github.com
geofft@cactuar:~/src/vireo$ time git fetch

real    0m0.140s
user    0m0.000s
sys     0m0.010s

This example also used the -f option, which puts the connection into the background as soon as authentication is complete. To close the connection, you can use the -O exit option to ssh, which, instead of opening a new connection, signals the ControlMaster process to exit.

geofft@cactuar:~/src/vireo$ ssh -O exit git@github.com
Exit request sent.
geofft@cactuar:~/src/vireo$ time git fetch

real    0m0.303s
user    0m0.010s
sys     0m0.007s

There's a good article on Wikibooks about connection multiplexing that covers more advanced use of this feature.

Creating tiny Debian packages and repositories for testing

2011-08-15T00:00:00+00:00

For those of you who do a lot of Debian upgrade path testing…

I got annoyed at equivs not supporting Breaks: lines at all ([Debian bug

571638](http://bugs.debian.org/571638)), so I wrote a small shell

script to spit out the minimal debhelper 7 rules file-using package. You can then edit debian/control the way you’d edit an equivs control file and run debuild, or you can continue writing a package for actual software, since it’s a real package and not just an equivs control file.

I also got annoyed at apt not really being able to install packages files on local disk, so I wrote an even tinier shell script to add all packages (recursively) in the current directory to an apt repo and add it via the file protocol to sources.list. It turns out dpkg-scanpackages does this, it’s just easier to have the two-line shell script to remember for you what the command is and what the sources.list line should be.

newpackage and fakerepo are in my Athena home directory. See the comments at the top for usage instructions. I might find a better place for these at some point, but I don’t anticipate ever having to update them…

Using Google ClientLogin

2011-05-13T00:00:00+00:00

The answer to this was much less obvious than I expected it to be, so I figured I’d blog it. If you’re using Google’s ClientLogin API for turning a username and a password into an auth token, what do you do with the token? The result of POSTing to /accounts/ClientLogin gets you three values, and Google says to “reference” the Auth=foo line and discard the rest.

The answer is to send back a header of the form Authorization: GoogleLogin auth=foo. (That is to say, your authentication token doesn’t go anywhere in the POST data, which is what I tried for a while…)

Incidentally, I’m doing this because I’m playing with Google Cloud Print. I successfully created a cloud printer printed a PDF from my phone to, uh, a shell wget command. More on that later.

Be My Escape

2010-09-13T00:00:00+00:00

My a cappella group is singing Relient K’s “Be My Escape” this semester. In discussion at rehearsal after we received the music, it was pointed out that the words are about as fundamental to how cool the song is as the music (and the music is pretty amazing), but it’s fairly intricate and rapid and hard to understand all the details. Here’s my interpretation.

I’ve given up on giving up slowly
I’m blending in so you won’t even know me
Apart from this whole world that shares my fate

The song has essentially one large conceit, of which fate is going to be an important part. One thing to note here is that “giving up slowly” is keeping trying at least a little bit, or if nothing else pretending to not have given up; if he’s “given up on giving up slowly”, he’s given up hard. Another is that the last two lines are a full sentence, so he’s not “apart” from the world, but indistinguishable from it, and resigned to share his fate with them. But that’s mostly straightforward.

This one last bullet you mention
Is my one last shot at redemption
Because I know to live you must give your life away

Suddenly there are a lot more things going on, of which the first is the wordplay of “shot”. It’s both his chance at redemption and firing the bullet; for redemption, or escape, he can kill himself and give his life away. But he then says he does that to live, which seems to be a contradiction. The statement, incidentally, sounds much like an oft-repeated quote from the Gospels, Matthew 10:39, Luke 9:24 and 17:33, etc. etc.

Relient K is a Christian band, and they are undoubtedly going for that reference. But more than that, there are parts near the end when “you” cannot be anything other than Christ, and I think there’s no good reason to say that it’s anyone else here. So what is the bullet Christ mentions? Of course it’s not literal, and this resolves the contradiction: the bullet is the chance to end the old life, the one that makes the singer just part of the rest of the world that shares his fate, and to find spiritual rebirth. Stepping back in one more place this is quoted, John 12, we find this verse before it: “I tell you the truth, unless a kernel of wheat falls to the ground and dies, it remains only a single seed. But if it dies, it produces many seeds. The man who loves his life will lose it, while the man who hates his life in this world will keep it for eternal life.”

And I’ve been housing all this doubt
And insecurity

More wordplay here: “housing” is used both in the sense of keeping it within himself, and building a kind of mental house to live in, out of those fears. The song continues on that imagery,

And I’ve been locked inside that house
All the while you hold the key
And I’ve been dying to get out

Keep the picture of the singer struggling with a locked door from the inside, and Christ outside holding a key (somewhat reminiscent of Revelation’s “I stand at the door and knock.”), for later in the song. At the moment, the focus is on the word “dying”, which is again used both in the idiomatic sense of desiring, and in a literal one,

And that might be the death of me

Once more, the idiom “the death of me” is particularly intentional; we’ve seen by here that the song is about spiritual death and rebirth. From here on, the chorus a lot more straightforward:

And even though there’s no way of knowing
Where to go, I promise I’m going becauses
I’ve got to get out of here
I’m stuck inside this rut that I fell into by mistake

As long as we’re here, it’s worth commenting that, though he may have fallen into the rut “by mistake”, it’s still entirely his responsibility to find a way out. The chorus after the second verse fills out this concept by changing that line to “’Cause I’m afraid that this complacency is something I can’t shake.”

There’s probably a full treatise on how this plays into the Christian conception of sin and especially original sin, but I won’t get into that detail here, except to note that this is a pretty fundamental part of the song.

I’ve got to get out of here
And I’m begging you
I’m begging you
I’m begging you to be my escape

I’ve given up, I’m doing this alone now

He’s no longer part of “this whole world that shares my fate”.

’Cause I’ve failed and I’m ready to be shown how
You’ve told me the way and now I’m trying to get there

The Way, the Truth, and the Life.

And this life sentence that I’m serving
I admit that I’m every bit deserving
But the beauty of grace is that it makes life not fair

The last line is, in my opinion, one of the most powerful of the song. Life’s not fair, it’s true, but if it were, he’d be stuck with his “life sentence” instead of having any options to escape it.

On the subject of a life sentence, I think this is more wordplay. At the least there’s more wordplay between that and the word “life” two lines later, but beyond that he’s not awaiting but currently serving that sentence, while “locked inside that house.” My reading is that the pun involves taking “life sentence” very literally — he is sentenced to have nothing beyond earthly life. As a human, human life is what we deserve, but the singer finds it constraining and seeks to escape and live more than that. After another chorus, this view becomes clear:

I am a hostage to my own humanity
Self-detained and forced to live in this mess I’ve made

and we start getting to the heart of the Christian message. God is not an almighty tyrant popping up out of nowhere to morally condemn humans for being human; we are already constrained by the very nature of our mortal life.

And all I’m asking is for you to do what you can with me
But I can’t ask You to give what you already gave

“While we were yet sinners, Christ died for us.” While we were still “hostages to our humanity”, God came down, took on those chains in his birth, and in his death broke them. In the attempt to figure out how to escape humanity, God put his very being to that cause.

One last chorus, drums and guitars quiet until two powerful downbeats on “dying to get out,” and at the end of that:

I fought you for so long, I should have let you in

Remember the imagery of Christ from outside holding the key. Also here, at least in my reading, is a little bit of a pun, in that we expect that phrase to go “I should have let you win.”

(Oh how we regret those things we do)
And all I was trying to do was save my own skin

One last idiom chosen very intentionally. Beyond the idiomatic meaning of salvation and preservation in general, “skin” in particular refers to his human nature, the earth-life that he was clinging to. But the God he was fighting was trying to do nothing else but provide perfect preservation and salvation for him,

Oh, but so were you
So were you

The Quota Corollary to Parkinson’s Law

2010-08-30T00:00:00+00:00

Parkinson’s Law, the famous quote from his 1955 essay, claims, “Work expands so as to fill the time available for its completion.” It seems there is an analogous principle for disk space:

Date: Mon, 10 Sep 2007 23:05:25 -0400 (EDT)
To: accounts@mit.edu
Subject: AFS quota increase

Hi,

Could I get my quota increased a little bit? I have a LOT of medium-size files for various projects… I tried to clear up my homedir about a month ago but it didn’t help more than about 100 MB. And today my usage jumped from 98% to 99% when I uploaded my first pset (for 6.170)…

Thanks,

Date: Thu, 4 Sep 2008 01:32:00 -0400 (EDT)
To: accounts@mit.edu
Subject: Quota bump

Hi,

Classes already having started, it’s going to be hard to spend time cleaning out my AFS volume and e-mail this week, although I’ll do so soon (I should probably move the things I don’t need regular access to to other storage, anyway). Any chance I could please have a little bit of a quota bump on each so I can do the first assignment for my classes without worrying?

Thanks,

Date: Tue, 1 Sep 2009 15:05:00 -0400 (EDT)
To: accounts@mit.edu
Subject: quota bump for my locker

Hi Accounts,

Can I get more Athena quota on the grounds that it would be helpful for continuing to work on Athena, or something? :)

Also, term is starting soon… I did look around for things that seem potentially likely to be removable, but nothing stands out. I can look harder or try moving things to other storage if you’d prefer I start with that.

Thanks a bunch,

Here are the current official disk policies:

maximum volume sizes
        reference snapshots (eg unchanging CD/DVD images) -- 25G
        swmaint/rel-eng -- 12G (but negotiable)
        course/cwis/leased -- 12G

quotas (user homedirs and activity lockers)
        default: 2G
        if they ask for more: 4G
        and justify it: 6G
        and are Thesing and have a signed letter from the Pope: 8G
                (sufficient academic need may substitute for the Papal letter)

My freshman year, the second half was 1G/1.5G/2G/3G (unchanged since 2004), so at least there’s more breathing room. But still, note carefully,

$ fs listquota
Volume Name                    Quota       Used %Used   Partition
user.geofft                  8000000    7586330   95%<<       78%    <<WARNING

I guess it makes sense: in the normal case, there isn’t a lot of incentive for me to care about disk space, and it’s not a worthwhile use of my time until pressure appears. Even so, I find it amusing that no matter how many quota bumps I’ve already had or how many iterations of Moore’s Law have doubled my quota without even asking, it has become a tradition that every fall (except my freshman year), before classes start, I’ve used 95% of my quota. Files expand so as to fill the disk available for their storage.

Looking at the policy, I guess this time I don’t have a choice but to clean it up.

Testing C code from the command line: gccrun

2010-08-23T00:00:00+00:00

I wrote a little utility the other day before I found myself in need of it: it takes its argument, compiles it as the body of a C program, and runs it.

This is way more convenient than you’d think at first glance: it turns out that, for one- or two-line programs where you’re just testing how something works, when 80% of your program is include boilerplate and making a source file and running gcc followed by the program, you’re somewhat disinclined to test small things. When you have a utility like this, you end up using it a lot…

How many file descriptors can I open?

dr-wily:~ geofft$ gccrun 'int fd; while ((fd = open("/dev/null", 0)) > 0) printf("%d\n", fd);' | tail -1
1023

What error does that get me?

dr-wily:~ geofft$ gccrun 'while (open("/dev/null", 0) > 0); perror("open");'
open: Too many open files

Can I go higher by not having things open on lower fds, and just opening something on fd 1024?

dr-wily:~ geofft$ gccrun 'dup2(0, 1023); perror("dup2");'
dup2: Success
dr-wily:~ geofft$ gccrun 'dup2(0, 1024); perror("dup2");'
dup2: Bad file descriptor`

What is errno 2?

dr-wily:~ geofft$ gccrun 'printf("%s\n", strerror(2));'
No such file or directory

What is my “session leader”?

dr-wily:~ geofft$ gccrun 'printf("%d\n", getsid(0));'
18291
dr-wily:~ geofft$ ps 18291
   PID TTY      STAT   TIME COMMAND
 18291 pts/28   Ss     0:00 /bin/bash

This is, of course, no substitute for reading formal docs or writing rigorous tests, but it’s great for getting an intuition about how something works.

gccrun is a very tiny shell script. Most of the convenience comes from it including a bunch of headers by default; recommendations on headers to add, and patches in general, welcome.

Software I want (other people) to try

2010-08-02T00:00:00+00:00

I find myself recommending software I’ve heard of but never tried with the desire to get other people to spend effort on making them work and letting me know if they’re worthwhile. I’m generally clear about that that’s what I’m doing, but still, it feels a little underhanded, so here’s an explicit list of (some) software that I haven’t gotten around to trying, would like to, and would appreciate comments about from people who’ve used them before:

Cream, a layer on top of vim with normal-people-style keyboard shortcuts. Mostly I’m curious if this is a reasonable thing to recommend to newcomers to the UNIX world who are used to Windows/Mac editors.
xpra, a more different way of running X applications across the network by making clever, perhaps too clever, use of compositing, and basically just compositing windows onto a remote screen and windows thereon. It seems to boast reconnection support and be able to guarantee much better security than real X forwarding can.
Alternative git frontends (“porcelains”), specifically EasyGit, Yap, vng, zit (for single files).
Canorus, a music score editor that looks like it might be very awesome. Compelling features include LilyPond integration and indications that it might actually have a reasonable UI.
Colloquy, a Mac OS X IRC client. I don’t use OS X any more, so this is perhaps the closest to honestly wanting to know if it’s a good recommendation.
fakeroot-ng, a reimplementation of fakeroot using ptrace instead of LD_PRELOAD, which but for speed addresses the limitations in fakeroot I mentioned previously.
neercs, an allegedly awesomer version of the venerable GNU screen. It has some nice tricks and is hopefully more extensible. I also hear the really cool kids use tmux.
most, a pager which is apparently even more than either more or less. Great job on the naming all around, guys…

All of these are, as far as I know, free software.

I’m sure I’ll think of more as time goes on (I already realized I forgot about the last couple within two minutes of writing this). Anyone have experience with any of these? Alternatively, what other potentially-nifty software have you heard of but haven’t gotten a chance to use?

Per-computer colored prompts in bash

2010-07-25T00:00:00+00:00

I use my Athena account from many computers, often multiple computers at once via SSH, and it’s useful to keep track of which terminal is showing a prompt from which computer. One reasonably easy way to accomplish that is to color the prompt as a function of which computer I’m on. Here’s what I use in my .bashrc:

export PS1="\[\e[1;$((31 + $(hostname | cksum | cut -c1-3) % 6))m\]\h\[\e[0m\]:\w \u\$ "

From inside out:

hostname gets the name of the current computer.

cksum gives me a checksum of its input, which I’m basically just using as a well-known hash function with numerical output.

cut picks out certain characters from the input, in this case the first three digits/characters, because checksums can be fairly large.

$(...) means to evaluate a command and substitute it’s output. (If you’re familiar with backticks, it’s the same thing, except that $(... $(...)) works the way you want and `... `...`` doesn’t.)

$((...)) evaluates arithmetic. In this case, we take the checksum and get the remainder when divided by six and add it to 31, so we get a number between 31 and 36. This happens to be the range of ANSI color codes excluding white and black.

\e[1;35m (or whatever the number happens to be is the full ANSI escape sequence to turn the prompt a certain color, bolded.

\[ and \] tell bash that the intervening text in a prompt is a formatting code, and shouldn’t be counted towards the length of the prompt, although it doesn’t seem to work for me. It might be related to this snide observation about the code that displays the prompt; I’ve never looked into it in detail.

\h actually displays the hostname in the appropriate color, and \[\e[0m\] resets the colors to default.

\w prints my working directory and \u my username: I sometimes use this prompt or my .bashrc on other accounts, so it’s preferable not to hard-code it.

\$ prints a dollar sign most of the time, except that it prints a pound sign if the shell belongs to root: this is a classic UNIX way of distinguishing the superuser account. Again, I sometimes reuse my prompt, so it’s nice to have this flexibility.

Finally, I stuff the result into the PS1 variable.

So, I get something like this:

kid-icarus:~ geofft$ logout Connection to kid closed. dr-wily:~ geofft$

If you want more background on bash prompts in general, check out the section of the bash manual on prompts.

Stupid tricks at the userspace/kernelspace boundary, part 1

2010-07-19T00:00:00+00:00

Basically every operating system structures execution into two logical parts, userspace and kernelspace. The former is where the application’s own code executes, and the latter is where the OS kernel services requests from applications that require privileged access of various kinds — reading and writing data on disk, getting more memory, interacting with other applications, making network connections, and so forth.

Let’s take a look at how recent versions of Linux (on x86-compatible CPUs) implement the transition between the two.

To start off, kernelspace is privileged and can easily return to userspace. However, as userspace is unprivileged, it can’t just start running code in kernelspace at will. So in order to make a request of the OS, a system call, it generates a software interrupt via the INT instruction, which causes essentially the same sort of response as a hardware interrupt generated by some physical device (a timer, an external input device, etc.): the processor switches to kernelspace and starts running a predefined function in the kernel. This interrupt handler determines what’s going on, acts on the system call, and returns (via IRET) from the interrupt back to userspace.

Usually application programmers don’t have to think about any of this. The standard C library includes a C function for each system call, which additionally takes care of things like passing function parameters the way the system call interrupt handler wants them, and providing unified error reporting. Moreover, this abstraction layer allows the interface to be changed: INT is somewhat time-consuming, since it makes the processor deal with the possibility of a hardware error, and eventually Intel introduced the faster SYSENTER instruction, and AMD a similar SYSCALL one, which was specifically for use for system calls and nothing else.

Well and good, except now code running on older processors needs to continue to use INT, and newer Intel ones should use SYSENTER, and newer AMD ones should use SYSCALL, and so we need a different C library for each processor type. And the problem isn’t just restricted to the C library: nothing is stopping applications from ignoring the C library and making system calls if they want to, and there are legitimate uses for skipping standard libraries entirely (statically-linked rescue utilities being one of the big ones). So we need a different abstraction layer, preferably one right at the syscall level with no C-library–style tricks.

Enter the VDSO, the “virtual dynamic shared object” that looks like it might be a normal library (“dynamic shared object”) on disk, but is actually provided by the kernel. Since the kernel does a lot of hardware probing at boot up, it knows the fastest way to make a system call, so in a stroke of genius the kernel provides to every process a userspace library that implements that fastest system call method. We can see evidence of this by looking at a process’s own “maps” file in the /proc directory, which contains information about each process:

dr-wily:~ geofft$ cat /proc/self/maps
08048000-0804f000 r-xp 00000000 fd:00 65412      /bin/cat
0804f000-08050000 rw-p 00006000 fd:00 65412      /bin/cat
08050000-08071000 rw-p 08050000 00:00 0          [heap]
b74e2000-b761c000 r--p 00000000 fd:00 630461     /usr/lib/locale/locale-archive
b761c000-b761d000 rw-p b761c000 00:00 0
b761d000-b7772000 r-xp 00000000 fd:00 66165      /lib/i686/cmov/libc-2.7.so
b7772000-b7773000 r--p 00155000 fd:00 66165      /lib/i686/cmov/libc-2.7.so
b7773000-b7775000 rw-p 00156000 fd:00 66165      /lib/i686/cmov/libc-2.7.so
b7775000-b7778000 rw-p b7775000 00:00 0
b778f000-b7791000 rw-p b778f000 00:00 0
b7791000-b7792000 r-xp b7791000 00:00 0          [vdso]
b7792000-b77ac000 r-xp 00000000 fd:00 49513      /lib/ld-2.7.so
b77ac000-b77ae000 rw-p 0001a000 fd:00 49513      /lib/ld-2.7.so
bffeb000-c0000000 rw-p bffeb000 00:00 0          [stack]

The line marked “[vdso]” indicates the virtual memory location of the VDSO, and its permissions: readable but not writable, executable, and private (i.e, not shared with other processes). If we’re curious, we can write out the VDSO with a simple C program to read its own /proc/self/maps, search for the “[vdso]” line, parse the addresses, and write out that section of its own memory.

Let’s take a look at it with the objdump command, which includes a simple decompiler:

dr-wily:/tmp/geofft geofft$ ./dump-vdso > vdso
dr-wily:/tmp/geofft geofft$ objdump -d vdso

vdso:     file format elf32-i386

Disassembly of section .text:

ffffe400 <__kernel_sigreturn>:
ffffe400:       58                      pop    %eax
ffffe401:       b8 77 00 00 00          mov    $0x77,%eax
ffffe406:       cd 80                   int    $0x80
ffffe408:       90                      nop    
ffffe409:       8d 76 00                lea    0x0(%esi),%esi

ffffe40c <__kernel_rt_sigreturn>:
ffffe40c:       b8 ad 00 00 00          mov    $0xad,%eax
ffffe411:       cd 80                   int    $0x80
ffffe413:       90                      nop

ffffe414 <__kernel_vsyscall>:
ffffe414:       51                      push   %ecx
ffffe415:       52                      push   %edx
ffffe416:       55                      push   %ebp
ffffe417:       89 e5                   mov    %esp,%ebp
ffffe419:       0f 34                   sysenter 
ffffe41b:       90                      nop    
ffffe41c:       90                      nop    
ffffe41d:       90                      nop    
ffffe41e:       90                      nop    
ffffe41f:       90                      nop    
ffffe420:       90                      nop    
ffffe421:       90                      nop    
ffffe422:       eb f3                   jmp    ffffe417 <__kernel_vsyscall+0x3>
ffffe424:       5d                      pop    %ebp
ffffe425:       5a                      pop    %edx
ffffe426:       59                      pop    %ecx
ffffe427:       c3                      ret

So, on the computer I’m using (a server with four dual-core AMD Opterons), the kernel recommends SYSENTER. Apparently AMD gave up on SYSCALL at some point and decided to implement Intel’s SYSENTER. Meanwhile, on my netbook with an Intel Atom that’s good for nothing except low power consumption, I get the much simpler:

geofft@white-elephant:/tmp$ ./dump-vdso > vdso
geofft@white-elephant:/tmp$ objdump -d vdso

vdso:     file format elf32-i386


Disassembly of section .text:

ffffe400 <__kernel_sigreturn>:
ffffe400:       58                      pop    %eax
ffffe401:       b8 77 00 00 00          mov    $0x77,%eax
ffffe406:       cd 80                   int    $0x80
ffffe408:       90                      nop
ffffe409:       8d b4 26 00 00 00 00    lea    0x0(%esi,%eiz,1),%esi

ffffe410 <__kernel_rt_sigreturn>:
ffffe410:       b8 ad 00 00 00          mov    $0xad,%eax
ffffe415:       cd 80                   int    $0x80
ffffe417:       90                      nop
ffffe418:       90                      nop
ffffe419:       8d b4 26 00 00 00 00    lea    0x0(%esi,%eiz,1),%esi

ffffe420 <__kernel_vsyscall>:
ffffe420:       cd 80                   int    $0x80
ffffe422:       c3                      ret

Now here’s where I got curious. /proc/self/maps, as noted above, listed the page as not writable. Could you make it writable? Let’s try calling the standard function to change memory permissions and seeing if it will let us:

        if (mprotect(vdso_begin, vdso_size, PROT_EXEC | PROT_READ | PROT_WRITE) != 0) {
                perror("mprotect");
        }

… and we get no error, and in fact if we print out /proc/self/maps we see

b774b000-b774c000 rwxp b774b000 00:00 0          [vdso]

Huh. I wonder if it’s actually writable. Going back to objdump, it looks like the SYSENTER operation is at 0×419 bytes into the page (the VDSO is one page, and a page is 0×1000 bytes long). What if we overwrite it to never make the system call?

        *(char *)(vdso_begin + 0x419) = 0x90;
        *(char *)(vdso_begin + 0x41A) = 0x90;
        printf("Hello world!\n");

0×90 being the opcode for a NOP (no operation, other than stepping one byte forward). Compile it and — no error, and… it doesn’t terminate either. Seems like printf is waiting for some system call to report success, and now that we’ve severed communication with the kernel, that’s never going to happen…

It turns out that even without the printf, the program hangs forever: “return 0;” is just another convenience from the C library. The actual “entry point” of a program is in the C library, which calls main() and waits for its return value. Once it has that, it makes the exit() system call — but again, without the ability to make system calls, the program can’t exit, and the C library’s exit() wrapper function just keeps trying to exit, forever.

Stay tuned for more useful stupid tricks involving modifying __kernel_vsyscall — what sorts of things can we do with this abstraction layer? Here’s a hint: take a look at the “BUGS” section of fakeroot’s man page. And yes, I’m aware of ptrace, but there is a reason not to use it…

(See this blog post from Johan Petersson for another treatment of the first 3/5 of this, and this blog post from Anomit Ghosh for a Python version of dump-vdso.)

HTTPS and the clustered trusted third party

2010-06-21T00:00:00+00:00

Rather than going into an in-depth rant about SSL/TLS/HTTPS and everything I hate about them, let me just discuss two problems I’ve run into recently.

As a bit of background, SSL/TLS addresses the issue that Internet connections are completely unauthenticated, and anyone who can intercept your request can forge a reply to it, all the more a concern these days with prevalent wireless networking in public places. The solution, using public-key cryptography, involves the website digitally signing its responses with a certificate that lists the website’s name. This certificate, in turn, is signed by one of several well-known certificate authorities who verify that anyone asking them to sign a certificate legitimately owns the domain name in question. An attacker can come up with his own certificate, but should not be able to get that certificate signed by any authority, and if he tried to invent his own certificate authority it would not be one of the well-known ones.

First, for MIT’s web hosting platform scripts.mit.edu, we give users hostnames of the form geofft.scripts.mit.edu. We have a single wildcard certificate for “*.scripts.mit.edu” that covers our users’ sites. This is all well and good, except that wildcard certificates for HTTPS, according to the spec, only cover one domain component. This is a problem for users who have dots in their username — most notably courses. If you vist, say, 6.001.scripts.mit.edu in your browser, you’ll notice that you’ll get an error about 6.001.scripts.mit.edu being an invalid hostname, as the certificate was “only” issued for *.scripts.mit.edu. I asked the RFC author what the rationale was for this restriction, and the related one that a *.*.scripts.mit.edu certificate or similar is useless for HTTPS, and apart from “wait, someone wants multiple wildcards?”, it boils down to protecting against malicious certificates, like tricking a certificate authority into getting a *.com or (if we prevent first-level wildcards) *.co.uk certificate, which is much more damaging than tricking a CA into issuing a particular site’s wildcard-less certificate to you. It seems that the CA/Browser Forum is moving away from being interested in wildcards in general; for instance, extended-validation (EV) certificates, which verify real-world identity in addition to domain name ownership (and are a lot more expensive), simply cannot be issued for wildcards. So even if we can convince the folks who run MIT’s local certificate authority (who know us in real life) to issue an EV certificate for scripts.mit.edu or some other MIT site, there’s no way they can generate one.

Second, I tried visiting https://bugs.freedesktop.org/ earlier tonight. If your browser is anything like mine (Firefox on Ubuntu), it will give another certificate error because freedesktop got their certificate signed by CAcert.org, a “community driven certificate authority” that unlike every other CA out there (Verisign, Equifax, etc.) is not a large corporation with both an equally large business stake in never being found untrustworthy and a bunch of business folks who can talk to the browser vendors and know how to run a respectable business, and importantly, gives free certificates. (Certificates from large corporations can cost hundreds of dollars a year.) CAcert has had some legitimacy and audit troubles in the past that are too long to go in here, but it’s a bit of an uphill battle for then to get themselves listed in the root certificate list of most operating systems (Debian, notably and unsurprisingly, notwithstanding) or browsers. Long story short, Firefox tells me that the signing authority isn’t trusted, and asks me if I want to accept the presented cert for bugs.freedesktop.org.

Now, this is a bit of a problem. I don’t really trust CAcert the way I trust Verisign and Equifax; if I tried to get to my bank’s website or Gmail or somesuch, and saw a CAcert-signed certificate, I would be highly suspicious. But for a site like freedesktop.org, it is completely in character for them to use CAcert. Unfortunately, since my browser doesn’t trust CAcert, it’s not going to tell me whether validation passed, and this certificate was actually signed by the CAcert authority certificate, or even whether the CA being passed off as CAcert belongs to the real CAcert. If I wanted to do this verification, I have two choices: either I can add CAcert to the list of CAs that my browser trusts by default for everything, or I can learn how SSL certificates work and a tool like OpenSSL, and manually do the verification, which is a completely unreasonable burden even given that I do know how to use the OpenSSL command-line tools, and just not even an option for most users, even including the majority of technical ones.

On the flip side, when I came to MIT and got my computer set up, I set up my browser to trust the MIT certificate authority. The purpose here is that MIT-internal services like online student records won’t need to pay an outside company for a certificate, if the target audience is just MIT folks who can do this configuration. However, if MIT decided one day it wanted to snoop on my Gmail traffic, and one day I got an MIT-signed certificate for mail.google.com, I would assuredly never notice: MIT’s on the list of trusted root certificates and is as good as any other. I certainly don’t check every HTTPS web site I visit to determine which CA signed it: in the normal case, I couldn’t care less whether it’s Verisign or Equifax or GoDaddy or Thawte or whomever. Of course, you might say MIT would never do that, but there’s an actual real-world concern that the government of China might be doing this. They have the same legitimate concern about not paying outside foreign companies for every site that only targets Chinese citizens, but the worry that CNNIC might sign a mail.google.com certificate and execute a man-in-the-middle attack against Gmail and its users is much more real in that context.

Ultimately, the design flaw here is that there is a single class of root certificate authorities, and either a CA is in or it’s out. I want to have the flexibility of saying that MIT’s CA can do whatever it wants, including signing EV certificates, for .mit.edu sites, but nothing else, and that CNNIC can generally do what it wants for .cn but definitely not for anything else. I want to have the flexibility to say that although no, I don’t trust CAcert in general, I would at least like to know if something is honestly signed by the real CAcert as opposed to just being signed by some fake authority calling itself CAcert, and I would like to change my browser’s default for some sites, as I see them, to permit CAcert-signed certificates. I want the error to say not that my browser doesn’t know who signed this certificate and would I like to accept this certificate on faith, but that my browser does know who CAcert is and doesn’t in general trust them but wants to know if it should for this website.

Can we restructure HTTPS to give us these capabilities? More importantly, are the real decision-makers here, the CA/Browser Forum, interested in this restructuring? This mostly only affects small, private CAs who aren’t part of the mostly-common list that all browsers and operating systems trust out of the box: the group of signing authorities that effectively operates as a single trusted third party distributed among multiple companies. So it’s not really in the interest of these commercial CAs to do something so invasive to the structure of HTTPS at this point in the game. Can we convince the browsers that there’s a way to pull this off, and that it’s worth it?

Installing Ubuntu Lucid via debian-installer over wireless

2010-06-13T00:00:00+00:00

I’m a huge fan of installing Debian and Ubuntu systems with debian-installer. Unlike debootstrap, which just installs a set of base packages into a directory, debian-installer does a number of nice things such as offering to do partitioning, setting /etc/fstab correctly, generating locales, installing a bootloader, etc. etc. — all things that get you an installed and working system as opposed to a pile of packages. Also, unlike Ubiquity, the standard (graphical) Ubuntu installer from the live CD, which despite reusing some d-i code works mainly by copying the live CD onto the hard disk and cleaning it up, d-i actually installs each package in turn, which allows you to customize your installs.

I can and probably will talk at length on why I love debian-installer and how I use it in another post, but this last ability was specifically important to me when reinstalling my Eee PC 900A, a machine with under 4 gigabytes of disk space, most of which I wanted free for myself and not for the operating system. What I can do is tell d-i not to install any “tasks”, thereby leaving me with a base (text-mode login!) system that I can later add packages to as I need.

Unfortunately my favorite method of doing this sort of install is via the network installer, and when I started the install process I quickly found that I was getting checksum errors from my local mirror when downloading udebs (special packages designed for the installer environment). Suspecting, at first, the mirror, I downloaded by hand all the udebs from another nearby computer, several times, and was able to verify their checksum. Switching Ethernet cables didn’t help my netbook have any more luck, nor did switching mirrors, so I’m left with the assumption that my Ethernet card is slightly bad. Since I only ever use wireless networking on my netbook, I wouldn’t care about that if I could do the install over wireless, but d-i never gave me an option to use a wireless Ethernet card, nor did one seem to be detected…

However, in the process of sanity-checking the mirror, I noticed the existence of a udeb called “wireless-tools-udeb”. I downloaded, verified, and copied that udeb (and its dependency, “libiw30-udeb”) to my disk via a live CD, and also copied the live CD’s copy of the kernel modules necessary for my particular wireless card, ath.ko and ath5k.ko. Then I restarted the installer, mounted my disk, ran “udpkg -i” on the two udebs and “insmod” on the two kernel modules, and confirmed that I started seeing a wireless device and was able to connect it to MIT’s network with iwconfig.

Much to my surprise, installing wireless-tools had two nice side effects:

After I selected wlan0 as my network device to install via, it asked me what ESSID I wanted to connect to, and what WEP password (if necessary). This completely surprised me; I expected that I’d have to configure wireless networking by hand, and let the installer treat it as essentially a wired-Ethernet device.
When the install completed, /etc/network/interfaces on the installed system was configured to automatically connect to the MIT network.

I was able to successfully complete the install just over wireless. Now I have a laptop that boots in approximately twelve seconds, including connecting to the network.

Free software and reusability

2010-01-11T00:00:00+00:00

The rising popularity of the Free Software Movement and the open source philosophy have perhaps yielded an incomplete understanding of them in popular culture, that “open source” means merely “the code is public”. This is far from complete, and not just in an ideological sense but in a practical one. One of the common trip-ups I’ve seen is forgetting the importance of reusability, that free code should be free whether or not it’s part of the software that it came from.

One of the examples that I ran across recently is that of SugarCRM, which loudly advertises that it’s “open source” in its page title and marketing. This is true today; since 2007 the community edition of the software has been released under the GPL v3. But before then, the license they used contained the following requirement:

… all copies of the Covered Code in Executable and Source Code form distributed must, as a form of attribution of the original author, include on each user interface screen (i) the “Powered by SugarCRM” logo …

plus terms on the size of the logo and the fact that it must be a link. Not too onerous, you’d think, until you decide that you want to take some function for HTML text formatting or web form rendering or somesuch and incorporate it into MediaWiki, and suddently every page on Wikipedia needs to have a “Powered by SugarCRM” logo at the bottom of it. Worse, you could find some useful data structure for organizing e-mail addresses or something and try to incorporate it in your command-line e-mail client, and now you have a user interface screen with no way to display a “Powered by SugarCRM” logo even if you wanted to.

The sad part about this is that these aren’t restrictions that you’d ordinarily think about. Most news articles focused on the ethics of running a company using “open source” in advertising but with a significantly more featureful proprietary version. Some did realize that the license hurt your right to fork, since you’d always have to link to SugarCRM’s website (even if the company collapsed). But what’s really hurt by the inability to learn from and reuse the software is the community of free software development, the “creative commons”, if you will.

(As a side note, the Creative Commons organization is pretty good about respecting this, often under the name of remix culture.)

Debian’s Social Contract demonstrates an awareness of this concern:

We will give back to the free software community

When we write new components of the Debian system, we will license them in a manner consistent with the Debian Free Software Guidelines. We will make the best system we can, so that free works will be widely distributed and used. We will communicate things such as bug fixes, improvements and user requests to the upstream authors of works included in our system.

So, too, do the Free Software Foundation’s four freedoms, and the three tests for whether a particular license is free outlined in an FAQ for Debian’s Free Software Guidelines. Software with conditions on how you can reuse the code for other software isn’t really up to the standards of free software.

Another surprising example of forgetting to take reusability into account is the Free Software Foundation’s own GNU Free Documentation License. Most of the requirements are along the lines of free software licenses, but the GFDL adds the option to specify “front-cover texts”, “back-cover texts”, and “invariant sections” which cannot be changed in redistribution. You can see a typical example of this in the GCC manual, with one-line front-cover and back-cover texts, and an appendix called “Funding Free Software” as the invariant section. While this might help the FSF’s fundraising goals, it does mean that if I want to clip the two-line list of known bugs for a list of errata included with my distribution, or adapt GCC’s description of C99 hex floats into a slide for a class on C, I can’t get by with simply putting my errata list or my slide deck under the GFDL. I have to include the two cover phrases and the two pages on “Funding Free Software” somewhere in those documents, no matter how little of the source document I’m excerpting. Debian in fact considers GFDL-licensed documents with invariant sections not to be free. Surprisingly enough, one of the alternative proposals suggested that a satisfactory solution to the matter of invariant sections was the ability to counter their politics with a new invariant section of your own! While Debian picked the correct answer here, that there is insufficient difference between free software and free documentation that the latter should have different standards, it’s worrisome to see people completely forgetting the importance of free works being free to be reused in different works.

This is why the attitude of Debian project members who dislike the existence of Ubuntu, a distribution that regularly updates to the latest Debian packages for almost all its software, particularly bothers me. People interested in making a free distribution should want to see their work reused in other free systems. An attitude of a complete lack of interest in helping Ubuntu unless the problem can be phrased as helping Debian itself, or anger at people using ubuntu.com e-mail addresses to contribute to Debian, shows that somehow the person is not quite committed to what it means to make a free operating system. In the words of Debian’s founder, “If Ubuntu is part of the Debian family, then we all win if Ubuntu is a success.” I don’t mean to say there aren’t valid reasons for only paying attention to one’s own direct work: one of those alluded to in the first link in this paragraph, a simple lack of time to understand systems other than Debian, is perfectly understandable. A couple of systems I maintain or know of at MIT, while technically free software, were written with so much of MIT’s infrastructure tacitly assumed at every step that bringing it even to another university is a nontrivial task. But this is something that we want to see, someday, when we have the time to generalize the system or can convince someone else to try.

Because people who develop free software and believe in the principles of free software should do so with not just an expectation but a hope that they will find their work reused in places that they never imagined.