March 2006 Archives

In part 1, I showed how to set environment variables using a shell script to enable malloc debugging features in a Dashboard widget. For some dumb reason, this wasn't good enough for me. No... I had to take this nice simple shell script, and re-write it in Ruby. The benefit? It's easier to conditionally set environment variables, e.g. set MallocStackLogging only for certain widgets. This is way overkill, since it's pretty easy to just edit the shell script with sudo when needed. It's not like widgets are launched so often where this really matters. But what's done is done, and I may as well share my big waste of time with the world:

#!/usr/bin/ruby -w

require 'etc'

def is_admin?
  username = Etc.getlogin
  return (Etc.getgrnam("admin").mem.include? username)
end

$DashboardClient = $0 + ".orig"
$ARGS = ARGV

conf_file = File.join(ENV['HOME'], ".MacOSX", "DashboardClientConfig.rb")
if is_admin? && (File.exist? conf_file)
    load conf_file
end

exec $DashboardClient, *$ARGS

The major difference is that all customization goes into ~/.MacOSX/DadshboardClientConfig.rb. The benefit is I don't have to run sudo to change environment variables. Also, each user has their own customization. As a precaution, only users in the admin group may customize Dashboard. This way, you're average user can't accidentally shoot themselves in the foot. Overkill #1: A user isn't going to "accidentally" create ~/.MacOSX/DadshboardClientConfig.rb without knowing what they are doing. Overkill #2: I'm the only user on my development machine. My excuse: I just wanted to learn how to use Ruby's Etc class. But like I said, what's done is done. So how is this used? Here's a sample DashboardClientConfig.rb:

case $ARGS[0]
when /Hello\ World\.wdgt/
  ENV['MallocStackLogging'] = 'YES'
when /Goodbye\ World.wdgt/
  ENV['DYLD_INSERT_LIBRARIES'] = '/usr/lib/libMallocDebug.A.dylib'
end

Since the path to the actual widget is the first command line argument, you can do per-widget customization. So there you have it. An overly complicated method of setting environment variables.

Finding memory leaks in a Dashboard widget that uses an Objective-C widget plugin is a little tricky. All the malloc debugging features are enabled using special environment variables. The problem is that DashboardClient.app is launched automatically by Dock.app. So you can't just set these environment variables in a terminal window and expect them to work. You also can't start DashboardClient.app manually. There are some magic command line arguments that Dock.app passes in.

You could use ~/.MacOSX/environment.plist to set the environment variables, but that would require logging in and logging out anytime you wanted to set them. Also all applications would get these variables which is not at all desirable. You could try and reverse engineer the command line arguments passed to DashboardClient.app, but that could be difficult. The solution I found is a nice little shell script that replaces the DashboardClient.app binary. The first step is to rename the binary:

% cd /System/Library/CoreServices/Dock.app
% cd Contents/Resources/DashboardClient.app
% cd Contents/MacOS                        
% sudo mv DashboardClient DashboardClient.orig

You could change directories in one step. I just did this for ease of reading. Now create a shell script named DashboardClient with the following contents:

#!/bin/sh

# export MallocStackLogging=YES
# export DYLD_INSERT_LIBRARIES=/usr/lib/libMallocDebug.A.dylib
# export DYLD_FRAMEWORK_PATH="${HOME}/Library/Frameworks"

exec "${0}.orig" "$@"

Finally, make it executable:

% sudo chmod +x DashboardClient

The key is the last line of the shell script which launches the original binary with the command line arguments passed in. However, this now gives us the opportunity to set some environment variables before launching it. I've given three examples of how this may be used in the comments.

The first example turns on malloc stack logging, which is really nice for leaks (and the Leaks GUI). The second example loads libMallocDebug.A.dylib, which allows you to attach to it with MallocDebug. The final example has nothing to do with memory leaks, but it allows you to override system frameworks with your own. One possible use for this is to use a custom build of WebKit with Dashboard.

You can always go back to the original binary by removing the shell script and renaming the binary again. Also, keep in mind that a system update may very well overwrite these changes, so you may have to redo them. But all in all, this is a nice easy hack to enable malloc debugging on one instance of a Dashboard widget.

Update: Want per-widget and per-user control over environment variables? Read part 2 to see how to do this in Ruby instead of a shell script.

I've been a NetNewsWire user for almost 2.5 years now. It's a wonderful program. It's a Mac OS X application through and through, and lives up to the quality I expect of Mac applications. However, after reading this post regarding syncing in NNW 2.1, I'm getting the feeling that NNW has jumped the shark (and you thought only TV shows could do that!). I've been worried about this ever since NewsGator acquired NNW about six months ago. Syncing, as of NNW 2.1 will, surprise, surprise have a NewGator syncing feature. This is great.... if you care about what NewsGator offers. What is not great is the fact that .Mac syncing is still going to suck. It doesn't look like the problems that exist in 2.0 are not going to get addressed. When NNW 2.0 first came out with the .Mac syncing, I was really excited. .Mac syncing works great with many of Apple's applications, and as of OS X 10.4 any application can hook into .Mac syncing. I was excited to have this kind of syncing for my feed reader. However, the .Mac syncing of NNW is really sub-par. It doesn't sync automatically. You have to manually do it, or do it on app startup and shutdown. However, if you leave leave the app running for days at a time, like I do, it's essentially manual. The syncing also takes place in the foreground, and you're not allowed to do anything until the sync is done, which can take anywhere from 30-60 seconds. Also, while it does sync the subscription lists, it doesn't sync the order and grouping of them. This means if I rearrange feeds on one computer, I have to do it all over again on the other machines. But I was willing to cut some slack for NNW 2.0. Syncing is a hard feature to implement well, and this was the 1.0 version of it. Surely all of these would be addressed in the next version. Unfortunately, NNW 2.1 doesn't look like it's going to address any of the .Mac syncing issues. The "fix" is to not sync via .Mac at all, but sync via NewsGator.

Now I completely understand the reasons behind the decision to sell to NewsGator, among them syncing to multiple platforms (like Windows, handhelds and the web) and better customer support. These are things the lone author just couldn't do all on his own. And I'm all for adding NewsGator syncing in addition to .Mac syncing, but to add it at the expense of it seems unfortunate. Surely I can hear chants of: "Quit your whining and use NewsGator! It's free after all." The point is .Mac, love it or hate it, is Apple's sanctioned mechanism for syncing. It's used by all of Apple's apps and it's got great OS integration. Thus in my opinion, any "true" Mac application that supports syncing should support robust .Mac syncing. It's not as if using NewsGator would save me any money. I still will be paying for and using .Mac to sync my address book, calendars, mail accounts, and keychains. All it does is add another account to keep track of. Another service provider that can have downtime. Another corporation to mine my personal data. And I'm sorry, but I don't necessarily trust NewsGator. Are they going to sell my subscription lists to advertisers? Will I have to endure ads to view my subscription lists online? Will the Consumer Standard Plan be free forever? Will I one day be paying for .Mac and NewsGator? Maybe I am just being paranoid and whiny since a feature I care about isn't getting the attention I feel it needs. Well so be it. I've been called worse. In the meantime, I can only hope NetNewsWire 2.2 will offer excellent NewsGator AND .Mac syncing.

While looking for something completely unrelated, I stumbled across the solution to a problem I encountered a while back: how to create a static library containing Objective-C code. Objective-C does not define linker symbols the same way that C and C++ do, due to the dynamic nature of the language. Specifically, a symbol is not created for each method, just each class. The problem manifests itself when trying to create a static library containing categories on existing classes, say NSString. When compiling against this static library, the code for the categories is not linked into the final executable. The result is a runtime "selector not recognized" exception.

In order to understand why, we must go back to how static libraries are implemented in Unix, and hence OS X. Static libraries are just a collection of object files combined into a single file called an archive. As an optimization, the linker only chooses the object files that are required to resolve symbols. This means that even if a static library contains 100 object files, the linker may only pick one of them, thus reducing the amount of code in the final executable. Back to Objective-C: if I use a class, say NSString, the compiler generates an undefined symbol for the the class, but not each method I use. Thus if the class and a category are in two different object files, the linker has no way to know it needs to link in the category object file. This is a non-issue for frameworks, which are dynamic libraries, where linking happens at runtime. But for static libraries, this results in code for the category not being included in the final executable. The solution: link the executable with the -ObjC option. The ld(1) documentation for this option is:

Loads all members of static archive libraries that define an Objective C class or a category.

Aha... just what I want! This forces the linker to pull in the object file with the category into the executable. One downside to this is you lose the optimization the linker typically does with static libraries. If I link against a static library with 100 categories, each in separate object files, and I only use one, the linker pulls them all in. This also puts the burden on the library user to use a non-standard linker flag. It would be nice to link the static library specially such that the library creator could tag this library as containing Objective-C. But due to the limitations of Unix static libraries being a collection of object files, this is not possible. I've long been annoyed with static libraries on Unix, and this is just another reason to dislike them. The way Windows does static libraries is actually much nicer. A full rant will have to wait for another post. However, static libraries do have their uses, and it's nice to know it is possible to put Objective-C code in them.

I needed to Base64 encode binary data in some Cocoa Objective-C code I was writing, but unfortunately this is not part of the standard library. I did find some code to do it. However, the license for the code was unclear. And did I really need an AliteVec implementation? CocoaDev has a wiki page about encoding and decoding to Base64. Most of that code didn't look too robust, though. Someone mentioned using the openssl command line tool, and even wrote up some code that spawns an NSTask to invoke openssl. That got me thinking... OS X ships with OpenSSL, so why not use the library directly?

A quick keyword search on the man pages lead me to BIO_f_base64(3). It even had some code examples! Now I just needed to figure out how to use the Cocoa classes instead of C file streams, and I'd be all set. Since this is my first attempt at using OpenSSL, I'm going to walk through the Base64 encoding code. While not terribly complex, it may be of use to someone who's new to Objective-C and/or new to interfacing Objective-C to Unix/C libraries. For the impatient who just want the code, I posted encoding and decoding implementations on the CocoaDev Base64 wiki page. Or download an Xcode 2.2 project, complete with unit tests. It's released under an MIT license.

Since NSData is the Cocoa class for holding binary data, I figured adding a category seemed like the perfect way to proceed. This effectively adds Base64 encoding methods to the NSData class:

@interface NSData (Base64)

- (NSString *) encodeBase64;
- (NSString *) encodeBase64WithNewlines: (BOOL) encodeWithNewlines;

@end

For the implementation of encodeBase64WithNewlines: I needed to figure out how to bridge NSData and OpenSSL. OpenSSL has its own I/O abstraction layer called BIO, which is essentially a stream API written in C. The memory BIOs, which back to a memory buffer, are just what I need. So the first step is to create a memory BIO that will contain the encoded data:

- (NSString *) encodeBase64WithNewlines: (BOOL) encodeWithNewlines;
{
    BIO * mem = BIO_new(BIO_s_mem());

Left like this, BIO_read() and BIO_write() will just copy bytes to the buffer, unmodified, which is not very useful. OpenSSL allows you to add and remove filters that modify data as it is read or written. Let's create a Base64 filter and add it to the chain:

    BIO * b64 = BIO_new(BIO_f_base64());
    if (!encodeWithNewlines)
        BIO_set_flags(b64, BIO_FLAGS_BASE64_NO_NL);
    mem = BIO_push(b64, mem);

Now as data is written to my memory BIO, it gets Base64 encoded on the way. The BIO_FLAGS_BASE64_NO_NL flag disables adding newlines to the encoded data, if the caller requested it. With the BIO chain setup, encoding the bytes contained by the NSData object is simple:

    BIO_write(mem, [self bytes], [self length]);
    BIO_flush(mem);

The final task is to get the encoded data out of the BIO and into an NSString. The memory BIOs have an API to get a pointer and length to its backing buffer:

    char * base64Pointer;
    long base64Length = BIO_get_mem_data(mem, &base64Pointer);

With a pointer to the raw bytes, it's just a matter of creating a new string from this data:

    NSString * base64String = [NSString stringWithCString: base64Pointer
                                                   length: base64Length];

Since I know this is ASCII data, I don't have to worry about Unicode encoding. Using this constructor, rather than one of the newer ones that takes an encoding parameter also means backward compatibility to older versions of OS X. Now that I've got a new NSString with the Base64 encoded data, I just need to clean up all the OpenSSL resources and return:

    BIO_free_all(mem);
    return base64String;
}

I've tested this on OS X 10.4.5, but it should work on OS X 10.2.x and 10.3.x as OpenSSL is included in the 10.2.8 and 10.3.9 SDKs of Xcode 2.2. Decoding Base64 is mostly the same except data is read from a memory BIO with a Base64 filter attached, rather than writing to one. See CocoaDev or the Xcode project for the decoding code.

While using the Unix command line, I often want to find a command I previously typed, e.g. the last "ssh" command I typed. Using the up and down arrows or "Ctrl-p" and "Ctrl-n" to go through each line is often very tedious because there's lots of other cruft in the history since the last time I typed "ssh". tcsh, the first "real" Unix shell I used, has a nice history search mechanism: type "ssh" then "ESC-p". It'll go back through your history and find the last time you typed "ssh". Hit "ESC-p" again , and it'll go to the one before that. Hit "ESC-n" and it'll go forward to the next one. Thus using "ESC-p" and "ESC-n" allows you to interactively go up and down through your history, like the up and down arrow keys, but only matching what you have typed so far. Also, "ESC" is an alias for the "META" key, which is mapped to "ALT" in most GUI terminal programs. Thus, you can just use "ALT-p" and "ALT-n".

My current shell is zsh, which I switched to maybe four years ago. By default, zsh doesn't have key bindings for "ESC-p" and "ESC-n". Since this was one of my favorite tcsh features, figuring out how to do this was a requirement for me to stick with zsh. Luckily, it has functions to do this, the key bindings just need to be setup. Here's the snippet from my ~/.zshrc:

# "^[" is what the "ESC" key looks like to zsh
bindkey "^[p" history-beginning-search-backward
bindkey "^[n" history-beginning-search-forward

Now, I've had my zsh setup like this for a very long time. But occasionally I'm forced to use bash or, more likely, some other program which uses GNU readline. The default GNU readline bindings for "ESC-p" or "ESC-n" do not behave like tcsh, so I lose one of my favorite features. I've tried using "Ctrl-r" to interactively search backward through history, however I find it's behavior annoying for a few reasons. First, it matches anywhere in the the command string, not just the start of a command. I find this matches many history commands that are irrelevant. Second, it only starts matching after you type "Ctrl-r". Finally, I always seem to hit "Ctrl-r" one too many times, and there doesn't seem to be a way to search forward in history. And again, luckily readline has functions to do history the way I like, the key bindings just need to be setup. readline looks in ~/.inputrc for customization, so I put these lines in there to get what I want:

# "\e" is what the "ESC" key looks like to readline
"\ep": history-search-backward
"\en": history-search-forward

Now, I can finally be (somewhat) happy in bash or any readline application.