One of the things that I have always found annoying about using iCloud Mail (and MobileMe before it) with iOS is that the push notification implementation is actually somewhat “incomplete” — at least for those of us who are used to Exchange ActiveSync, Blackberry Enterprise Server or even a desktop mail client such as Apple Mail.

What I mean by this is that iCloud will happily notify your iOS device of new e-mail messages as they come in, complete with an updated badge count on the Mail icon and even a banner and entry in the Notification Center in iOS 5. However, that’s about as far as it goes; read or file a message on another device, and the notification remains, stuck there seemingly forever. The count will adjust itself as more new mail comes in, but it will never actually disappear unless you open the Mail app, often only to see that you actually don’t have any new messages.

This was bad enough when the iPhone was the only iOS device that most people were using, as it still ended up being a problem when reading messages on a desktop computer. With the iPad in the mix, however, it becomes a real issue if you’re using push notifications on all of your devices. One way around it, of course, is to give up on immediate mail notifications and set your device to “Fetch” new mail every 15 minutes. You’ll still have a stale badge count in many cases until the next fetch interval, but at least it will have an opportunity to sort itself out every 15 minutes (accounts set to “Push” actually never fetch mail on any kind of schedule; they only do so when a push notification is received for that account, so ironically in that way they’re actually less functional than a standard IMAP account).

Perhaps not surprisingly, it seems that while OS X Lion Server has implemented push notifications for e-mail, it’s done the exact same thing as iCloud — notifications only go out when a new message comes in.

Unlike iCloud, however, with Lion Server I actually have direct control over the box that is sending out the push notifications. This led me to investigate further, on the assumption that if it’s my server initiating the notifications (albeit sent via Apple’s Push Notification servers), I should be able to generate such notifications myself on demand or as part of another programmatic routine.

The good news is that it turns out this is actually quite possible, although it required a fair bit of research and digging to figure out how all of the pieces fit together as documentation on many of Apple’s internal services on Lion Server is virtually non-existent, at least in any Apple-specific form.

Fortunately, however, many of the individual components are open source (e.g. postfix, dovecot, etc), and generic documentation can be easily be found for these. Apple also maintains an open source repository for much of its own code. This information, combined with digging through the actual configuration files on my own server helped me figure out enough of the mail push notification process to be able to bend it to my own will.

The Push Notification services on Lion Server are a bit convoluted, but basically operate using an intermediate Jabber server. Services such as Mail, Calendar and Contacts simply open a TCP port to the jabberd process running on the same box and send notifications via XMPP PubSub. Jabberd sends these through an APNBridge service to Apple’s Push Notification servers, which handle the actual sending of notifications out to the iOS devices.

A packet trace (tcpdump) on localhost (lo0) actually provides a complete transcript of a typical push notification session, although it’s important to note that persistent sessions are used; a session is initiated when the appropriate notification process first starts up (generally at server boot time) and is maintained throughout the life of the process — individual notifications are sent through this already open channel. So in other words, to get a complete trace, you’ll have to stop and reload the necessary components (e.g. the push_notify daemon for Mail), otherwise you’ll get the PubSub notification commands but miss all of the initial connection setup and authentication procedures.

My first thought was to try and build the necessary XMPP commands and pass them through to jabberd directly via TCP. Ultimately, however, this is a bit more complicated than it looks due to challenge-response authentication requirements and the need to supply the information necessary to identify specific users and devices. Basically, the Jabber server and APNBridge is simply the final step in sending the notification out, and expects a fully-formed notification request to have been created at an earlier stage.

However, it turns out that push notifications from Lion’s Mail server are generated through a custom plug-in that Apple has written for Dovecot, which is the open-source IMAP server used in Lion. The plug-in hooks into the “deliver_hook” in Dovecot, so it is specially written to only fire off when new mail arrives in a given mailbox. Fortunately, the source code for the plug-in is published by Apple and available from here.

By examining the code, it seems what actually happens here is that the plug-in communicates with a Unix socket (/var/dovecot/push_notify), sending a payload that is ultimately just a user name of the mailbox in question, using a specific data structure. Another process on Lion server, push_notify listens on this Unix socket and takes care of the necessary transmogrification of the user name into an XMPP request that it submits to the jabberd process on the same box (source code for push_notify can also be found online).

The Unix socket between the Dovecot plug-in and the push_notify daemon process is actually the easiest place to hook in. Unlike the Jabber process, there is no challenge/response authentication or any other kind of security on the Unix socket. Further, the only information that it requires fed to it is a simple user name, albeit it wrapped up in a specific data structure. So, with a quick bit of hacking on Apple’s original push-notify-plugin.c source code, I was able to come up with the following:

	/* Adapted from push-notify-plugin.c found at  
	http://www.opensource.apple.com/source/dovecot/dovecot-239/dovecot/src/plugins/push-notify/push-notify-plugin.c  
	Modified version by Jesse Hollington to support on-demand invocation 
	from the command line on OS X Server 10.7.  
	Requires push-notify-plugin.h available from  
	http://www.opensource.apple.com/source/dovecot/dovecot-239/dovecot/src/plugins/push-notify/push-notify-plugin.h  
	Portions Copyright (c) 2008-2011 Apple Inc. All Rights Reserved.  
	This file contains Original Code and/or Modifications of Original Code
	as defined in and that are subject to the Apple Public Source License 
	Version 2.0 (the 'License'). You may not use this file except in compliance
	with the License. 
	Please obtain a copy of the License at http://www.opensource.apple.com/apsl/ 
	and read it before using this file.  
	The Original Code and all software distributed under the License are distributed 
	on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR 
	IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES, 
	INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY, 
	FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT. 
	Please see the License for the specific language governing rights and limitations under the License.  
	*/  

	#include <sys/un.h>  
	#include <sys/stat.h>  
	#include <sys/socket.h>  
	#include <sys/unistd.h>  
	#include <string.h>  
	#include <stdio.h>  
	#include <errno.h>  
	#include "push-notify-plugin.h"  
	                 
	struct message_info {  
			const char     *from;  
			const char     *subj;  
			};  
			
	//     main ()  
	main(int argc, char *argv[])  
		{  
		int debug = 0;  
		int notify_sock = 0;  
		const char *sock_patH = "/var/dovecot/push_notify";  
		ssize_t rc = 0;  
		socklen_t sock_len = 0;  
		struct sockaddr_un     sock_addr;  
		struct msg_data_s     msg_data;  
		notify_sock = socket( AF_UNIX, SOCK_DGRAM, 0 );  
		if ( notify_sock < 0 ) {  
			printf( "pushmailnow: open socket: \"%s\" failed.\n", sock_path );  
			}
		sock_addr.sun_family = AF_UNIX;  
		strlcpy( sock_addr.sun_path, "/var/dovecot/push_notify", sizeof(sock_addr.sun_path) );  
		sock_len = sizeof(sock_addr.sun_family) + strlen(sock_addr.sun_path) + 1;  
		rc = connect(notify_sock, (struct sockaddr *) &sock_addr, sock_len);  
		if ( rc < 0 ) {  
			printf( "pushmailnow: connect() to socket: \"%s\" failed: %s\n", sock_path, strerror(errno) );  
			}  
		memset( &msg_data, 0, sizeof( msg_data ) );  
		msg_data.msg = 3;  
		/* set user/account id */
		strlcpy( msg_data.d1, argv[1], sizeof(msg_data.d1) ); 
		rc = send(notify_sock, (void *)&msg_data, sizeof(struct msg_data_s), 0); 
		close(notify_sock);
		} 
	// main

This code requires the push-notify-plugin.h header file available from here; other header files are simply part of the standard OS X Developer tools. Running this code through gcc creates a binary that can be used to send out a push notification on-demand simply by providing a username:

pushmailnow jesse

An interesting thing about Apple’s Mail Push Notifications is that unlike the Push Notifications generated by most third-party iOS apps, the Mail push notification doesn’t actually update the badge count directly at all; since the iOS Mail app has the privilege of running in the background on iOS devices, the Push Notification actually does nothing more than trigger a check for new mail — the exact same thing that would happen at regular intervals for an account set to Fetch instead of Push. It is this process of checking for new mail that updates the badge count,

What this means is that it’s not necessary for any of the server-based push processes to know or care about what’s actually in the user’s mailbox; it simply needs to initiate a push notification when something changes in the mailbox — it’s then up to the target iOS device to poll for new mail and update the mailbox (and badge count) in the process.

So armed with a command-line utility that can generate these push notifications on demand, the next step was to figure out how to actually trigger this process automatically when something actually changes in the user’s mailbox. Initially I started looking for hooks in the Dovecot mail server or thinking in terms of other background processes, but that line of thinking brought me to the realization that OS X already has a background process that can do the job for me: launchd.

Since Dovecot stores each mailbox in the file system, and launchd is capable of launching processes based on changes to files and folders, all we need to do is build a launchd configuration file that monitors the mail folders for a given user and runs the on-demand push app with the appropriate user name.

The Dovecot mailbox folders are stored under /Library/Server/Mail/Data/mail; each user’s mailbox is stored in an individual sub-folder under that with a GUID based name. The simplest way to determine the specific folder name for each user is by using the Server Admin utility found in Apple’s Server Admin Tools. In the Mail server configuration, the Maintenance section will display a list of all of the users configured on your server with the full paths to each of their mailbox folders.

Directly within each user’s mail folder is a sub-folder named “cur” which represents the messages stored in the user’s inbox, so this is the folder we want to monitor for each user. More information on the folder structure within each mailbox can be found at the Dovecot wiki, which states “new messages arrive in new and read shall be moved to cur by the clients.” In reality, messages are moved to the cur folder as soon as they are fetched by any IMAP client, whether they are marked read or not. Regardless, however, the standard push notification system in Lion server takes care of new mail notifications already, so we don’t need to concern ourselves with the “new” folder. However, as messages are read, deleted, or moved from any IMAP client, the cur folder gets updated. If launchd is watching this folder for changes, it can easily fire off our custom pushmailnow process, advising all of that user’s iOS devices (via Apple’s Push Notification Servers) to poll the server for new mail and update themselves accordingly. Since the top-level cur folder only applies to the user’s Inbox, this also does not trigger unnecessary push notifications when messages are modified or moved around in other sub-folders, which is fine since iOS has no way of polling other IMAP folders automatically anyway.

It’s important to note that this particular solution requires one launchd configuration file per user, so it may not be particularly scalable to large organizations. I’ve found it perfectly adequate for my three-user home server (not that my two-year-old daughter cares much about push notifications on her iPod touch :) ), but anybody looking to do something similar in a larger organization would probably want to write a custom app that could monitor several mailboxes as part of a single process. Such an app could either read the Dovecot mailbox configuration directly or simply use a static configuration file to map user names to mailbox folder names. I’ll leave that as an exercise to a more knowledgeable and adventurous programmer than myself.

The following is an example of one of the launchd scripts used to monitor the mailbox folder and fire off a push notification. I duplicated this for my other users, changing only the Label, ProgramArguments and WatchPaths directives, as appropriate, and dropped each of the scripts into my /Library/LaunchDaemons folder on the server. The pushmailnow obviously needs to be placed in an accessible path like /usr/sbin and made executable, but I expect anybody who knows how to run a server and compile code with gcc can probably figure that out.

<!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN” “http://
www.apple.com/DTDs/PropertyList-1.0.dtd”>

Label com.jdhnet.push-jesse RunAtLoad ProgramArguments /usr/sbin/pushmailnow jesse WatchPaths /Library/Server/Mail/Data/mail/0C69E1E7-6DC7-41B1-AD12-1C596AC7EBC8/cur

I actually figured all of this out back in late April but haven’t had time to finish writing about it until today. It’s been running on my server since that time without any problems at all and works very seamlessly. It was originally implemented on OS X Server 10.7.3 and survived the upgrade to 10.7.4 without any problems at all. I have no idea whether this solution will continue to work in Mountain Lion when it comes out, as I have no desire to try “preview” software on a production server.

It’s unfortunate that Apple has not chosen to include a similar solution in OS X Server or iCloud — particularly since it seems relatively straightforward to do. To be fair, this may be due to concerns about scalability; sending push notifications for every inbox update would significantly multiply the number of push notifications that Apple’s servers would need to deal with. My own three-user Mac Mini server isn’t likely going to cause any problems, but 100 million iCloud users could definitely create a more significant load. Sadly, however, this does mean that iCloud’s (and OS X Server’s) push mail notifications remain an incomplete solution for many users.