Sunday, January 24, 2010

New Gmail's COMPRESS=DEFLATE feature

Two days ago I noticed that Google implemented RFC4978 for their mail service. In short, Gmail is able now to compress all traffic with deflate method. Wanderlust shall support this feature!

Unfortunately, my attempts to find ready-to-use deflate/inflate code in Emacs Lisp have been failed. Even it is impossible to access zlib functions from Emacs Lisp (Emacs links with -lz) without hacking Emacs first. So I decided to write a wrapper, which will stay in between of Emacs and gnutls-cli, which I use for TLS. Gnutls-cli is fully text-oriented client, so it has problem with handling binary data from standard input. I did a small patch to fix it:

--- src/cli.c.orig      2009-06-02 20:59:32.000000000 +0200
+++ src/cli.c   2010-01-24 17:48:05.636222100 +0100
@@ -749,7 +749,8 @@ after_handshake:
       if (FD_ISSET (fileno (stdin), &rset))
-         if (fgets (buffer, MAX_BUF, stdin) == NULL)
+         int bytes;
+         if ((bytes = read (fileno(stdin), buffer, MAX_BUF)) < 0)
              if ( == 0)
@@ -778,11 +779,13 @@ after_handshake:
          if (crlf != 0)
              char *b = strchr (buffer, '\n');
-             if (b != NULL)
+             if (b != NULL) {
                strcpy (b, "\r\n");
+               bytes++;
+             }
-         ret = socket_send (&hd, buffer, strlen (buffer));
+         ret = socket_send (&hd, buffer, bytes);
          if (ret > 0)

The wrapper is mostly functional now, but I need to beatify it first before releasing to public.

As for results... Deflate is known for its ability to recognize similar patterns and replace them with only one pattern definition and references for the rest. I tried to do uid search uid 1:* undeleted in my Inbox folder. At first, Gmail sent me 12401 byte of deflated text. For the same query, sent for 2nd time, it returned only 581 byte, which consisted mostly of the references to already received patterns. Usual uncompressed text is of 25993 bytes. That's pretty amazing!

1 comment:

13-49 said...

Gnutls path was accepted upstream yesterday.