Understanding an email message encoded with
MIME can be very, very, very difficult. It can get frustrating due to
the number of options and different ways to do the actual
encoding. Now add to that the sometimes too-liberal interpretations of
the relevant RFCs by the email client designers and you will begin to
get the idea. This article will show you how this task can be
laughably simple thanks to Perl's extensive bag of tricks, CPAN
.
I started out with a simple and straightforward mission: Fetch an
Email from a POP mailbox and display it in a 7-bit, text-only capable
device. This article describes the different stages for a simple tool
that accomplishes this task, written in Perl with a lot of help from
CPAN modules. I hope this to be useful to other Perl folks who might
have a similar mission. Let's discuss each part of this task
in turn, as we read through mfetch, the script I prepared as
an example. Keep in mind that TIMTOWTDI.
Setting up the script
The first thing, as you know, is loading up all of the modules I
will be using. I'm sure you already know strict and
warnings. We'll see how do we use the rest of the modules a
bit later.
1: #!/usr/bin/perl
2:
3: # This script is (c) 2002 Luis E. Muñoz, All Rights Reserved
4: # This code can be used under the same terms as Perl itself. It comes
5: # with absolutely NO WARRANTY. Use at your own risk.
6:
7: use strict;
8: use warnings;
9: use IO::File;
10: use Net::POP3;
11: use NetAddr::IP;
12: use Getopt::Std;
13: use MIME::Parser;
14: use HTML::Parser;
15: use Unicode::Map8;
16: use MIME::WordDecoder;
17:
18: use vars qw($opt_s $opt_u $opt_p $opt_m $wd $e $map);
19:
20: getopts('s:u:p:m:');
21:
22: usage_die("-s server is required\n") unless $opt_s;
23: usage_die("-u username is required\n") unless $opt_u;
24: usage_die("-p password is required\n") unless $opt_p;
25: usage_die("-m message is required\n") unless $opt_m;
26:
27: $opt_s = NetAddr::IP->new($opt_s)
28: or die "Cannot make sense of given server\n";
Note the lines 27 and 28, where I use NetAddr::IP to convert whatever the user gave us through the
-s option into an IP address. This is a very common use of
this module, as its new() method will convert many common IP
notations into an object I can later extract an IP address from. It
will even perform a name resolution for us if required. So far,
everything should look familiar, as a lot of scripts start like this
one.
It is worth noting that the error handling in lines 22-25 is not a
brilliant example of good coding or documentation. It is much better
practice to write your scripts' documentation in POD, and use a module
such as Pod::Usage to provide useful error messages to the user. At
the very least, try to provide an informative usage message. You can
see the usage_die() function if you download the complete script.
Fetching a message via POP3
On to deeper waters. The first step in parsing a message, is getting
at the message itself. For this, I'll use Net::POP3, which implements the POP3 protocol described in
RFC-1939. This is all done in the code below.
30: my $pops = Net::POP3->new($opt_s->addr)
31: or die "Failed to connect to POP3 server: $!\n";
32:
33: $pops->login($opt_u, $opt_p)
34: or die "Authentication failed\n";
35:
36: my $fh = new_tmpfile IO::File
37: or die "Cannot create temporary file: $!\n";
38:
39: $pops->get($opt_m, $fh)
40: or die "No such message $opt_m\n";
41:
42: $pops->quit();
43: $pops = undef;
44:
45: $fh->seek(0, SEEK_SET);
At line 30, a connection to the POP server is attempted. This is a
TCP connection, in this case to port 110. If this connection succeeds,
the USER and PASS commands are issued at line 33,
which are the simplest form of authentication supported by the POP
protocol. Your username and password are being sent here through the
network without the protection of cryptography, so a bit of caution is
in order.
Net::POP3 supports many operations defined in the POP
protocol that allow for more complex actions, such as fetching the
list of messages, unseen messages, etc. It can also fetch messages for
us in a variety of ways. Since I want this script to be as lightweight
as possible (i.e., to burn as little memory as possible), I want to
fetch the message to a temporary on-disk file. The temporary file is
nicely provided by the new_tmpfile method of IO::File in line 36, which returns a file handle to a
deleted file. I can work on this file, which will magically disappear
when the script is finished.
Later, I instruct the Net::POP3 object to fetch the required
message from the mail server and write it to the supplied filehandle
using the get method, on line 39. After this, the connection
is terminated gracefully by invoking quit and destroying the
object. Destroying the object insures that the TCP connection with the
server is terminated, freeing the resources being held in the POP
server as soon as possible. This is a good programming practice for
network clients.
The interaction required by mfetch with the POP server is
really simple, so I'm not making justice to Net::POP3. It
provides a very complete implementation of the protocol, allowing for
much more sophisticated applications.
Note that in line 45, I rewind the file so that the fetched
message can be read back by the code that follows.
For this particular example, we could also have used
Net::POP3Client, which provides a somewhat similar
interface. The code would have looked more or less like the following
fragment.
1: my $pops = new Net::POP3Client(USER => $opt_u,
2: PASSWORD => $opt_p,
3: HOST => $opt_s->addr)
4: or die "Error connecting or logging in: $!\n";
5:
6: my $fh = new_tmpfile IO::File
7: or die "Cannot create temporary file: $!\n";
8:
9: $pops->HeadAndBodyToFile($fh, $opt_m)
10: or die "Cannot fetch message: $!\n";
11:
12: $pops->Close();
Parsing the MIME structure
Just as email travels inside a sort of envelope (the headers), complex
messages that include attachments and generally, HTML messages, travel
within a collection of MIME entities. You can think of these
entities as containers that can transfer any kind of binary
information through the Email infrastructure, which in general does
not know how to deal with 8-bit data. The code reproduced below, takes
care of parsing this MIME structure.
47: my $mp = new MIME::Parser;
48: $mp->ignore_errors(1);
49: $mp->extract_uuencode(1);
50:
51: eval { $e = $mp->parse($fh); };
52: my $error = ($@ || $mp->last_error);
53:
54: if ($error)
55: {
56: $mp->filer->purge; # Get rid of the temp files
57: die "Error parsing the message: $error\n";
58: }
Perl has a wonderful class that provides the ability to understand
this MIME encapsulation, returning a nice hierarchy of objects that
represent the message. You access this facilities through the MIME::Parser class, part of the MIME-Tools bundle. MIME::Parser returns a hierarchy
of MIME::Entity objects representing your message. The parser
is so smart, that if you pass it a non-MIME email, it will be returned
to you as a text/plain entity.
MIME::Parser can be tweaked in many ways, as its
documentation will show you. One of the points where this toggling
might be important, is the decoding process. Remember that I need to
be as light in memory usage as possible. The default behavior of
MIME::Parser involves the use of temporary files for decoding
of the message. These temporary files can be spared and core memory
used instead by invoking output_to_core(). Before doing this,
note all the caveats cited in the module's documentation. The most
important one is that if a 100 MB file ends up in your inbox, this
whole thing needs to be slurped into RAM.
In line 47 I create the parser object. The call to
ignore_errors() in line 48 is an attempt to made this parser
as tolerant as possible. extract_uuencode() in line 49, takes
care of pieces of the email that are uu-encoded automatically,
translating them back into a more readable form. The actual request
to parse the message, available through reading the $fh
filehandle, is in line 51. Note that it is enclosed in an
eval block. I have to do this as the parser might throw an
exception if certain errors are encountered. The eval allows
me to catch this exception and react in a way that is sensible to this
application. In this case, I want to be sure that any temporary file
created by the parsing process is cleared by a call to
purge(), as seen in lines 56 and 57.
Setting up the HTML parser
Parsing HTML can be a tricky and tedious task. Thankfully, Perl has a
number of nice ways to help you do this job. A number of excellent
books such as The Perl Cookbook (from O'Reilly &
Associates) has a couple of recipes that came very close to what I
needed, especially recipe 20.5, "Converting HTML to ASCII",
which I reproduce below.
1: use HTML::TreeBuilder;
2: use HTML::FormatText;
3:
4: $html = HTML::TreeBuilder->new();
5: $html->parse($document);
6:
7: $formatter = HTML::FormatText->new(leftmargin => 0, rightmargin => 50);
8:
9: $ascii = $formatter->format($html);
I did not want to use this recipe because of two reasons: I needed
fine-grained control in the HTML to ASCII conversion and I wanted to
have as little impact as possible in resources. I did a small benchmark that shows the kind of performance
difference among the two options while parsing a copy of one of my web
articles. The result below shows that the custom parser explained
later runs faster than the Cookbook's recipe. This does not mean that
the recipe or the modules it uses are bad. This result simply means
that the recipe is actually doing a lot of additional work, which just
happens to not be all that useful for this particular task.
bash-2.05a$ ./mbench
Benchmark: timing 100 iterations of Cookbook's, Custom...
Cookbook's: 73 wallclock secs (52.82 usr + 0.00 sys = 52.82 CPU) @ 1.89/s (n=100)
Custom: 1 wallclock secs ( 1.17 usr + 0.00 sys = 1.17 CPU) @ 85.47/s (n=100)
Rate Cookbook's Custom
Cookbook's 1.89/s -- -98%
Custom 85.5/s 4415% --
HTML::FormatText does an awesome job of converting the HTML
to plain text. Unfortunately I have a set of guidelines that I need to
follow in the conversion and that are not compatible with the output
of this module. Additionally, HTML::TreeBuilder does an excellent job of parsing an HTML
document, but produces an intermediate structure - the parse tree -
that in my case, wastes resources.
However, Perl has an excellent HTML parser in the HTML::Parser module. In this case, I chose to use this class
to implement an event-driven parser, where tokens (syntactic elements)
in the source document cause the parser to call functions I
provide. This allowed me complete control on the translation while
sparing the intermediate data structure.
Converting HTML to text is a lossy transformation. This means that
what goes out of this transformation is not exactly equivalent to what
went in in the first place. Pictures, text layout, style and a few
other information elements are lost. My needs required that I noted
the existence of images as well as a reasonably accurate rendition of
the page's text, but nothing else. Remember that the target device can
only display 7-bit text, and this is within a very small and limited
display. This piece of code sets up the parser to do what I need.
62: my $parser = HTML::Parser->new
63: (
64: api_version => 3,
65: default_h => [ "" ],
66: start_h => [ sub { print "[IMG ",
67: d($_[1]->{alt}) || $_[1]->{src},"]\n"
68: if $_[0] eq 'img';
69: }, "tagname, attr" ],
70: text_h => [ sub { print d(shift); }, "dtext" ],
71: ) or die "Cannot create HTML parser\n";
72:
73: $parser->ignore_elements(qw(script style));
74: $parser->strict_comment(1);
Starting on line 71, I set up the HTML::Parser
object that will help me do this. First, I tell it I want to use the
latest (as of this writing) interface style, which provides more
flexibility than earlier interfaces. On line 65, I tell the object
that by default, no parse events should do anything. There are other
ways to say this, but the one shown is the most efficient.
Lines 66 through 69 define a handler for the start
events. This handler will be called each time an opening tag such as
<a> or <img> is recognized in the source
being parsed. Handlers are specified as a reference to an array whose
first element tells the parser what to do and its second element,
tells the parser what information to pass to the code. In this
example, I supply a function that for any img tag, will
output a hopefully descriptive text composed with either the
alt or the src attributes. I request this handler to
be called with the name of the tag as the first argument and the list
of attributes as further arguments, through the string "tagname,
attr" found in line 69. The d() function will be
explained a bit later, but it has to do with decoding its
argument.
The text event will be triggered by anything inside tags
in the input text. I've set up a simpler handler for this event that
merely prints out whatever is recognized. I also request that HTML
entities such as € or ñ be
decoded for me through the string "dtext" on line 70. HTML
entities are used to represent special characters outside the
traditional ASCII range. In the interest of document accuracy, you
should always use entities instead of directly placing 8-bit
characters in the text.
Some syntactic elements are used to enclose information that is not
important for this application, such as
<style>...</style> and
<script>...</script>. I ask the parser to ignore
those elements with the call to ignore_elements() at line
73. I also request the parser to follow strict comment syntax through
the call to strict_comment() on line 74.
Setting up the Unicode mappings
MIME defines various ways to encode binary data depending on the
frequency of octets greater than 127. With relatively few high-bit
octets, Quoted-Printable encoding is used. When many high-bit
octets are present, Base-64 encoding is used instead. The
reason is that Quoted-Printable is slightly more readable but very
inefficient in space while Base-64 is completely unreadable by
standard humans and adds much less overhead in the size of encoded
files. Often, message headers such as the sender's name are encoded
using Quoted-Printable when they contain characters such as a
'ñ'. These headers look like From:
=?ISO-8859-1?Q?Luis_Mu=F1oz?= <some@body.org> and should be
converted to From: Luis Muñoz <some@body.org>. In plain
english, Quoted-Printable encoding is being used to make the extended
ISO-8859-1 characters acceptable for any 7-bit transport such as
email. Many contemporary mail transport agents can properly handle
message bodies that contain high-bit octets but will choke on headers
with binary data, in case you were wondering why all this fuzz.
Lines 92 through 102 define setup_decoder(), which can use
the headers contained in a MIME::Head object to setup a suitable decoder based on the
MIME::WordDecoder class. This will translate instances of
Quoted-Printable text, to its high-bit equivalent. Note that I
selected ISO-8859-1 as the default in case no proper character set can
be identified. This was a sensible choice for me, as ISO-8859-1
encloses spanish characters, which happen to be my native language.
92: sub setup_decoder
93: {
94: my $head = shift;
95: if ($head->get('Content-Type')
96: and $head->get('Content-Type') =~ m!charset="([^\"]+)"!)
97: {
98: $wd = supported MIME::WordDecoder uc $1;
99: }
100: $wd = supported MIME::WordDecoder "ISO-8859-1" unless $wd;
101: }
But this clever decoding is not enough. Getting at the original
high-bit characters is not enough. I must recode these high characters
into something usable by the 7-bit display device. So in line 76 I set
up a mapping based on Unicode::Map8. This module can convert 8-bit characters such
as ISO-8859-1 or ASCII into wider characters (Unicode) and then back into
our chosen representation, ASCII, which only defines 7-bit
characters. This means that any character that cannot be properly
represented, will be lost, which for our application is acceptable.
76: $map = Unicode::Map8->new('ASCII')
77: or die "Cannot create character map\n";
The decoding and character mapping is then brought together at line
90, where I define the d() function, that simply invokes the
adequate MIME decoding method, transforms the resulting string into
Unicode via the to16() method and then, transforms it back
into ASCII using to8() to insure printable results
in our device. Since I am allergic to warnings related to
undef values, I make sure that decode() always get a
defined string to work with.
90: sub d { $map->to8($map->to16($wd->decode(shift||''))); }
As you might notice if you try this code, the conversion is again
lossy because there are characters that does not exist in
ASCII. You can experiment with the addpair() method
to Unicode::Map8 in order to add custom character
transformations (i.e., '€' might be 'E'). Another way to achieve
this, is through deriving a class from Unicode::Map8 and
implementing the unmapped_to8 method to supply your own
interpretation of the missing characters. Take a look at the module's
documentation for more information.
Starting the decode process
With all the pieces in place, all that's left is to traverse the
hierarchy of entities that MIME::Parser provides after
parsing a message. I implemented a very simple recursive function
decode_entities starting at line 103. This is a recursive
function because recursion comes naturally as a way to handle trees
such as those produced by MIME::Parser. At
least to me.
103: sub decode_entities
104: {
105: my $ent = shift;
106:
107: if (my @parts = $ent->parts)
108: {
109: decode_entities($_) for @parts;
110: }
111: elsif (my $body = $ent->bodyhandle)
112: {
113: my $type = $ent->head->mime_type;
114:
115: setup_decoder($ent->head);
116:
117: if ($type eq 'text/plain')
118: { print d($body->as_string); }
119: elsif ($type eq 'text/html')
120: { $parser->parse($body->as_string); }
121: else
122: { print "[Unhandled part of type $type]"; }
123: }
124: }
The condition at line 107 asks if this part or entity contains
other parts. If it does, it extracts them and invokes itself
recursively to process each sub-part at line 109.
If this part is a leaf, its body is processed. Line 111 gets it as a
MIME::Body object. On line 155 I setup a decoder for this
part's encoding and based on the type of this part, taken at line 113,
the code on lines 117 to 122 call the proper handlers.
In order to fire the decoding process, I call
decode_entities() with the result of the MIME decoding of the
message on line 86. This will invoke the HTML parser when needed and
in general, produce the output I look for in this example. After this
processing is done, I make sure to wipe temporary files created by
MIME::Parser on line 88. Note that if the message is not
actually encoded with MIME, MIME::Parser will arrange for you
to receive a single part of type text/plain that contains the
whole message text, which is perfect for our application.
86: decode_entities($e);
87:
88: $mp->filer->purge;
And that's about it
After these less than 130 lines of code, I can
easily fetch and decode a message, such as in the following
example:
bash-2.05a$ ./mfetch -s pop.foo.bar -u myself \
-p very_secure_password -m 5
Date: Sat, 28 Dec 2002 20:14:37 -0400
From: root <root@foo.bar>
To: myself@foo.bar
Subject: This is the plain subject
This is a boring and plain message.
More complex MIME messages can also be decoded. Look at this
example where I dissect a dreaded piece of junk mail, but don't
worry. I used head to spare you pages and pages of worthless
image links:
bash-2.05a$ ./mfetch -s pop.foo.bar -u myself \
-p very_secure_password -m 2 | head -20
Date: Sun, 22 Dec 2002 23:22:25 -0400
From: Luis Muoz <lem@foo.bar>
To: Myself <myself@foo.bar>
Subject: Fwd: Get $860 Free - Come, Play, Have Fun!
Begin forwarded message:
> From: Cosmic Offers <munged@migada.com.INVALID>;
> Date: Sun Dec 22, 2002 20:59:43 America/Caracas
> To: spam@victim.net
> Subject: Get $860 Free - Come, Play, Have Fun!
>
>
[IMG http://www.migada.com/email/Flc_600_550_liberty_mailer_.gif]
[IMG http://www.migada.com/email/Flc_600_550_liberty_mail-02.gif]
[IMG http://www.migada.com/email/Flc_600_550_liberty_mail-03.gif]
[IMG http://www.migada.com/email/Flc_600_550_liberty_mail-04.gif]
If you're curious, please download the complete
script and play with it a bit. I hope this tutorial and its
related script to be as helpful for you as it has been for me