Ask Hackaday: Is Bigger (E-mail) Better?

While pundits routinely predict the end of e-mail, we still get a ton of it and we bet you do too. E-mail has been around for a very long time and back in the day, it was pretty high-tech to be able to shoot off a note asking everyone where they wanted to go to lunch. What we had on our computers back then was a lot different, too. Consider that the first e-mail over ARPANET was in 1971. Back then some people had hardcopy terminals. Graphics were unusual and your main storage was probably a fraction of the smallest flash drive you currently have on your desk. No one was sending photographs, videos, or giant PDF files.

Today, things are different. Our computers have gigabytes of RAM and terabytes of storage. We produce and consume richly formatted documents, photographs at high resolutions, and even video. Naturally, we want to share those files with others, yet e-mail has turned up woefully short. Sure, some systems will offer to stash your large file in the cloud and send a link, but e-mailing a multi-megabyte video to your friend across town is more likely to simply fail. Why?

The Life of an E-mail

As you might expect, some of this is due to historical artifacts. The system wasn’t made to handle your latest vacation video. More importantly, e-mail is meant to be widely interoperable over a wide variety of systems. Sure, these days, it is unlikely any of your e-mails travel over a twisted pair linking two far-flung outposts, but it could.

To understand the genesis of the limits, you have to know about how an e-mail lives its life. It is a lot more complicated than you might expect. Today, mail travels using a protocol called SMTP — Simple Mail Transfer Protocol. Let’s take the simple cases first.

Let’s assume that we have a Hackaday.com e-mail address and that we run our own e-mail servers (we don’t, but go with me for a minute). A user connects to the SMTP server and sends e-mail to another address at Hackaday.com. That’s easy. The server recognizes that it is the final destination for the e-mail, so it accepts it and delivers it. What does deliver mean? That depends, but it usually means putting in some file that the user will check with a mail program or with another kind of server like POP3 or IMAP. But for our purposes, it means the mail is put somewhere that the user can find it.

A slightly more difficult case is when someone from, say, wd5gnr.com wants to send e-mail to someone at Hackaday (again, assuming that domain is handling its own e-mail which isn’t as common as it used to be). In this case, the user connects to their own SMTP server. It realizes that @hackaday.com is an alien e-mail address (maybe in more ways than one) and looks up the domain in the DNS directory. One of the records in DNS (the MX record) is the domain’s mail server address. The SMTP server on wd5gnr.com will then send the mail to hackaday.com which will actually do the delivery.

And Then…?

That’s all there is to it, right? Not exactly. These days, that DNS entry is more than likely pointing to some service like a Gmail server or some other hosting company that will actually accept the e-mail. Probably the sender is attaching to some vendor’s server, too. So maybe the sender attaches to Zoho’s SMTP server which then connects to the Gmail server to deliver the mail.

Even more complicated is when you get into private networks. If the wd5gnr network is behind a firewall, it may not be possible to directly contact the Hackaday server regardless of where it is. What then? An SMTP server can accept mail on one interface and then relay it to another server. This could go on for some time. For example, the wd5gnr server might relay everything to Gmail which then finds the main Hackaday server which further sends it to the internal mail server for the part of the world that particular user is in.

This may sound farfetched, but it happens all the time. For example, you might be on a ship at sea with intermittent connection to a satellite link. A server might collect e-mail and wait for a connection to the satellite. Then it sends the mail to a server via the satellite link that will take care of putting it out to the real network.

You can see why e-mail has been so ripe for exploitation. Early servers would allow anyone to connect and might even attempt to relay mail for anyone. Spam changed all that, of course, and now an open relay is a rarity. Some SMTP servers go as far as to reject mail that comes through known open servers. This is controversial in some circles as a form of censorship, but the majority of servers now will require you to authenticate and will only accept e-mail destined for certain domains without authentication. There’s also a rise of servers that check an SPF record to ensure that a server is known to handle mail for a particular domain. DKIM goes a step further and uses a cryptographic signature.

The Weight of History

So the fact that mail rarely, these days, leaves the network or flows through an open relay, the point is that it can. Why does that matter? Let’s go back to the ship at sea problem. The server for the ship may have to encode our e-mails into some wacky format or compress it. The satellite ground station might only send e-mail via UUCP which has further processing to do. So large e-mails are more than just a network bandwidth issue. The server may have to process and convert files.  SMTP, therefore, can require 7-bit ASCII which is great for text. But for anything fancier, the message (including attachments) gets encoded using something like base 64 encoding as specified by MIME (Multipurpose Internet Mail Extensions). This also means that your attachment is likely to get even larger after encoding.

Even a fully connected host may want to process e-mail messages. For example, it is common to check for spam or virus payloads. So even with infinite network bandwidth, there is still a processing overhead associated with large attachments.

Granted, processing power is not in short supply these days, either, but some mail servers may not be very state-of-the-art, while others may be getting hit by millions of e-mails, many of which are spam.

Ask Hackaday

How do you send large files?  Should e-mail take bigger files? Do you think e-mail is on the way out? Replaced by what? How could it be easier? There are services like WeTransfer and SendAnywhere, but honestly, we are more likely to just upload a file to one of our servers and send a link that way. Then there’s always the cloud storage like Dropbox, Drive, or iCloud.

We’ve come a long way since carrier pigeons. Then again, maybe your mail is going over that network, too. If you want to dig even deeper into SMTP, check out the video from Iowa State University, below.