A few years ago, I found this great article that answered all my questions regarding how to send emails (in this case various CSV spreadsheets) from the AS400 and iSeries machines. I was just discussing it so why not stuff it in a blog entry:
Send E-Mail Messages with SMTP
Posted August 17th, 2006
In a previous newsletter (July 27, 2006, article ID 52911), I discussed the format of e-mail messages. Formatting the messages is only one piece of the puzzle, however. You also need to be able to send the e-mail across a network.
The iSeries comes with tools for sending e-mail messages, but they have their limitations. In this article, I explain how e-mail moves across the Internet, and I present my own utility for sending e-mail — a utility that overcomes the limitations that I’ve faced with the i5/OS tools.
What’s Wrong with the i5/OS Tools?
IBM provides three tools for e-mail access from programs. They are the Send Distribution (SNDDST) command, the Send MIME Mail (QtmmSendMail) API, and the JavaMail API.
Limitations of SNDDST:
- It can send only a relatively short message with relatively simple formatting.
- It can attach only one document, and that document has to be located in the ancient /QDLS file system.
- It wasn’t originally designed for Internet e-mail, and its Internet capabilities aren’t always intuitive.
- It requires the i5/OS Simple Mail Transfer Protocol (SMTP) server to be installed, configured, and running.
Limitations of the QtmmSendMail API:
- You have to manually build in the IFS a stream file that contains the e-mail message. Even for simple messages, you have to muck with the IFS APIs.
- It’s not always intuitive to troubleshoot problems with QtmmSendMail().
- The format of the parameters can be awkward.
- The API hands off the stream file containing the message body to a background process and deletes it for you. This means that you have to create a unique name for each stream file, and you never know exactly when it’s done loading it into the system.
- It requires the i5/OS SMTP server to be installed, configured, and running.
JavaMail doesn’t have the same limitations as SNDDST or QtmmSendMail(). However:
- It’s written in Java, so it can be difficult for an RPG programmer to understand and use.
- The Java Virtual Machine (JVM) takes a long time to start on my machine (about 2 minutes). My users don’t like waiting for this.
Many free e-mail utilities that solve these problems are available, but because e-mail is so important to me and my business, and because I enjoy this type of programming, I decided to write my own utility.
How E-Mail Is Delivered
E-mail is sent over networks and the Internet using a protocol known as Simple Mail Transfer Protocol (SMTP). Sending e-mail to an SMTP server involves four steps:
- Identify yourself to the server. Or, “say hello” to the server.
- Tell the server who the message is from.
- Tell the server who the recipient(s) are.
- Send the message itself.
As its name implies, SMTP is a simple protocol to use. In fact, if you want to, you can use an ordinary Telnet tool to manually type SMTP commands to an SMTP server. This is useful when you want to test that a server is running properly, and it’s a great way to learn how SMTP works.
To try it out, you need to know the name of an SMTP server that you’re allowed to access. Your company might have a Sendmail or Exchange server running on the LAN, or you might even be running the SMTP server on i5/OS. If nothing else, your ISP should have provided you with one when you signed up. SMTP servers run on port 25, so you need to tell the Telnet software to connect to port 25. To do that, type the following command from a command prompt on your PC (I don’t recommend using the i5/OS Telnet program. The one supplied with Windows, Linux, *BSD, or even Mac should work much better than the i5/OS one for this test):
telnet smtp.example.com 25
Replace “smtp.example.com” in the preceding command line with the name of the computer running the SMTP service. After you type that command, the Telnet session starts. The following shows a sample SMTP session. The lines in red are the data that I typed to the server, and the parts in blue are responses from the server back to me.
220 SMTP Service ready. HELO mypc.example.com 250 Hello mypc.example.com, I'm AS400.EXAMPLE.COM. MAIL FROM: 250 OK. RCPT TO: 250 OK. DATA 354 Enter mail body. End mail with a '.' in column 1 on a line by itself. From: Scott Klement 250 OK. QUIT 221 AS400.EXAMPLE.COM Service closing transmission channel.
The exact text of the messages that the server sends you varies depending on the server software. The important part isn’t the text itself — it’s the number that precedes it. 220 means that a mail server is ready and able to receive mail. 250 means that the preceding command completed successfully. 354 (and other numbers in the 300 range) mean that the preceding command was understood but that more data is needed to complete the command. Numbers in the 500 range are errors.
The HELO (which stands for “Hello”) command is used to tell the server who you are. In the preceding example, I told the server that I was named mypc.example.com. The server usually responds with its own name.
The MAIL FROM command is used to tell the server who an e-mail message has been sent from. This should only be the e-mail address itself, not the person’s name. This address isn’t part of the e-mail message itself, but is part of the e-mail “envelope.” In other words, it’s the information actually used to route and deliver the mail, rather than the human-readable stuff that gets printed on the screen. One of the more controversial aspects of the SMTP protocol is that it doesn’t require the envelope to match what’s printed on the user’s screen!
The RCPT TO command specifies one e-mail recipient. Like the MAIL FROM command, it’s part of the envelope of the message and isn’t printed on the recipient’s screen. Again, this should only contain the e-mail address itself, not the human-readable name. You can send as many RCPT commands as needed to specify all the recipients of a message.
The DATA command designates the start of the file containing the e-mail message itself. When the DATA command has been accepted by the server, everything that follows is considered part of the e-mail message body. This message body should be formatted the way I described in the previous article (article ID 52911).
When you’ve typed (or copied/pasted) the entire e-mail message, enter a period in column one of a line by itself to signal the SMTP server that your message is complete. If all is well, the server sends back a 250 response to tell you that the message has been accepted.
After the message has been accepted, you can send more messages by specifying the MAIL FROM, RCPT TO, and DATA commands again, or you can quit by sending the QUIT command.
Wait a minute! If a message ends when a period is sent on a line by itself, wouldn’t there be a chance for a message to get messed up? What if I had a line with a period by itself in my message body?! To accommodate that possibility, any line of text in the e-mail message that starts with a period should be preceded by one extra period. The server will strip the first period from each line. For example:
DATA 354 Enter mail body. End mail with a '.' in column 1 on a line by itself. 250 OK.
When the server reads each line of the preceding message, it looks for a period in the first column, and if one is there, it removes it. If the resulting line is empty, it’s considered the end of the message. If not, the line is added to the e-mail message, and the next line is processed. As long as an extra period is added to any line that starts with a period, the message is interpreted successfully.
SMTP Server Roles
The preceding section describes how the SMTP protocol sends messages from one place to another. But wait. Isn’t SMTP used only for outgoing mail? Why do I have to connect to an SMTP server on my LAN to send mail? Can’t I just send it to the person who needs it?
If you think about it, outgoing and incoming mail are opposite sides of the same coin. Each time a message is transferred over the network, it’s outgoing from one server and incoming to another. In reality, SMTP is used for mail sent to an SMTP server. Ultimately, a destination server receives the message and stores it in a mailbox to wait for the user to pick it up.
The other protocols that you might have heard about, such as Post Office Protocol (POP) and Internet Mail Access Protocol (IMAP), are used only to transfer mail from that last server’s mailbox to your PC’s hard drive.
Technically, e-mail software falls into three different roles: Mail User Agents (MUAs), Mail Transfer Agents (MTAs), and Mail Delivery Agents (MDAs). All these roles use SMTP for at least part of the job.
An MUA is the software that sits in front of the user. Outlook, Thunderbird, Eudora, Pegasus, and SNDDST are all examples of MUA software. APIs such as QtmmSendMail() and JavaMail are tools that can help programmers develop their own MUAs. MUAs use SMTP to send a message from the user to an MTA.
Why doesn’t an MUA try to deliver the mail directly to the end user? Because e-mail is supposed to be reliable. If the network goes down, or the destination user’s computer is down or turned off, you still want the e-mail message to go through. If the first attempt fails, the MUA should retry at regular intervals. Because an MUA is often run from a desktop machine and is frequently opened and closed by the user, it’s ill equipped to retry a message at intervals.
So the MUA needs a server that it trusts to hand the message to. Then, that server can queue the mail up and retry it if necessary. That’s what an MTA is for. An MTA receives the message and “relays or requeues” that message. In other words, it either sends it to the destination or queues it up and retries later.
The i5/OS SMTP server is an MTA. If your company has an Exchange, Sendmail, or Domino server on the LAN that e-mail is supposed to be sent through, that’s also an MTA. Often, companies configure their MTAs to scan messages for viruses and spam in addition to relaying and requeuing the messages.
Eventually, the MTA succeeds in sending the e-mail message to the recipient’s MTA. The recipient’s MTA examines the e-mail address of the recipient and says to itself, “Yep, that message is for me!” and passes the message off to the MDA. The MDA’s job is to save it to the appropriate user’s mailbox. MDAs sometimes also do virus or spam checking. Ultimately, they put the message in the mailbox, and at that point, the SMTP process is finished.
MUA software designed to run on the server itself might directly access the mailbox that the MDA saved the mail in so that it can display the message to the server. Remote MUA software uses either the POP or IMAP protocols to access the mailbox.
Scott’s SMTP Utility
So far in this article, I’ve provided a lot of text and very little code. It’s time to rectify that problem!
In previous newsletters, I explained that TCP/IP programming is done using a set of APIs called sockets. In fact, in one article, I even demonstrated how to use sockets to send an e-mail message.
I’ve written a utility aimed at the SMTP protocol. To use it, you call a routine that creates an SMTP handle (which is really a temporary area of memory used to keep track of the state of the SMTP session), and you pass it to subprocedures that carry out the same SMTP commands typed in the Telnet session.
For example, consider the following sample code:
H DFTACTGRP(*NO) OPTION(*SRCSTMT) BNDDIR('SMTP') /copy smtp_h D CRLF C X'0d25' D FAIL C const(*OFF) D wholemsg s 32767A varying D hsmtp s like(SMTP_HANDLE) D count s 10I 0 inz /free hsmtp = SMTP_new('smtp.example.com'); if ( SMTP_connect(hsmtp) = FAIL ); // handle error endif; if ( SMTP_from(hsmtp: 'firstname.lastname@example.org') = FAIL ); // handle error endif; if ( SMTP_recip(hsmtp: 'email@example.com') = FAIL ); // handle error endif; wholemsg = 'From: ME <firstname.lastname@example.org>' + CRLF + 'To: Faithful Reader <email@example.com>' + CRLF + 'Subject: Version 2.0' + CRLF + CRLF + 'Hello there. Nice day for frolicking in the' + CRLF + 'beautiful Hawaiian sun.' + CRLF + '(Actually, they have the same sun we do.)' + CRLF + CRLF + 'Sincerely,' + CRLF + ' Richard M. Nixon'; if ( SMTP_data_var( hsmtp : wholemsg ) = FAIL ); // handle error endif; SMTP_quit(hsmtp); SMTP_free(hsmtp); *inlr = *on; return; /end-free
This code connects to an SMTP server at smtp.example.com. (The HELO command is also sent automatically by the SMTP_connect() subprocedure.) It then specifies the envelope fields (the MAIL FROM and RCPT TO fields) followed by the message data.
The SMTP_data_var() subprocedure provides the message data as a variable. That way, I don’t have to write my e-mail data to a stream file before I send it. This saves me the hassle of working with the IFS APIs when I want to send a simple message. There are other ways to provide that SMTP data, but more about that later.
The SMTP_quit() subprocedure sends the QUIT command, and the SMTP_free() subprocedure releases the temporary memory that SMTP_new() reserved for its internal work variables.
Because I can specify the MTA server on the call to SMTP_new(), I no longer need the i5/OS SMTP server running on my machine. I can specify the name of an Exchange or Domino or Sendmail server provided by my company or by my ISP, and it uses that for the MTA. If you do not pass any parameters to SMTP_new(), it tries to use the SMTP server running on the local System i5.
To make troubleshooting easy, the SMTP utility logs all the SMTP protocol commands to the job log. After the sample program runs, the same commands that you saw during the Telnet session will be visible in your job log. That way, if something does go wrong, the job log shows exactly what happened.
While writing this utility, I thought about all the existing programs that I have (e.g., the ones from the July 27 article) that use the QtmmSendMail() API. I don’t want to rewrite those from scratch, do I? So I added another subprocedure to my SMTP service program that acts the way that QtmmSendMail() acts. For example, consider the following code that calls QtmmSendMail():
QtmmSendMail( FileName : %len(FileName) : fromAddr : %len(fromAddr) : recip : %elem(recip) : NullError );
My utility contains a subprocedure called SMTPSendMail() that accepts the exact same parameters that QtmmSendMail() does. To convert the preceding code to use my new utility, all I have to do is change the name of the API it calls, as follows:
SMTPSendMail( FileName : %len(FileName) : fromAddr : %len(fromAddr) : recip : %elem(recip) : NullError );
And then, of course, I have to re-create the program and bind it to my SMTP service program so that it can use the new subprocedure. Because these changes are relatively minor, upgrading my programs is easy.
I also added a new subprocedure that works in conjunction with SMTPSendMail() to tell it to use a different MTA from the one on the local machine. To upgrade my QtmmSendMail() program to switch to a different server, all I have to do is change it to SMTPSendMail() and call the SMTP_SetServer() beforehand. For example:
SMTP_SetServer('smtp.example.com'); SMTPSendMail( FileName : %len(FileName) : fromAddr : %len(fromAddr) : recip : %elem(recip) : NullError );
The SMTPSendMail() API provides parameter-level compatibility with QtmmSendMail(), making it an easy way to upgrade existing programs. Because QtmmSendMail() isn’t the easiest API to use, I wouldn’t use this interface to write new programs. Instead, I’d use the SMTP_new(), SMTP_from(), and SMTP_recip() APIs that I demonstrated earlier.
The SMTP_data_var() method of providing the message body makes it easy to assemble the message body in a variable and pass it to the SMTP tool. But as I was writing that, I thought to myself, “What if I want to send something really big? Something that wouldn’t be practical to fit in a variable? It’d be nice if I could still use a stream file, but without having to use the SMTPSendMail() interface.”
To provide the ability to use a stream file, I added a subprocedure called SMTP_data_stmf(). I can call that instead of SMTP_data_var() when I want to send a message body in a stream file.
H DFTACTGRP(*NO) OPTION(*SRCSTMT) BNDDIR('SMTP') /copy smtp_h D CRLF C X'0d25' D FAIL C const(*OFF) D hsmtp s like(SMTP_HANDLE) D count s 10I 0 inz /free hsmtp = SMTP_new('smtp.example.com'); if ( SMTP_connect(hsmtp) = FAIL ); // handle error endif; if ( SMTP_from(hsmtp: 'firstname.lastname@example.org') = FAIL ); // handle error endif; if ( SMTP_recip(hsmtp: 'email@example.com') = FAIL ); // handle error endif; if ( SMTP_data_stmf( hsmtp : '/tmp/mesg.txt' ) = FAIL ); // handle error endif; SMTP_free(hsmtp); *inlr = *on; return; /end-free
In the preceding example, the DATA portion of the SMTP session is taken from the stream file instead of a variable. But that’s not the only way to specify it, either. Sometimes the code is cleaner if you use RPG’s compile-time data arrays instead of fussing with a variable or a stream file. So I added yet another way of retrieving the data:
H DFTACTGRP(*NO) BNDDIR('SMTP') /copy smtp_h D msg s 78A dim(12) CTDATA D hsmtp s like(SMTP_HANDLE) /free msg(4) = %trimr(msg(4)) + ' ' + SMTP_getTime(); hsmtp = SMTP_new(); SMTP_connect(hsmtp); SMTP_from (hsmtp: 'firstname.lastname@example.org'); SMTP_recip(hsmtp: 'email@example.com'); SMTP_data_ary(hsmtp: msg: %elem(msg)); SMTP_free(hsmtp); *inlr = *on; /end-free **