Server-Side Email Filtering Service

Basic Concepts

This system allows you to tell the IMSS mail server how to do certain things with your email for you, most notably:

• throw it away
• file it into folders
• resend it to another email address

when certain conditions apply. Since the mail server is doing these things, it won't slow down your own computer, and you can access the folders using the IMSS IMAP server or webmail system.

These instructions will take you through the basics of setting up filtering rules with this system. If you have questions about the system, please email help@caltech.edu.

Preparatory steps

To get to the setup system, first point your web browser at https://utils.its.caltech.edu. Enter your access.caltech username and password in the appropriate boxes and press the "Log in" button. On the resulting page, containing account information for your account, click on the "Change email settings" link.

Please pause at this point and examine the settings on the resulting page, and note the following things:

• If you have "Forward all mail to another address" set to "yes, don't keep a copy", your mail is being forwarded elsewhere and any filtering rules you set up for your account will have no effect. If it is set to "yes, do keep a copy", the filtering rules will take effect unless you are keeping your mail on the legacy server.
• If you have "Keep your mail on the legacy mail server" set to "yes", then any filtering rules you set up will have no effect. However, you will be able to use the procmail program on the IMSS Unix cluster to manipulate your mail there (email unix-admins@caltech.edu for more information). Appendix A's subsection on resending mail to other addresses explains how to forward some but not all of your messages to the legacy server.
• If you have "File messages marked as spam into a folder named spam on the mail server" set to "yes", that filing will take place before any filtering rules are applied. Appendix A's subsection on filing into folders explains how you can file messages into the spam folder via the filtering system after other rules have been applied.

Now, click on either "Email filtering" or "Show email filtering rules" to go to the page showing your list of rules.

The filtering rule list screen

If this is your first time using this system, you will be at a page saying you are editing email filter rules for your account, a line saying "you have not defined any rules yet", and a link to "insert new rule here". Click on "Insert new rule here". The text of the skeleton of a rule will appear, saying it is "Rule 1 (not named)", with no condition or action, and controls to activate, edit, and delete the rule.

The filtering process works like an assembly line. When you receive an email message the system starts at the beginning of the list of rules. For each rule in turn, it checks to see if the condition is satisfied by the message. If it isn't, it hands the message on to the next rule. If it is, though, it executes the action, and either hands the message to the next rule or stops processing altogether, depending on how you have set up the rule. If the system gets to the end of the rule list without stopping processing, it deposits the message in your inbox.

Whenever your create a new rule by clicking an "Insert new rule here" link, the rule text will be greyed out, indicating it is inactive. Rules are inactive until you click the associated "Activate rule" link. That way, you can set up one or more rules and not put them into effect until everything is precisely right. For the purposes of this tutorial, we'll set up a rule but not activate it, so you can see what the process is, without necessarily doing anything to your mail.

We're going to create a rule to file all mail larger than a megabyte in size into a folder named "Big Emails". Click on the "Edit rule" link for the new rule to go to the rule editing screen.

The rule editing screen

You should now be on a screen saying you are editing an unnamed email filtering rule for your account, with a blank box next to "Rule name" and links to add a new condition and a new action.

If you add a name to the rule you'll be able to see it on the page listing all of your rules, and this may help you manage all of your rules more easily. To see how it works, enter "Big email rule" (without the quotes) in the box and press "Save name". The page will refresh with "Name saved" in green at the top, and the page will say you are now editing the "Big email rule" rule. Now, click on "Show all email filtering rules". Your rule list will now say you have a "Big email rule" rule. Click on "Edit rule" again to get back to the rule editing screen.

Filter rules have two parts: conditions and actions. If a rule has no conditions, and is active, the system will always execute it when processing reaches the rule. If a rule has no actions, the rule will do nothing, and processing will skip to the next rule.

Click on "Add new condition here" now to go to the condition selection screen.

The condition selection screen

You should now be on a screen telling you you are adding a new condition to the "big email rule" rule for your account. You now have a list of choices of "simple conditions" and "container conditions".

A "simple condition" can actually be quite complicated, but it is so named because it involves a single very specific type of test. "Container conditions" allow you to make a condition that involves evaluating one or more other conditions and seeing if some or all of them are or are not true; in a sense, they are "containers" into which you put other conditions. You can freely put container conditions inside other container conditions, if you want.

The four types of simple condition correspond to asking the questions:

1. Does the message have (or not have) the following headers in it?
2. Are certain patterns found (or not found) in certain headers?
3. Are certain patterns found (or not found) in the message body?
4. Is (or is not) the message size above or below a certain value?

The four container conditions should be self-explanatory from their names. (If you know about Boolean logic, they correspond to the OR, AND, NOT AND, and NOT OR operations.)

We want to set up the condition "if the message size is greater than 1 megabyte". To do that, click on "check if the message size is or is not above or below some value". You'll now be on a page saying you are setting up a message size condition on your rule. Set the drop-downs and the type-in box so the condition is true if the message is greater than "1M", then click "Save". You will return to the rule editing screen, only now you will see the condition you have chosen.

Since you've added a condition, "Edit condition" and "Delete condition" links have appeared. "Delete condition" does what you think it does. "Edit condition" takes you back to the same kind of screen you used to set up the condition in the first place, so you can change it (for example, changing the size you're looking for, or changing "greater than" to "less than").

Now that we have the condition, we need to set up the action (saving to a mailbox) by clicking "Add new action here".

The action selection screen

You should now be on a screen saying you are adding a new action to your rule. Once again, you have a menu of choices, this time a list of things that can be done to your email. What each of them does should be fairly obvious, although Appendix A of this document explains each in more detail. However, there is one thing you should know right now. The "discard the message" action does not immediately and permanently discard messages with it is triggered. Instead, it moves the message into a folder on the IMSS IMAP server named filter discards. An automatic process on the server looks through everyone's filter discards folder every day and throws out any messages there that are more than a day old. That way, if you accidentally set up email filters that start throwing away all of your mail, you have a chance to get it back if you act within 24 hours.

We want to store our big mail in a special folder, so click on "store a copy of the message in another folder". You'll then see a page where you can choose the folder. It has a drop-down menu where you can choose one of your existing folders, if there are any, and a box where you can type in a folder name. Go ahead and type Big emails in the box and press "Save".

You're now back at the rule editing screen. Your action has been added to the action list, along with edit and delete links like with the condition. There's also a link to add a new action, since a rule can do more than one thing (saving in a folder and resending to another address simultaneously, for example). You can also see some text stating that filter processing will stop for a message if this rule is triggered and the actions are performed. Just to see what will happen, click on the link that allows processing to continue to other rules. You'll see that the action list now tells you processing will continue to other rules, and the explanatory text changes. Click on the link that makes processing stop after the actions are executed to get things back to the way they were.

Your rule now has the condition and action you want, so click on "Show all email filtering rules" to get back to your rule list.

Enabling rules and other manipulations

The rule list now shows your complete "Big email rule", with the correct condition and action. It is grayed out, though, because it is inactive. If you wanted to make it active, you'd click on "Activate rule"; it would change to a normal color and it would then be in effect for any incoming messages.

Before the end of this tutorial we will go through a few other things you can do on the main filter rules page. Click on either of the "Insert new rule here" links. You will then get a new, inactive, unnamed, and empty rule. You'll also see that next to each rule there are now links to "Move up" or "Move down". Click on one of them to see what happens. This is how you reorder rules. If you click on an "Insert new rule here" link again, you'll see that you'll also get "Move to start" and "Move to end" links that do what they say they do.

To get rid of these rules you probably don't want, go ahead click on the "Delete rule" links to get rid of the rules.

We've reached the end of this tutorial. What follows are two appendices, going over some of the specifics of the different conditions and actions, and explaining how you'd set up filtering rules to do some things people commonly want to do with their email. Remember, if you have any questions, please email help@caltech.edu.

Appendix A: Specifics of the conditions and actions

Simple Conditions

Check for the existence or nonexistence of certain headers in the message: This should be self-explanatory, but there is one important thing to note. With the basic form of this kind of rule you can only check if all of a given set of headers exist or if all of them do not exist. If you want to check for at least one of a given set of headers to exist or not exist, you must use an "If any of the following conditions are true" or "If at least one of the following conditions is not true" container condition, then put in a header existence condition for each individual header.

Check if certain headers match or do not match certain patterns: Header match conditions compare one or more headers against one or more patterns, and are true if any of the headers matches any of the patterns, or if none do. If you want to check for all of a given set of headers matching certain paterns, you must use an "If all of the following conditions are true" container conditions, then put in a header match condition for each individual match.

However, you can check for a pattern or patterns matching any header at all by using a * by itself in the header text box.

Please also note that the wildcard type of match does not automatically include * characters at the beginning and end of the pattern. This allows you to match for things occuring only at the beginning or end of a header (by only putting a * at the end or beginning of the pattern, respectively), but to look for a wildcard expression in the middle of a header you should put * characters at both the beginning and end.

Check if the message body matches or does not match certain patterns: Body match conditions compare the message body against one or more patterns, and are true if any of the patterns is matched. If you want to check for all of a given set of patterns matching, you must use an "If all of the following conditions are true" container conditions, then put in a body match condition for each individual match.

Please also note that the wildcard type of match does not automatically include * characters at the beginning and end of the pattern. This allows you to match for things occuring only at the beginning or end of the message body (by only putting a * at the end or beginning of the pattern, respectively), but to look for a wildcard expression in the middle of the body you should put * characters at both the beginning and end.

Check if the message size is or is not above or below some value: If you just put a number in the size box, it specifies a number of characters in the message (both header and body). You can put a "K" after the number to specify kilobytes (1024 bytes) or an "M" to specify megabytes (1024 kilobytes, or 1048576 bytes).

Container Conditions

When you set up a container condition, you'll be taken back to the rule editing page for your rule. The condition will show the container and have an "Add new condition here" link inside the container, which you can click to add a condition insider the container. You will also have "Delete condition and any sub-conditions" links instead of simple "delete condition" links.

An "empty" container condition does not affect the truth or falsehood of the condition. A condition composed entirely of empty containers is the same as an empty condition - that is, the actions in the rule will always be executed if the rule is active.

Actions

General: The system tries to guess whether or not filter processing should stop after the actions of a rule by looking at the last action in the list of actions. If the action is one of the "store" actions or the "resend" action, it assumes processing should stop. If the action is "add header" or "indicate receipt", it assumes processing should continue. If the action is "discard", it forces processing to stop and does not allow additional actions to be added to that rule.

Store a copy of the message in another folder: The folders are created on the IMSS IMAP server, where folder names have the following rules:

• You may use spaces, but not forward or back slashes.
• Periods are used as folder separators, like slashes with directories; a folder named work.meetings.planning is a folder named planning inside the folder meetings inside the folder work.
• When you actually look at your mail using the IMSS IMAP server or webmail system, all of the folders besides your inbox are found inside the special folder Mail. However, you should not refer to this Mail folder in the rule. The folder referred to in the previous bullet point will show up in your IMAP client as "Mail.work.meetings.planning".

If you store messages in a folder named spam, the messages there will be subject to automatic deletion as sepcified by the controls on the main email settings page of the IMSS Web Utilities system.

Send a copy of the message to another email address: You can separate the addresses by commas or spaces (or both). Please only include the actual address part (i.e. person@example.com) and not any full name parts. If you don't specify a domain for the address (the part after the @) it is assumed to be @caltech.edu.

If you want to forward messages to your account on the legacy mail server, you can do so by forwarding to the address username@legacy-spool-smtp.its.caltech.edu, where username is your own username on the IMSS Unix cluster. Messages forwarded there will be processed by your .forward file, which could potentially invoke other programs (like procmail).

Add a header to the message: Header names may ont contain spaces, colons, or non-printable-characters, or non-7-bit-ASCII characters.

Indicate receipt of email using some protocol: The purpose of this action is to notify you through some means that new mail has arrived in your account. This could involve contacting your desktop computer through some other protocol, or sending you some kind of instant message.

At the time of this writing (March 30, 2004) we have the following methods available. Please let us know if you'd like different ones and we'll see if we can set it up.

• finger - you can have our mail server finger some address on some other machine. The request will appear to come from the machine outgoing-mail.its.caltech.edu.
• ping - you can have our mail server ping another machine. The ICMP packet involved will again appear to come from outgoing-mail.its.caltech.edu.

Discard the message: Messages "discarded" by the system go into a special filter discards folder, and the system removes messages older than a day from that folder on a daily basis.

Appendix B: Some examples

Put messages to a certain mailing list in a folder

There are a number of ways you can do this; they all use header match conditions:

• If the mailing list puts a special string in the subject line (like the list name in brackets), you could match on the Subject header, with an "includes" style match, and the special string as the pattern.
• If mailing to the list always have the same address in the To or Cc header, you could match on the headers To and Cc (remember, if either matches, the condition is true), with the pattern being the email address sent to.
• If the mailing list is run under the IMSS Mailman system, you can look for a List-Id header equals the string , where listname is the name of the list in all lower case.

No matter what condition your choose, set up a file-into-folder action that files into the folder of your choice.

Forward all mail not sent by your assistant to your assistant

You would do this by setting up a rule with a header match condition that checked if the headers From or Resent-From did not contain the email address of your assistant, and an action that resent the mail to your assistant's address. (Thus, any mail that did have From or Resent-From with your assistant's email would not satisfy the condition and would be processed by other rules and/or put into your inbox.)

File spam using a different threshold level

The IMSS Spam tagger adds a header X-Spam-Level to all messages in scans. The header has a number of stars (*) equal to the message's "spam score" with fractions discarded. Our system adds the header X-Spam-Flag: YES if the message has a score of at least 5. If you want to file based on a different threshold, first turn off the option for automatically filing spam on the mail email configuration page. Then, set the first rule of your filtering rules to be one where:

• The condition is a header match on the header X-Spam-Level, the pattern is a number of stars equal to the level you want to file at, and the match type is "contained in"
• The action is one that puts the message in a folder named spam

Make IMSS tagged spam recognizable by Outlook

Our suggestion to users that fetch their mail with POP is to set their email client program to filter messages based on our X-Spam-Flag header. Unfortnuately, some versions of Microsoft Outlook do not allow filtering on custom headers. They do, however, allow filtering on other headers, such as X-Priority and X-MSMail-Priority. If you are in this situation you can set up a rule where:

• The condition is a header match on the header X-Spam-Flag checking if it is equal to YES
• There are two actions, both of which are of the "add header" variety, adding a header of X-Priority with value 5 and a header of X-MSMail-Priority with value Low.
• Processing continues on to further rules.

It's important that processing continue to further rules, so your mail can be dropped into your inbox (where the POP server can find it) after adding the headers.

Have your email start a program on the IMSS Unix cluster

In the past, IMAP and POP users have been able to have procmail on the IMSS Unix cluster invoke programs to do things based on their incoming mail, and still be able to access it with their preferred protocol. With our current setup this requires setting things up in two locations.

• With the Web Utilities system, create an email filtering rule whose condition selects the email messages that you want to cause a program invocation, and an action that forwards the email to your own address at legacy-spool-smtp.its.caltech.edu and then has processing continue to further rules.
• On the IMSS Unix cluster, set up a .forward file that invokes procmail in the usual way, specifying the programs to be invoked in your .procmailrc file.

Please note that:

• It's important that processing continue to further rules; otherwise, messages that are forwarded to the legacy server won't be accessible by POP or IMAP (or, indeed, on the legacy server, if you set up procmail incorrectly).
• The message on the legacy server is a completely different copy, so you can't use this technique to modify the contents of messages as they appear on the IMAP server.