Mail Filtering Tips

Table of Contents

1. Introduction
2. Getting started
3. Filtering spam
4. Organizing mail into folders
5. Forwarding mail
6. Setting up auto-reply/vacation
7. Creating filter rules on the fly
8. Putting it all together
9. For brave souls: Writing sieve scripts directly

1. Introduction

• Mail filtering is independent of any particular mail reader. Although the filtering tool is accessed through the webmail program (SquirrelMail) that comes with the new server, it does not depend on using webmail to read your mail. The filtering takes place at the time a message is received by the server, and the results of the filtering are the same, regardless of whether you read the mail with webmail, or with some another mail reader like pine or mutt. (For more about webmail, see the separate page Webmail Tips.)
• Mail filtering only affects incoming mail; there is no (easy) way to have the filtering action also apply to outgoing mail. This is an inherent limitation of all so-called "server-side" mail filtering programs, and there is no good way around it. While most filtering tasks (such as vacation replies, or spam filtering) only make sense for incoming mail, some (such as auto-saving messages into folders) would be useful to have applied to both incoming and outgoing mail. as well. Unfortunately, this is not possible with standard filtering tools.

2. Getting started

1. Go to the webmail address, https://webmail.math.uiuc.edu, log in with your AD (Active Directory) username and password. (See ??? for help with the AD authentication.)
2. Click on "Filters" at the top of the main window.
3. Click on the button "Add a New Rule" near the bottom of the screen. This will bring up the form in which you specify the rules for your filter. The following sections show how to do this for various common scenarios.

3. Filtering spam

1. Follow the steps in the "Getting started" section to get to the fill-out form for a new filter.
2. From the drop-down menus in the "Condition" section, pick "Header", "X-Spam-Score", and "is greater or equal to". In the text field specify a value for the spam score above which you want to filter away spam, say "50".
3. In the "Action" section, pick "Move messages into". This pops up a submenu; pick "a new folder, named", and choose a name for the folder, say "spam50", to indicate spam with a score of at least 50. This folder will be a subfolder of the main folder, INBOX.
4. In the "Additional Actions" section, check the "Stop" box. This prevents messages from being processed by other rules you might have set up. When dealing with likely spam, this is a good thing to do.
5. Click on "Add New Rule" at the bottom of the screen. This activates the filter.

• Using multiple spam folders: If you still get some spam, you might want to lower the spam cutoff, or set up a second rule, with different spam cutoff, and a different folder to hold the spam. I have three folders, spam50, spam10, and spam3, set up in the above manner to hold spam with cutoffs of 50, 10, and 3. I do examine these folders every once in a while, but it is very rare that a false positive makes it into one. Even the spam3 folder contains almost nothing but spam.
• Whitelisting addresses: An effective way to cut the risk of false positives is to whitelist addresses or address patterns. While there is no direct way to create a "whitelist", addresses can be effectively whitelisted by entering appropriate negated conditions in the "Conditions" section. For example, to ensure that email from "edu" addresses is not being filtered away, add a second condition, by picking "Header", "From", "does not contain", from the drop-down lists, and specifying ".edu" in the text field. Additional addresses can be whitelisted by adding similar lines. (Click on "More" to increase the number of lines displayed.)
• Note on spamassassin. Spamassassin, the old spamfighting standby, cannot be used on the new server, for the same reason that the procmail, forward, and vacation programs cannot be used there. However, using spam filtering based on the CITES Spam Score as described above eliminates nearly all spam and is a more than adequate replacement for spamassassin.

4. Organizing mail into folders

Note: All examples assume that you are already at the page where you can enter a new filter rule. The "Getting started" section above shows how to get to this page.

• Scenario 1: You correspond regularly via email with your colleague K.F. Gauss of the University of Goettingen and you want to file all messages from him into a folder named "gauss".

  Solution: First, check the "From:" header in recent messages from Dr. Gauss to determine the email address from which you receive his correspondence; say it is kfgauss@mathematik.uni-goettingen.de. Next, in the drop-down menus of the "Condition" section, pick "Address", "From", "is", and enter the string "kfgauss@mathematik.uni-goettingen.de" (without quotes) in the text field. In the "Action" section, select "Move messages into", then select "a new folder, named", and in the text field enter "gauss". Click on "Add Rule" to activate the recipe.

  Notes: A more conservative strategy would be to specify "contains" in place of "is" to allow substring matches, and to enter "kfgauss" instead of the full email address. This will match possible alternate addresses of your correspondent, such as kfgauss@uni-goettingen.de or kfgauss@ias.edu, but there is a small chance of matching addresses of other people as well.

• Scenario 2: You are on the Undergraduate Affairs Committee. An email alias, uac@math.uiuc.edu has been set up for the members of the committee, and you receive committee-related correspondence via this alias. You want to file this correspondence into a folder named "uac".

  Solution: In the drop-down menus of the "Condition" section, pick "Address", "To or Cc", "contains", and enter the alias, "uac@math.uiuc.edu" in the text field. In the "Action" section, select "Move messages into", then select "a new folder, named", and in the text field enter "uac". Click on "Add Rule" to activate the recipe.

  Notes: Make sure to use "contains" rather than "is", for otherwise messages having additional addresses in the "To:" field (e.g., messages sent to both uac@math.uiuc.edu and chair@math.uiuc.edu) would not be matched. Note that this only works on messages that use the uac alias. Messages addressed manually to some, or all, committee members won't get caught in this way. This is a good reason to set up email aliases in cases like this and to encourage the use of the aliases.

• Scenario 3: You are an Editor of the Antarctic Journal of Mathematics (AJM). An email alias, ajm@math.uiuc.edu, has been set up that forwards to you, and all correspondence regarding the Antarctic Journal of Mathematics is to be sent to this alias. You want to file this correspondence into a folder named "ajm".

  Solution: In the drop-down menus of the "Condition" section, pick "Address", "To or Cc", "contains", and enter the alias, "ajm@math.uiuc.edu" in the text field. In the "Action" section, select "Move messages into", then select "a new folder, named", and in the text field enter "ajm". Click on "Add Rule" to activate the recipe.

• Scenario 4: You are teaching a large section of Math 241, and you want to file all correspondence from students in this class to a folder named "math241".

  Solution: This is a tougher case, as there is no simple way to uniquely identify email from your calculus students. What I do is ask students to include an identifying string, say "Math 241", in the subject line of all their class-related correspondence. (To ensure that this request is followed, I advertise it widely on course materials and in class, and - this may be the most effective incentive - I make it clear that the tag in the subject line ensures that the message will get my immediate attention and won't get lost amongst junk messages.) Matching and filing away messages with this particular string is then straightforward: In the drop-down menus of the "Condition" section, pick "Header", "Subject", "contains", and enter the string "Math 241", or just "241". (The latter is more conservative and will match strings like "Math241", though it might also catch a few unrelated messages.) In the "Action" section, select "Move messages into", then select "a new folder, named", and in the text field enter "math241". Click on "Add Rule" to activate the recipe.

5. Forwarding mail

• Scenario 5: You are an Editor of the Antarctic Journal of Mathematics (AJM). An email alias, ajm@math.uiuc.edu, has been set up that forwards to you, and all correspondence regarding the AJM is to be sent to this alias. You want to file all such correspondence in a folder named "ajm", and in addition have a copy sent to the Editorial Manager of the AJM at editorialmanager@ajm.com.

  Solution: This requires two rules, one for each action. First, follow the steps in Scenario 3 above to create a rule that saves the AJM mail to a folder "ajm". Now create a second rule, with the same conditions (i.e., pick "Address", "To or Cc", "contains", and enter "ajm@math.uiuc.edu"), and in the "Action" section, select "Redirect to the following email address": In the text field that appears, enter the forwarding address, editorialmanager@ajm.com, and check the box "Keep a local copy as well". Click on "Add rule" to activate the rule.
  Note: You can specify multiple email addresses in the forwarding field. Just separate the addresses by commas, making sure that there are no blank spaces in the before or after the comma (i.e., the entire multi-address string cannot have embedded spaces).

• Scenario 6: You will be visiting the University of Goettingen for a few weeks; a temporary email account, guest1729@mathematik.uni-goettingen.de, has been set up for you. You want all your mail to be forwarded to this address during your stay at Goettingen.

  Solution: Since you want the forwarding action to apply to all incoming email, i.e., without any conditions that need to be met, you might be tempted to just leave the "Condition" section blank. However, the program will complain if you do that. Instead, do the following: First, in the "Condition" section, click on the "Less" button twice, until only a single condition line appears. Then click on the first drop-down list (the one that originally says "Header"), and choose the last of the options presented, "All". After clicking on "All", the other fields magically disappear, and the text "All messages" is displayed in the condition section, which is exactly what you want.
  The rest is straightforward: In the "Action" section, select "Redirect to the following email address": In the text field that appears, enter the forwarding address, guest1729@mathematik.uni-goettingen.de. You will likely want the forwarded mail to also remain on the local system, so check the box "Keep a local copy as well". When done, click on "Add rule".
  To discontinue the forwarding, click on "Filters" from the main webmail screen. This will show a list of filter rules you have set up; check the forwarding rule, and click on "delete" or "disable" to deactivate the rule. Choosing "disable" is probably the better option, since this will allow you to reactivate the rule at a later point, or to adapt it to a similar situation.

6. Setting up auto-reply/vacation

1. Follow the steps in the "Getting started" section to get to the fill-out form for a new filter. The form has three sections, "Condition", "Action", "Additional Action".
2. In the first section, "Condition", you must specify a condition that needs to be met in order for the vacation reply to be triggered. The program will complain if you leave this section completely blank, so you must fill something in. You could select "All Messages" as condition (pick "All" from the drop-down list under "Headers", then click on "Less" until only the single condition "All" is left). However, this is generally a bad idea as then vacation replies will be sent in response to spam email and to auto-generated commercial email (e.g., travel confirmations, newsletters, special offers), where vacation replies are not appropriate. Thus, you might want to restrict the vacation mechanism at least somewhat. Here are two examples;
   • From the drop-down menus pick "Header", "From", "does not contain", and enter ".com" in the text field. This prevents vacation replies from being triggered by ".com" addresses.
   • Pick "Header", "From", "contains" and enter ".edu" in the text field. This restricts the vacation replies to messages from ".edu" addresses. Since almost no spam originates from such addresses, this effectively prevents spam from triggering vacation replies.
3. In the next section, "Action", click on "Vacation". This pops up several pre-filled text fields:
   • The first is the email address that triggers the vacation response. Check that this address is the one under which you normally receive mail; if it doesn't, change it. (This might be necessary if your AD login is different from your MUSes login.)
   • The second field contains the text of the message; change this as needed. (Tip: Avoid language like the one suggested here ("automated reply"), as this might trigger spam filters at the other end.)
   • A third fields specifies the minimal interval (in days) between successive vacation replies to the same email address. You can set it to as low as 1 (day), but not 0. Having such an interval in place helps prevent mail loops.
4. You can ignore the final section, "Additional Actions". It allows one to specify optional additional actions, such as having the program send an email notification of any incoming email to an alternate address.
5. When you are done, click on "Add Rule". This brings you back to the main Filters screen, and you should see a description/summary of the vacation rule in your list of rules. The vacation program is now active.
6. To be sure the program is working as expected, have some friends send you email and check their mailboxes for any vacation replies. Note: When testing out the recipe, be aware that, after the initial vacation reply is sent out to a given address, there is a blackout interval before the next reply is sent out to that address. Thus, you cannot use the same address for more than one test.
7. To deactivate the vacation program, click on "Filters" from the main screen to display the filtering rules that you have defined. Check the box next to the vacation rule, and click on "Disable" to disable it. (There is also a "delete" option, but "disable" keeps the rule in the system and makes it easy to re-activate the vacation program with a simple mouse click.)

7. Creating filter rules on the fly

8. Putting it all together

You can edit a rule by clicking on the first icon in the options column. The second icon creates a copy of the rule, which can then be edited to form another rule. The triangular icons allow you to move rules up and down the chain.

If you have set up multiple rules, perhaps with overlapping conditions, it is important to understand how these rules are processed. Unfortunately, the available documentation on this is very spotty, and the "Help" button doesn't help in this case - there is no section in the Help manual discussing filtering. What follows are some general principles that I have been able to piece together from various sources and which I could confirm with trial and error.

• Filter rules are applied sequentially. Thus, an incoming message gets first tested against the condition in Rule 1, then against the condition of Rule 2, etc. At each step, if the condition of a rule is satisfied, the actions defined in this rule (e.g., filing the message into a folder, or forwarding a message) are applied.
• A message is processed by all filter rules unless it matches the condition of a rule in which the "STOP" box has been checked; in the latter case subsequent rules will not be applied. The "STOP" action is one of the options in the "Additional Actions" section. This option is mainly useful for a spam filtering rule: If you place the spam filtering rule at the top of the rule list, with the "STOP" box checked, then messages identified as likely spam will be filed into the spam folder right away, and not be processed by any other rules (e.g., filed into other folders). While in most cases this is the appropriate thing to do, there are situations where you would want to place a rule ahead of any spam filtering rules; for example, if you have a regular correspondent whose messages tend to trigger high spam scores, and you have a rule saving this person's correspondence onto a folder, that rule should be placed ahead of any spam filtering rule.)
• If a message matches none of the conditions of the rules, it is left in the INBOX. This is the expected behavior. The INBOX is the default mailbox, and this is where all messages would go if no rules have been set up.
• If a message matches the condition of at least one "move into folder" rule, it gets removed from the INBOX. This is no surprise either, and is usually what one wants. (The same goes for forwarding/vacation type rules unless the box "keep copy in INBOX" is checked.)
• If a message matches the conditions of several "move into folder" rules, it gets filed into all of the folders specified by these rules and removed from the INBOX, assuming none of the rules has the STOP box checked. This is a bit counterintuitive since one might expect that after applying one "move into folder" action the message is no longer available for further actions. However, it is a useful policy. For example, if you have a folder "gauss" for correspondence from K.F. Gauss, and another folder "conference" for correspondence regarding an upcoming number theory conference, you'll probably want to have a message from Gauss about the conference be filed in both folders. If you don't want this and instead want such a message to only show up in, say, the "gauss" folder, you can achieve this by checking the "STOP" option in the Gauss rule and placing the Gauss rule ahead of the "conference" rule.

9. For brave souls: Writing sieve scripts directly

Instead of having the script generated automatically via this frontend, one can instead create a sieve script with a text editor and upload it to the server. This is, in fact, what I have been doing, partly because it allowed me to access the full power of the sieve language (the filtering tool is more limited in its capabilities), and also in part because I am a bit of geek at heart, I prefer hand-coding things on the keyboard rather than pointing-and-clicking, and I like the challenges, and don't mind the frustrations, that this presents.

If you want to go down this road, be prepared for a very bumpy road, lots of frustration along the way, and no help available (sieve scripts are not officially supported by our system administration). You will be largely on your own (but see my "imap.notes" diary below). The main problem is that the "sieve" language is relatively new, very poorly documented (the only official piece of documentation is the technical description of the language in the form of a nearly inscrutable "RFC" document), and is very much dependent on the particular imap implementation; what may work at other places may not work here, and vice versa.

That said, if you want to give it a try, my own experiences might be helpful; I have recorded these in the file "imap.notes" under my public directory "/home/users/hildebr/pub" (this is intended for local users only). The same directory also contains a file "main.siv", the sieve script I am currently using. You will, however, be on your own, and I won't be able to help with any problems you run in. I simply don't know enough to be of much help; although I eventually managed to get things to work for me, it was a long and frustrating road, involving a lot of trial and error, and stumbling in the dark, and in many cases I didn't know why something worked one way, but not another way.