Iowa State University IT

Using Filters to Sort Email (GAG 322)

July 2007

Overview

Developing filters to sort email is not a new feature in email packages, but the process has garnered more attention with the increase in unwanted email (often called "spam") coupled with the advent of a spam detection system at Iowa State. This handout presents specific instructions for using the special header generated by the spam detection system and general filtering concepts.

1 Spam Detection in Perlmx

Information Technology Services (ITS) has implemented a spam detection system known as Perlmx. By comparing an email to a set of rules, the Perlmx tagging system determines the probability that the mail is spam and denotes it accordingly with a specific tag. Then you can decide how you want to handle the mail. Many choose to set up filters and transfer the flagged mail to its own mail folder for reviewing later.

The Perlmx system adds a "X-Perlmx-Spam" header to each piece of email. A sample might look like this:

X-Perlmx-Spam: Gauge=XXXXXXXXIIIIIII, Probability=87%, Report="BIG_FONT"

The data in the "Gauge=" parameter can be used to set up a filter for directing suspected spam to a specific folder. Note that the spam detection software adds an "X" for every 10 percent probability that the piece of mail is spam. You can set the sensitivity based on the number of Xs you enter. For example, if you want to separate mail that has a 50 percent or greater chance of being spam, indicate "Gauge=XXXXX" when creating the filter.

Pattern matching in a filter is done character-by-character starting on the left. If you put in filters that specify very precise gauge values, you will need to put in additional rules to cover all the possibilities. For example, if you enter XXXII, it will match the strings for 32, 33, 34, and so on up to 39, but it won't match XXXX (40) or higher. We recommend keeping things simple and only matching on Xs.

2 Creating Mail Filters Using the "Gauge" Parameter

Following are specific instructions on how to set up filters in WebMail, Eudora (Windows and Macintosh), and Outlook (2000 and XP) based on the "Gauge=" parameter, described in section 1 above. (Outlook Express for Windows and Novell Groupwise can only filter on standard headers such as the subject line. These clients are not able to filter on the special PerlMX header containing the Gauge data.)

2.1 WebMail

WebMail has a SPAM filter built into it that filters on the special PerlMX header containing the Gauge data. You can either enable the built-in SPAM filter or set up your own filter.

To use the built-in SPAM filter:

  1. In a folder listing view, click the Filters button.
  2. Click the Create SPAM Filter button.
  3. Select the percent probability; the default is 50 percent.
  4. Under Actions, select one of the options to file the message into an existing or new folder, or delete the message.
  5. Verify the priority level of the filter.
  6. Click on the Save button to enable the filter.

To create your own filter:

  1. In a folder listing view, click the Filters button.
  2. Click the Create Filter button.
  3. Click the Advanced Options button.
  4. In the "Other header line" box, type in "X-Perlmx-Spam:".
  5. In the "Contains" box, type in "Gauge=XXXXX". Enter one X for each 10 percent probability that the message is spam. The example here is for 50 percent probability.
  6. Under Actions, select one of the options to file the message into an existing or new folder, or delete the message.
  7. Verify the priority level of the filter.
  8. Click on the Save button to save changes to the filter.

2.2 Eudora for Windows

  1. Click on Tools and Filters.
  2. Click on "New".
  3. Verify "Incoming" and "Manual" are checked.
  4. In the Header box, type in "X-Perlmx-Spam:".
  5. Select "contains" and type in "Gauge=XXXXX". Enter one X for each 10 percent probability that the message is spam. The above example is for 50 percent probability.
  6. In "Action", select "Transfer To" and select a new or existing folder for filing.
  7. If you do not want any more filters to be processed, select "Skip Rest" in the next action box. If you want other filters to still be processed, leave this as none.
  8. Click on the "X" in the upper righthand corner to close the window.
  9. Click on "Yes" to save changes to the filters.

2.3 Eudora for Macintosh

  1. Click on Window and select "Filters".
  2. Click on the "New" button.
  3. Verify that "Incoming" is checked.
  4. In the "Header" field, type "X-Perlmx-Spam:".
  5. Select "contains" and type in "Gauge=XXXXX". Enter one X for each 10 percent probability that the message is spam. The above example is for 50 percent probability.
  6. In "Actions" select "Transfer To" and then select a new or existing folder by clicking on the "Transfer" menu at the top of the Macintosh screen.
  7. If you do not want any more filters (after this filter) to be processed for messages that fit this rule, select "Skip Rest" in the next action box.
  8. Close the window and select "Save" to save changes to the filters.

2.4 Outlook 2000/2003/XP

  1. Outlook 2000/XP: Click Tools -> Rules Wizard, then click the "New..." button.
    Outlook 2003: Click Tools -> Rules and Alerts, then click the "New Rule..." button.
  2. (Outlook 2003/XP only) Select "Start from a blank rule."
  3. Highlight "Check messages when they arrive." Click Next.
  4. Check "with specific words in the message header".
  5. Click "specific words" in the Rule description box.
  6. Enter "Gauge=XXXXX" in the text box and click the Add button. Click OK. Click Next. (Enter one X for each 10 percent probability that the message is spam. The above example is for 50 percent probability.)
  7. Check "move it to the specified folder".
  8. Click "specified" in the Rule description box.
  9. Select or create a folder to move the messages to. Click OK. Click Next.
  10. Add any exceptions that you feel are needed. Click Next when finished.
  11. Name the rule and make sure the ?Turn on this rule? box is checked. Click Finish.
  12. Click OK.

3 Creating Mail Filters Using Standard Header Fields

When you start filtering your email for spam, you may want to create additional filters to direct specific email to folders. For example, newsletters and mail from discussion lists often contain content that fits enough rules to be tagged as spam. If you add filters to send newsletters and discussion group mail to their own folders before filtering for spam, it will be easier for you to find such mail where you expect it. Here are general instructions for creating those types of filters in WebMail, Eudora, and Outlook.

3.1 Creating Filters in WebMail

Creating a filter using standard headers:

  1. In a folder listing view, click the Filters button.
  2. Click the Create Filter button.
  3. Select a header field to filter on (i.e., To, CC, From, or Subject).
  4. In the "Contains" box, type the text to search for.
  5. Under Actions, select one of the options to file the message into an existing or new folder, or delete the message.
  6. Verify the priority level of the filter.
  7. Click on the Save button to save changes to the filter.

Creating a filter using special headers:

  1. In a folder listing view, click the Filters button.
  2. Click the Create Filter button.
  3. Click the Advanced Options button.
  4. In the "Other header line" box, type in the name of the header field if you wish to isolate the filter to a specific header field.
  5. In the "Contains" box, type the text to search for.
  6. Under Actions, select one of the options to file the message into an existing or new folder, or delete the message.
  7. Verify the priority level of the filter.
  8. Click on the Save button to save changes to the filter.

3.2 Creating Filters in Eudora

  1. Click on Tools and Filters.
  2. To create a new filter, click on "New" in the lower lefthand corner of the filters window.
  3. Select the text that you want to match in the message on the top half of the screen.
  • The "Incoming" and "Outgoing" checkboxes indicate whether the messages should be filtered when they come in or when they are sent out, respectively.
  • The "Header" field lets you determine which field is scanned to look for a match to some text. Most commonly used are the "Subject" and "From" fields.
  • The first drop-down box (with "contains" in it as a default) lets you determine how to search for the text. When "contains" is selected, the filter searches for a word or phrase anywhere in the selected field. Choosing "is" requires an exact match for the filter to be processed. The "starts with" and "ends with" options enable checking to see if a header begins or ends with a certain word or phrase.
  • Enter the text that you want to match in the empty text box by the first drop-down box. The combination of how to search and for what text is sometimes referred to as a "rule".
  • The second drop-down box (with "ignore" in it by default) lets you select a conjunction to filter on two rules. Selecting "ignore" sets the filter to ignore the second rule. Selecting "and" means that if the message matches both the first and second rules, the filter is processed. Choosing "or" means that if the text matches either rule, the filter is processed. Choosing "unless" means that if the text matches the first rule, the filter is processed unless the text also matches the second rule, in which case do not use the filter. (This lets you exclude certain variations of the first rule.)

4. The next step is to decide what action to take when a message does match your filtering rules.

  • "Transfer To" allows you to decide into which folder a message is placed. This is especially useful if you are on mailing lists and you want to group those messages together or if you are setting up a filter to put junk mail messages into a specific folder.
  • "Copy To" allows you to make a second copy of a message into another folder.
  • "Skip Rest" indicates that this should be the last filter that is processed on this message.
  • There are many other actions available; use the Eudora help function to learn more.

5. To save your customized filters, close the window and click "Yes" to save.

3.3 Creating Filters in Outlook 2000/2003/XP

The Rules Wizard in Outlook allows you to create filters and file messages in different folders as they arrive to make it easier to find messages.

  1. Outlook 2000/XP: Click Tools -> Rules Wizard, then click the "New..." button.
    Outlook 2003: Click Tools -> Rules and Alerts, then click the "New Rule..." button.
  2. Click the "New..." button.
  3. (Outlook XP only) Select "Start from a blank rule."
  4. Select the type of rule you wish to create. Usually, rules are used to check messages when they arrive. Click Next.
  5. Select the condition(s) you wish to check for. If you checked a condition that requires a value, click the underlined text in the rule description box in the lower half of the window. Enter the value(s) you wish to filter on. Click Next.
  6. Select the action(s) you wish to take with the message. If you checked an action that requires a value, click the underlined text in the rule description box in the lower half of the window. Enter the desired value. Click Next.
  7. Add any desired exceptions. This is optional. If you checked an exception that requires a value, click the underlined text in the rule description box in the lower half of the window. Enter the desired value. Click Next.
  8. Specify a name for the rule.
  9. Check the box labeled "Turn on this rule" (if it isn't already selected). Click Finish.

4 Determining the Order of Mail Filters

The order in which mail filters are processed determines where a piece of email finally resides. We recommend placing spam filters last, after the filters for other uses. For example, your filter list might begin with ones to put departmental mail or special correspondents in separate folders. Next might be filters for newsletters, mailing lists, and vendor information. Last in the list would be your spam filters. (If you decide to filter on more than one probability level of spam, filter the 90 percent Gauge level first, then 80 percent, then 70 percent, and so on.) That way, your wanted mail will be filtered first.

In Eudora and Outlook, you will also want to specify what action to take when a message meets the criteria for more than one filter. The following sections have instructions for doing that in Eudora and Outlook as well as how to change the order of filters.

4.1 Ordering Filters in Webmail

WebMail processes filters in the order of the priority that has been set for each filter. To change the priority level for a filter:

  1. In a folder listing view, click the Filters button.
  2. On the Filters listing, click the Edit button next to the filter for which you wish to change the priority level.
  3. Select the priority level for the filter from the pulldown box at the bottom of the page.
  4. Click on the Save button to save changes to the filter.

4.2 Ordering Filters in Eudora

Changing the order that filters are processed in Eudora can be done by simply dragging the filter that you want run first to the top of the list, put the second filter right below the first filter, and so on. However, Eudora will run all the filters on every message by default, so it is the last filter for which the message meets the criteria that will determine the action that is taken. Therefore, under default settings with the spam filter processed last, messages that met the criteria for an earlier filter and which you wanted to keep out of the spam folder will get put in the spam folder when the spam filter is reached. There are two ways to prevent this from happening:

Place the "Skip Rest" line at the end of filters that you don't want to be processed by other filters and place those filters at the beginning. The "Skip Rest" command takes a message that meets a filter's requirements and stops processing other filters. This is especially useful with a spam filter. If you place the spam filter at the bottom of your list of filters and have some messages that may be marked as spam but that you want to keep, set up a filter for those messages; transfer them to the desired folder (you can pick the inbox); and place the Skip Rest command so they do not get moved to your spam folder.

OR

Place your most important filters at the bottom of the list. This will configure Eudora to run that filter last. The last filter that is reached by the message will be processed.

4.3 Ordering Rules in Outlook 2000/2003/XP

If you have created multiple rules, you may want to set the priority for executing the rules.

  1. Outlook 2000/XP: Click Tools menu and select Rules Wizard...
    Outlook 2003: Click Tools menu and select Rules and Alerts...
  2. Highlight the rule on which you wish to set the priority.
  3. Click the "Move Up" or "Move Down" button to increase or decrease the priority of the rule. The rule at the top of the list will be executed first.
  4. When you have the rules ordered as you want them, click OK.

Some messages may fall under the conditions of more than one rule. After a specific condition is met, you may want to stop processing additional rules on the message. To manage processing in this manner, do the following:

For Outlook 2000/XP:

  1. Click the Tools menu and select Rules Wizard...
  2. Select the rule you wish to work with, and click the "Modify..." button.
  3. Click Next twice.
  4. Sroll down the list of actions. Check the box next the "Stop processing more rules." Click Finish.
  5. Click OK.

For Outlook 2003:

1. Click the Tools menu and select Rules and Alerts.
2. Select the rule you wish to work with and click the change rule button.
3. Select "Edit Rule Settings..."
4. Click Next.
5. Sroll down the list of actions. Check the box next to "Stop processing more rules." Click Finish.
6. Click OK.

5 For More Assistance

If you have questions about filters in email, contact the Solution Center in 195 Durham Center at 515-294-4000, solution@iastate.edu.