WinDeveloper Software
Innovation. The seed to our Solutions
Follow us on Twitter WinDeveloper on Facebook WinDeveloper YouTube Channel WinDeveloper Updates
HomeDownloadOrderSupportPartnersCompanyContact
IMF Tune v7.1 - Bringing Back the Exchange Connection Filter
IMF Tune v7.1 - Bringing Back the Exchange Connection Filter

Dissecting a Keyword Report - Part 2

In Verify IMF Tune with Keyword Reporting we laid the groundwork necessary to work with Keyword Reports. Today we jump straight in and start reading these reports.

 

What's in the Keyword Report?

Here is what a keyword report looks like for a single email.

Keyword Report for 1 email

The Standalone Keyword Report file will have many of these, one after another, each reporting on a different email. So we need to get used to distinguish between multiple reports.

The report starts with a header block showing the From, To, Subject etc. This marks the starting point of each report. Following that we find one or more filter report sections. Each time an email matches some filter, a section reporting on the match is added.

The header block is fairly self-explanatory. From, To, Subject and Date help us search for specific emails. You will most often want to make searches on the Subject field since spammers can easily spoof senders and hide the true recipients.

At the header we also find the Initial SCL, Final SCL and number of Matches. We defined the initial and final SCLs in the first part of this article. Matches is the count of filter reports following the header section. The above example has a match count of 2, thus we have 2 filter report sections.

The filter report sections describe specific filter matches. Here we have a 3 column table Header/Expression/Source followed by the type of Action the match triggered.

 

Header/Expression/Source

Each filter report section starts with a 3 column table. This information is organized in a table because of Advanced SCL Rules that can be made up of multiple conditions. Here is a match for an Advanced Rule made up of 3 conditions.

Header, Expression, Source table

The table column information includes:

Header - Identifies the filter type that flagged the match. Simple filters such as the Subject white/black lists use the email header name to identify themselves here. Other filters use more descriptive names to identify themselves more clearly. More on this in the section Naming the Filter.

Note how in an Advanced Rule we have multiple filters working together to check different bits of information.

Expression - The configuration relevant to the email match. Very often here we see a snippet taken from the IMF Tune configuration that shows how the filter is configured. For example if at the sender blacklist we configured *@adminstop.com, on blocking an email from bob@adminstop.com the Expression will show *@adminstop.com.

Source - The actual email information that triggered the filter. Taking further the example where we are blacklisting *@adminstop.com, Source will contain the actual email sender address as extracted from the email.

 

Filtering Action

Each Filter report section is concluded with an action statement. In all a filter can trigger one of three actions:

  • Operation set SCL to <new SCL>
  • Operation increment SCL by <amount>
  • Operation decrement SCL by <amount>

Filter action

Here we need to refer back to the first part of this article series. Each filter match contributes an action. Multiple matches lead to multiple actions which when computed together lead to the final SCL. What we see at the end of each Filter match section is the action specific to the match being described. The Final SCL resulting from these actions is shown at the report header.

 

Naming the Filter

Nine years ago, when IMF Tune Keyword Reporting was first released, most IMF Tune filters operated against specific email headers such as the From, To, Subject headers etc. Hence at the time the header name itself was used to identify the IMF Tune filters. This is where the Keyword Report column name 'Header' comes from.

Today we have more complex filters that don't quite fit this definition. Simple filters still make use of email header names to identify themselves. However more complex filters use other names for clarity. The following table lists how each filter identifies itself at the Keyword Report Header column

Keyword Report
Filter Name
Filter Description Configuration
IP Sender IP filter Whitelists | Accept IPs
Blacklists | Block IPs
SCL Rules | Simple SCL Rules | <Remote IP>
SCL Rules | External SCL Rules | <Remote IP>
SCL Rules | Advanced SCL Rules | Sending host matches IPs
Sender Sender address filter Whitelists | Accept Senders
Blacklists | Block Senders
SCL Rules | Simple SCL Rules | <Sender>
SCL Rules | External SCL Rules | <Sender>
SCL Rules | Advanced SCL Rules | Received from addresses or domains
Recipient Recipient address filter Whitelists | Accept Recipients
Blacklists | Block Recipients
SCL Rules | Simple SCL Rules | <Recipient>
SCL Rules | External SCL Rules | <Recipient>
SCL Rules | Advanced SCL Rules | Sent to recipient addresses or domains
Subject Subject keyword filter Whitelists | Accept Subjects
Blacklists | Block Subjects
SCL Rules | Simple SCL Rules | Subject
SCL Rules | External SCL Rules | Subject
SCL Rules | Advanced SCL Rules | Subject contains words
Body Body keyword filter Whitelists | Accept Bodies
Blacklists | Block Bodies
SCL Rules | Simple SCL Rules | <Body>
SCL Rules | External SCL Rules | <Body>
SCL Rules | Advanced SCL Rules | Body contains words
Body/Subject Body/Subject keyword filter Whitelists | Accept Subjects/Bodies
Blacklists | Block Subjects/Bodies
SCL Rules | Simple SCL Rules | <Body/Subject>
SCL Rules | External SCL Rules | <Body/Subject>
SCL Rules | Advanced SCL Rules | Body or Subject contains words
Attachment Name Attachment name/keyword filters Whitelists | Accept Attachments
Blacklists | Block Attachments
SCL Rules | Simple SCL Rules | <Attachment>
SCL Rules | External SCL Rules | <Attachment>
SCL Rules | Advanced SCL Rules | Attachment name contains words
SCL Rules | Advanced SCL Rules | Attachment name is exactly/matches filename
Email Character Set Language (character set) filter Blacklists | Block Foreign Spam
SCL Rules | Advanced SCL Rules | Email character set matches words
Total Recipients Recipient count filter SCL Rules | Advanced SCL Rules | Total number of recipients is within range
Header List Test for present/missing headers SCL Rules | Advanced SCL Rules | Standard header(s) are present and not empty
SCL Rules | Advanced SCL Rules | Custom header(s) are present and not empty
Has NO Body Text Empty body filter SCL Rules | Advanced SCL Rules | Email has NO body text
Has HTML Body HTML body filter SCL Rules | Advanced SCL Rules | Email contains HTML body
Has Attachments Test for email attachment presence SCL Rules | Advanced SCL Rules | Email contains attachments
Body Content Media Type Content media filter SCL Rules | Advanced SCL Rules | Content media type matches words
Email Size Email size filter SCL Rules | Advanced SCL Rules | Email size is within range
Received Time Email receive time SCL Rules | Advanced SCL Rules | Received time is within range
SCL Original SCL SCL Rules | Advanced SCL Rules | Spam Confidence Level (SCL) is within range
SMTP HELO/EHLO EHLO/HELO SMTP protocol command filter SCL Rules | Advanced SCL Rules | SMTP protocol HELO/EHLO command contains words
SMTP MAIL FROM MAIL FROM SMTP protocol command filter SCL Rules | Advanced SCL Rules | SMTP protocol MAIL FROM command contains words
SMTP RCPT TO RCPT TO SMTP protocol command filter SCL Rules | Advanced SCL Rules | SMTP protocol RCPT TO command contains words
header name Any standard/custom header keyword filter SCL Rules | Simple SCL Rules | header name
SCL Rules | External SCL Rules | header name
SCL Rules | Advanced SCL Rules | Standard header(s) contain words
SCL Rules | Advanced SCL Rules | Custom header(s) contain words
Sender (AWL) Sender Auto-Whitelist Auto-Whitelist Senders
DNS URI Block List DNS URI block lists DNS Lists | DNS Block URI Lists | *
DNS IP Block List DNS IP block lists DNS Lists | DNS Block IP Lists | *
DNS IP Allow List DNS IP allow lists DNS Lists | DNS Allow IP Lists | *

In this table we are mapping the filter name shown at the Keyword Report to the IMF Tune configuration. Let's go back to the initial example and apply this information.

Keyword Report Filter Match

The first filter match in this report shows:

Header: Sender
Expression: spammer@domain.com
Source: spammer@domain.com
Operation: set SCL to blacklisted

From this report we can immediately say that the Sender filter triggered a blacklisting operation because of the sender address spammer@domain.com. But where is the sender address configured? There are a number of places from where the Sender filter loads its configuration.

Looking up the previous table we get all these locations:

Whitelists | Accept Senders
Blacklists | Block Senders
SCL Rules | Simple SCL Rules | <Sender>
SCL Rules | External SCL Rules | <Sender>
SCL Rules | Advanced SCL Rules | Received from addresses or domains

Of course we can exclude Whitelists | Accept Senders since the filter action is Blacklist. The remaining 4 locations all provide the ability to blacklist a sender address, so to pinpoint the correct location we would need to check manually.

Mapping the second filter match report to the configuration is easier. The second filter report is showing 3 conditions that had to be satisfied for the email to be matched. The conditions are testing for an empty body, an attachment name and the email size. Only Advanced SCL Rules have the ability to combine multiple conditions, so the location must be:
SCL Rules | Advanced SCL Rules | *

 

End of Part 2

Today we looked closely at the Keyword Report information. We saw how the report ties together email processing to the filter configuration. It should already be evident how this helps us in reviewing filtered emails and go back to the configuration to apply any necessary adjustments. What remains is for us to look at some examples. This is the subject of the final part of this article series.

 

Copyright © 2004 - 2017 WinDeveloper Software Ltd. All rights reserved.