Debugging a Sieve detail that was confounding me

This Sieve behaviour had been bugging me, I just solved it and needed to share.

Sieve (Wikipedia) is a mail filtering tool. It's brutally effective and if your mail provider supports it (as my work and personal emails do), I recommend it highly to sort your email for you (and more).

However ... a specific filter seemed to be misbehaving. The intended behaviour was:

• For emails from Gitlab, if the project path contains "clientname", place the message in my "Projects/Clientname" folder^[1].
• Unless I'm explicitly mentioned, mark the message as "Seen"^[2].

Here's what I used to have:

  if header :contains "X-GitLab-Project-Path" [
    "clientname/"
  ] {
    fileinto "INBOX.Projects.Clientname";
    if not body :contains "@chrisburgess" {
      setflag "\\Seen";
    }
    stop;
  }

My gut feel was that this worked fine - but I could see evidence (emails marked unread which didn't @mention me) it wasn't working now. A slightly hard bug to be confident about as there's a lot of emails, and it felt like it worked most of the time. I had that Heisenbug feeling. This configuration was still filing Gitlab messages into the correct folder ... but it seemed like it was inconsistently marking items as Seen.

I wondered if Gitlab's emails contained the string @chrisburgess in the headers - not how I expected body :contains to work, but I did start to wonder. I checked the email source and found no matches.

Perhaps I was considering the @ symbol wrong for matching purposes?

Then I found, via a StackOverflow question, a link to Fastmail's Sieve tester. That tool appears to be available whether or not you're logged into Fastmail.

Using a web-based (or CLI^[3]) tester, I can much more quickly iterate over testing changes than I could if each test required an email delivery (and in this case, a Gitlab issue update). Similar to the difference between a full setup and execution to validate, and testing in a REPL.

With a hosted web-based tool, I do need to consider confidentiality (eg not submitting client names in my Sieve configuration OR the sample email source) and implementation (Fastmail's Sieve has a couple of plugins missing relative to my work email).

With those things in mind I was able to reduce my full Sieve configuration to a version that reproduced the incorrect behaviour - the web interface showing the results of my filter:

"filing message into 'INBOX.Projects.ClientName'".

I was then able to test a few things out and quickly identify a fix - a trivial one! Faster test iteration lets us try things out more freely.

Fastmail now reported the desired outcome:

"filing message into 'INBOX.Projects.ClientName' with flags '\seen'".

Here's the solution:

  if header :contains "X-GitLab-Project-Path" [
    "clientname/"
  ] {
    if not body :contains "@chrisburgess" {
      setflag "\\Seen";
    }
    fileinto "INBOX.Projects.Clientname";
    stop;
  }

What changed? I moved the setflag above the fileinto, establishing that ordering of setflag and fileinto is significant; setflag action applies to subsequent fileinto actions.

Now the messages will all be filed in the per-client folder (searchable), but only marked unread if they explicitly @mention me.

Reviewing the rest of my Sieve configuration for the same fix, I saw that other similar filters had setflag before fileinto - which probably contributed to my uneasy sense that this rule configuration had worked in the past, but inconsistently today; I hadn't twigged to that detail being out of order for this one customer.

I don't see this ordering detail mentioned in the published version of Sieve's RFC5232, but once I knew what to look for, I saw it was mentioned in an earlier draft, section 3.1.