Sieve Email Filtering: Detecting Duplicate Deliveries
draft-ietf-appsawg-sieve-duplicate-09

[Ballot comment]
Background info:
While doing a PC migration, along with an email migration, I found myself in the situation where
- I had some duplicate emails in thunderbird
- Some of my emails were already manually classified into folders. So it was hard to discover those without redoing the manual classification.

This thunderbird add-on , http://removedupes.mozdev.org/, was very helpful to me.

Discussing with Stephan Bosch, I understand that Thunderbird add-on is used to remove duplicates from the user's mailbox, after delivery, while this Sieve extension is used to detect duplicates while they are being delivered. This is performed using a persistent duplicate tracking list with unique ID values (typically the Message-ID) of previous deliveries and not by searching the destination folder(s) for messages with a matching ID.

The issue with the SIEVE extension is that you have to enable this functionality in advance ... when you know that you're going to do something dangerous. Maybe that works ... but that reminds me of access-list management: "No, I will not do anything stupid! ... oops, I can't access my device any longer!". Or maybe you probably expect this SIEVE extension to be always on? I don't think it's mentioned.

Along the same line (and with my use case in mind):

  Implementations SHOULD let entries in the tracking list expire after
  a short period of time.

I was thinking: "short" = seconds, or minute. So not applicable to my email/PC migration use case.
I saw later:

  A default expiration time of around 7 days is usually
  appropriate.

Good.

I like your 4 examples, but the one I was more interested into was: can we send the duplicates into the same folder.
That would allow me to troubleshoot the root cause being the duplicates (subscribed to 2 mailing lists, synch issue, redirection, etc.)

Bottom line: this draft could be improved by discussing the use cases you have in mind. Certainly not a blocking factor though

[Ballot comment]
I wondered what'd happen if you used a DKIM-Signature
header with this, but I guess it should just work.
However, I don't recall if that header value is ok to
compare case-sensitive (e.g. "d=" might not be?).  I
don't think any of your examples show how to do the
tolower thing with a header field value, (or is that the
default for "set"?) so I guess you could add one that
does, but up to you, since I assume the intended
readership know this stuff.

Nothing to do with this draft in the end, but I think the
security/privacy discussion ended up raising a couple of
interesting issues that might be worth revisiting if/when
someone has energy: those were a) if we could make some
good privacy-friendly (but also admin friendly)
recommendations about logging mail and b) if we could
consider the privacy implications of sieve scripts or
other filters (I liked the "stupid boss" folder name one,
and am guilty of that for some of my own mail:-) and what
those might expose. For (a) I could imagine a useful
informational RFC, not sure for (b).

[Ballot comment]
Thanks for addressing my DISCUSS points.

As for protecting scripts/address books in transit, per Ned's email, I would be interested to know if there is anyone interested in taking that work up. Or if not, if we could at least note it somewhere as an outstanding vulnerability -- maybe at https://trac.tools.ietf.org/group/ppm-legacy-review/ (which I can't load right now because the tools site seems to be down, so not sure if that is a good place)?

[Ballot comment]
I have no objection to this extension; I think it will be helpful for some folks. Interestingly, I can't see ever using it myself: When I get duplicates from a mailing list, I *always* want the one that was sent from the list, with the List-* header fields on it, and *not* the one sent directly to me. Unfortunately, the one from the list is almost always going to arrive after the one sent directly to me, and the extension doesn't give me enough state to find the message(s) that came in earlier so I can decide which one to keep. But like I said, I can see others using this.

As to specific comments:

  As a side-effect, the "duplicate" test adds the message ID to an
  internal duplicate tracking list once the Sieve execution finishes
  successfully.

I think this may have been mentioned in response to someone else's comment, but perhaps this should be:

  As a side-effect, the "duplicate" test adds a unique identifier
  (again, by default the contents of the Message-ID header field) to an
  internal duplicate tracking list once the Sieve execution finishes
  successfully.

And then change other occurrences of "message ID" to "unique identifier" elsewhere in the document as appropriate.

[Ballot discuss]
The security considerations seem light, shouldn't there be a security consideration for the possibility of overwriting a message with another?  If an attacker wants to prevent someone from receiving a certain version of a message, couldn't this be used to achieve that goal?  I don't see anything to prevent this, with the exception of good security practices, trusted administrators, and monitoring for any tampering on sending servers.  I'm not looking for too much of a description, but a statement on this possible risk.

[Ballot comment]
A small point that is not at the level of a Discuss, but...
Are there not some implications on the integrity of the message ID on a message that should be stated. Clearly, if a message ID can be touched, the message can be made to appear to be a duplicate causing the sieve to throw it out.

[Ballot discuss]
o Section 3.3:
"A default expiration time of around 7 days is usually
  appropriate. ... If that limit is exceeded by the ":seconds" argument, the
  maximum value MUST silently be substituted"

The suggested default seems really long, especially for the example use case described in Section 1. On the other hand, it seems odd that the user's choice would be overriden by the preset maximum. This would make more sense to me if the default expiration were shorter and the user could override it with a longer :seconds argument if he wanted. What is the rationale for doing it the opposite way?

o Section 6:
It seems like this section is missing a discussion of the privacy considerations associated with enabling the duplicate test. In the case where the Sieve filter is implemented server side, making use of the duplicate test means that some record of the receipt of a particular message will be persisted (for as long as specified by the logic in Section 3.3) even if the user downloads his messages or deletes a received message on the server that matches a test string in the meantime. If I want to filter all messages with duplicate message IDs, for example, this puts a
requirement on the server to maintain a list of the message IDs of all messages I’ve received (for the amount of time specified by the timing parameters). So if a third party wanted to find out that list, it could go to the operator
of the server and ask for it. This introduces a privacy risk that is not discussed in the document and is exacerbated by the long default expiration time mentioned above. Tests that make use of the :header or :uniqueid arguments are also potentially problematic since the list of strings to match is kept on the server. It seems like these aspects should at least be noted in the document for implementers to consider.

o Section 6:
Sieve scripts that include duplicate tests contain potentially sensitive information (e.g., subject or body strings). So it seems like the scripts should be confidentiality protected in transit. I checked with Barry and he said that there is no RFC that specifies if/when scripts should be protected in transit, and I understand that this document is probably not the right place to specify required behavior there, but I'd like to discuss (more with the ADs than the authors) if there is some plan for specifying that behavior somewhere.

o Section 6:
I think it would make sense to require that the tracked message list be stored with an equivalent level of protection as the user's messages themselves. E.g., if message headers and bodies are stored encrypted, then the duplicate tracking list should be as well. And the duplicate tracking list should be subject to the same access controls as the mailbox (perhaps this is obvious but I'm not sure).

[Ballot comment]
o Section 3.1:
"When the ":uniqueid" argument is used, such normalization
  concerns are the responsibility of the user."

I don't quite get this. Do we expect users to, e.g., specify that uniqueid strings should only be compared after conversion to UTF-8? That would seem to rely on a level of technical sophistication that almost no users actually have.

o Section 3.2:
"This means that it does not matter whether values are
    obtained from the message ID header, from an arbitrary header
    specified using the ":header" argument or explicitly from the
    ":uniqueid" argument.

I had trouble understanding what "it does not matter" meant in this sentence. I would suggest:

"This means that the values in the duplicate list should be used for
duplicate testing regardless of whether they were
  obtained from the message ID header, from an arbitrary header
  specified using the ":header" argument or explicitly from the
  ":uniqueid" argument."

IESG/Authors/WG Chairs:

IANA has reviewed draft-ietf-appsawg-sieve-duplicate-06.  Authors should review the comments and/or questions below.  Please report any inaccuracies and respond to any questions as soon as possible.

IANA's reviewer has the following comments/questions:

IANA understands that, upon approval of this document, there is a single action which IANA must complete.

In the Sieve Extensions registry located at:

http://www.iana.org/assignments/sieve-extensions/

a new extension will be registered as follows:

Capability name: duplicate

Description: Adds test 'duplicate' that can be used to test whether a particular message is a duplicate; i.e., whether a copy of it was seen before by the delivery agent that is executing the Sieve script.

RFC number: [ RFC-to-be ]

Contact address: Sieve mailing list

IANA understands that this is the only action required upon approval of this document.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is only to confirm what actions will be performed.

The following Last Call announcement was sent out:

From: The IESG
To: IETF-Announce
CC:
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (Sieve Email Filtering: Detecting Duplicate Deliveries) to Proposed Standard


The IESG has received a request from the Applications Area Working Group
WG (appsawg) to consider the following document:
- 'Sieve Email Filtering: Detecting Duplicate Deliveries'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2014-06-12. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

Abstract


  This document defines a new test command "duplicate" for the "Sieve"
  email filtering language.  This test adds the ability to detect
  duplications.  The main application for this new test is handling
  duplicate deliveries commonly caused by mailing list subscriptions or
  redirected mail addresses.  The detection is normally performed by
  matching the message ID to an internal list of message IDs from
  previously delivered messages.  For more complex applications, the
  "duplicate" test can also use the content of a specific header or
  other parts of the message.




The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-appsawg-sieve-duplicate/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-appsawg-sieve-duplicate/ballot/


No IPR declarations have been submitted directly on this I-D.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet
    Standard, Informational, Experimental, or Historic)?
   
    Proposed standard.

    Why is this the proper type of RFC?
   
    This document specifies a Sieve extension. Per RFC 5228 section 6.2, such
    extensions must be published as either standards track or experimental.
    Experimental status seems inappropriate given the straightforward nature
    of this extension.
   
    Is this type of RFC indicated in the title page header?

    Yes.

(2) The IESG approval announcement includes a Document Announcement Write-Up.
    Please provide such a Document Announcement Write-Up. Recent examples can
    be found in the "Action" announcements for approved documents. The approval
    announcement contains the following sections:

    Technical Summary:

      This document defines a new test command "duplicate" for the "Sieve"
      email filtering language.  This test adds the ability to detect
      duplications.  The main application for this new test is handling
      duplicate deliveries commonly caused by mailing list subscriptions or
      redirected mail addresses.  The detection is normally performed by
      matching the message ID to an internal list of message IDs from
      previously delivered messages.  For more complex applications, the
      "duplicate" test can also use the content of a specific header or
      other parts of the message.

    Working Group Summary:

      The document was discussed by a few participants within the Applications
      Area Working Group and it has the consensus of the working group. The
      document was also discussed on the Sieve mailing list, sieve@ietf.org

    Document Quality:

      There are at least two implementations of this extension. Reviews have
      been done at various times by:

        Ned Freed
        Kristin Hubner
        Alexey Melnikov
        Tom Petch
        Aaron Stone

    Personnel:

      Document Shepard: Ned Freed
      Responsible Area Director: Barry Leiba < barryleiba@computer.org>

(3) Briefly describe the review of this document that was performed by the
    Document Shepherd. If this version of the document is not ready for
    publication, please explain why the document is being forwarded to the
    IESG.

      The Document Shepherd has reviewed every revision and has updated his own
      implementation of the extension accordingly.

(4) Does the document Shepherd have any concerns about the depth or breadth
    of the reviews that have been performed?

      No.

(5) Do portions of the document need review from a particular or from broader 
    perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML,
    or internationalization? If so, describe the review that took place.

      N/A

(6) Describe any specific concerns or issues that the Document Shepherd has
    with this document that the Responsible Area Director and/or the IESG
    should be aware of? For example, perhaps he or she is uncomfortable
    with certain parts of the document, or has concerns whether there really
    is a need for it. In any event, if the WG has discussed those issues and
    has indicated that it still wishes to advance the document, detail those
    concerns here.

      No concerns.

(7) Has each author confirmed that any and all appropriate IPR disclosures
    required for full conformance with the provisions of BCP 78 and BCP 79
    have already been filed. If not, explain why?

      The author is in compliance with BCPs 78 and 79.

(8) Has an IPR disclosure been filed that references this document? If so,
    summarize any WG discussion and conclusion regarding the IPR disclosures.

      N/A

(9) How solid is the WG consensus behind this document? Does it represent the
    strong concurrence of a few individuals, with others being silent, or
    does the WG as a whole understand and agree with it?

      The interest level isn't high, but there appears to be consensus.

(10) Has anyone threatened an appeal or otherwise indicated extreme
    discontent? If so, please summarise the areas of conflict in separate
    email messages to the Responsible Area Director. (It should be in a
    separate email because this questionnaire is publicly available.)

      N/A

(11) Identify any ID nits the Document Shepherd has found in this document.
    (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
    Checklist). Boilerplate checks are not enough; this check needs to be
    thorough.

      The only nit was a comment about coding tags. The only code in the
      document is example Sieve scripts.

(12) Describe how the document meets any required formal review criteria,
    such as the MIB Doctor, media type, and URI type reviews.

      N/A

(13) Have all references within this document been identified as either
    normative or informative?

      Yes.

(14) Are there normative references to documents that are not ready for
    advancement or are otherwise in an unclear state? If such normative
    references exist, what is the plan for their completion?

      No.

(15) Are there downward normative references references (see RFC 3967)? If
    so, list these downward references to support the Area Director in the
    Last Call procedure.

      N/A

(16) Will publication of this document change the status of any existing
    RFCs? Are those RFCs listed on the title page header, listed in the
    abstract, and discussed in the introduction? If the RFCs are not listed
    in the Abstract and Introduction, explain why, and point to the part
    of the document where the relationship of this document to the other
    RFCs is discussed. If this information is not in the document, explain
    why the WG considers it unnecessary.

      N/A

(17) Describe the Document Shepherd's review of the IANA considerations
    section, especially with regard to its consistency with the body
    of the document. Confirm that all protocol extensions that the document
    makes are associated with the appropriate reservations in IANA
    registries. Confirm that any referenced IANA registries have been
    clearly identified. Confirm that newly created IANA registries
    include a detailed specification of the initial contents for the
    registry, that allocations procedures for future registrations are
    defined, and a reasonable name for the new registry has been suggested
    (see RFC 5226).

      This specification defines a Sieve extension. The IANA Considerations
      contain the necessary registration template for the extension. The
      registration has been checked and looks ok.

(18) List any new IANA registries that require Expert Review for future
    allocations. Provide any public guidance that the IESG would find
    useful in selecting the IANA Experts for these new registries.

      N/A

(19) Describe reviews and automated checks performed by the Document Shepherd
    to validate sections of the document written in a formal language, such as
    XML code, BNF rules, MIB definitions, etc.

      All of the Sieve examples have been run through a Sieve parser and
      have been found to be valid.

(1) What type of RFC is being requested (BCP, Proposed Standard, Internet
    Standard, Informational, Experimental, or Historic)?
   
    Proposed standard.

    Why is this the proper type of RFC?
   
    This document specifies a Sieve extension. Per RFC 5228 section 6.2, such
    extensions must be published as either standards track or experimental.
    Experimental status seems inappropriate given the straightforward nature
    of this extension.
   
    Is this type of RFC indicated in the title page header?

    Yes.

(2) The IESG approval announcement includes a Document Announcement Write-Up.
    Please provide such a Document Announcement Write-Up. Recent examples can
    be found in the "Action" announcements for approved documents. The approval
    announcement contains the following sections:

    Technical Summary:

      This document defines a new test command "duplicate" for the "Sieve"
      email filtering language.  This test adds the ability to detect
      duplications.  The main application for this new test is handling
      duplicate deliveries commonly caused by mailing list subscriptions or
      redirected mail addresses.  The detection is normally performed by
      matching the message ID to an internal list of message IDs from
      previously delivered messages.  For more complex applications, the
      "duplicate" test can also use the content of a specific header or
      other parts of the message.

    Working Group Summary:

      The document was discussed by a few participants within the Applications
      Area Working Group and it has the consensus of the working group. The
      document was also discussed on the Sieve mailing list, sieve@ietf.org

    Document Quality:

      There are at least two implementations of this extension. Reviews have
      been done at various times by:

        Ned Freed
        Kristin Hubner
        Alexey Melnikov
        Tom Petch
        Aaron Stone

    Personnel:

      Document Shepard: Ned Freed
      Responsible Area Director: Barry Leiba < barryleiba@computer.org>

(3) Briefly describe the review of this document that was performed by the
    Document Shepherd. If this version of the document is not ready for
    publication, please explain why the document is being forwarded to the
    IESG.

      The Document Shepherd has reviewed every revision and has updated his own
      implementation of the extension accordingly.

(4) Does the document Shepherd have any concerns about the depth or breadth
    of the reviews that have been performed?

      No.

(5) Do portions of the document need review from a particular or from broader 
    perspective, e.g., security, operational complexity, AAA, DNS, DHCP, XML,
    or internationalization? If so, describe the review that took place.

      N/A

(6) Describe any specific concerns or issues that the Document Shepherd has
    with this document that the Responsible Area Director and/or the IESG
    should be aware of? For example, perhaps he or she is uncomfortable
    with certain parts of the document, or has concerns whether there really
    is a need for it. In any event, if the WG has discussed those issues and
    has indicated that it still wishes to advance the document, detail those
    concerns here.

      No concerns.

(7) Has each author confirmed that any and all appropriate IPR disclosures
    required for full conformance with the provisions of BCP 78 and BCP 79
    have already been filed. If not, explain why?

      The necessary BCP 78/79 language is present.

(8) Has an IPR disclosure been filed that references this document? If so,
    summarize any WG discussion and conclusion regarding the IPR disclosures.

      N/A

(9) How solid is the WG consensus behind this document? Does it represent the
    strong concurrence of a few individuals, with others being silent, or
    does the WG as a whole understand and agree with it?

      The interest level isn't high, but there appears to be consensus.

(10) Has anyone threatened an appeal or otherwise indicated extreme
    discontent? If so, please summarise the areas of conflict in separate
    email messages to the Responsible Area Director. (It should be in a
    separate email because this questionnaire is publicly available.)

      N/A

(11) Identify any ID nits the Document Shepherd has found in this document.
    (See http://www.ietf.org/tools/idnits/ and the Internet-Drafts
    Checklist). Boilerplate checks are not enough; this check needs to be
    thorough.

      The only nit was a comment about coding tags. The only code in the
      document is example Sieve scripts.

(12) Describe how the document meets any required formal review criteria,
    such as the MIB Doctor, media type, and URI type reviews.

      N/A

(13) Have all references within this document been identified as either
    normative or informative?

      Yes.

(14) Are there normative references to documents that are not ready for
    advancement or are otherwise in an unclear state? If such normative
    references exist, what is the plan for their completion?

      No.

(15) Are there downward normative references references (see RFC 3967)? If
    so, list these downward references to support the Area Director in the
    Last Call procedure.

      N/A

(16) Will publication of this document change the status of any existing
    RFCs? Are those RFCs listed on the title page header, listed in the
    abstract, and discussed in the introduction? If the RFCs are not listed
    in the Abstract and Introduction, explain why, and point to the part
    of the document where the relationship of this document to the other
    RFCs is discussed. If this information is not in the document, explain
    why the WG considers it unnecessary.

      N/A

(17) Describe the Document Shepherd's review of the IANA considerations
    section, especially with regard to its consistency with the body
    of the document. Confirm that all protocol extensions that the document
    makes are associated with the appropriate reservations in IANA
    registries. Confirm that any referenced IANA registries have been
    clearly identified. Confirm that newly created IANA registries
    include a detailed specification of the initial contents for the
    registry, that allocations procedures for future registrations are
    defined, and a reasonable name for the new registry has been suggested
    (see RFC 5226).

      This specification defines a Sieve extension. The IANA Considerations
      contain the necessary registration template for the extension. The
      registration has been checked and looks ok.

(18) List any new IANA registries that require Expert Review for future
    allocations. Provide any public guidance that the IESG would find
    useful in selecting the IANA Experts for these new registries.

      N/A

(19) Describe reviews and automated checks performed by the Document Shepherd
    to validate sections of the document written in a formal language, such as
    XML code, BNF rules, MIB definitions, etc.

      All of the Sieve examples have been run through a Sieve parser and
      have been found to be valid.