Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures
draft-ietf-cbor-cddl-08

[Ballot comment]
Most of my comments have already been captured by others, save one:

Is there a specific reason the normative appendices are not part of the main body? I think a lot of RFC readers assume that appendices are optional to read. We should not surprise them without reason.

[Ballot discuss]
Thanks for all the work that went into creating this document. I have one point
that I think needs discussion, although it's entirely possible that I'm thinking
about this the wrong way.

§3.8:

Given that the list of control operators can be expanded in the future, it's
not clear what automated tools are supposed to do if they encounter controls
that they do not understand. I initially thought that it might be possible to
just ignore control operators (and their parameters) if they aren't understood,
as this would simply result in a more permissive validation of data against a
schema; but the ".and" control gives an example of a control operator where this
kind of elision would fail.

With the lack of any version indicators in CDDL, this seems like a straight-up
interoperability issue.

[Ballot comment]
I also have a handful of non-critical comments of varying importance.

Please expand "CBOR":
(1) In the title
(2) Upon first use in the document body

See https://www.rfc-editor.org/materials/abbrev.expansion.txt for details.

---------------------------------------------------------------------------

§1.2:

>  New terms are introduced in _cursive_.  CDDL text in the running text
>  is in "typewriter".

This is perplexing, as I know of no tool that will render the canonical form
of current RFCs in the way being described. Is the intention to hold this
document until the new RFC format is available?

---------------------------------------------------------------------------

§2:

>  The rest of this section introduces a number of basic concepts of
>  CDDL, and section Section 3 defines additional syntax.  Appendix C

Nit: "...and Section 3..."

---------------------------------------------------------------------------

§2.2.2:

>  delimited by a "//" (double slash).  Note that the "//" operators
>  binds much more weakly than the other CDDL operators, so each line

Nit: "...operator binds..." or "...operators bind..."

---------------------------------------------------------------------------

§3.1:

>  o  A name can consist of any of the characters from the set {'A',
>    ..., 'Z', 'a', ..., 'z', '0', ..., '9', '_', '-', '@', '.', '$'},

This looks like a formal syntax of some kind, but I don't know where it's
defined. Notably, since this document has just defined ".." to be an inclusive
range operator and "..." to be an exclusive range operator, defining the set of
allowed characters in this way seems to run the risk of interpreting, e.g., "Z"
to be disallowed.

I suggest either defining the set of allowed characters using a formally defined
and cited grammar (e.g., ABNF), or using prose.

---------------------------------------------------------------------------

§3.1:

>  o  outside strings, whitespace (spaces, newlines, and comments) is
>    used to separate syntactic elements for readability (and to
>    separate identifiers or numbers that follow each other); it is
>    otherwise completely optional.

This seems nominally at odds with the following text in §2.2.2.1, which points
to at least one other case where whitespace is mandatory:

>  When using a name as
>  the left hand side of a range operator, use spacing as in
>
>    min .. max
>
>  to separate off the range operator.

---------------------------------------------------------------------------

§3.1:

>    If prefixed as "h" or "b64", the string is
>    interpreted as a sequence of pairs of hex digits (base16) or a
>    base64(url) string, respectively

Please normatively cite RFC 4648, sections 8 and 5 respectively.

---------------------------------------------------------------------------

§3.8.1:

>  When applied to an unsigned integer, the ".size" control restricts
>  the range of that integer by giving a maximum number of bytes that
>  should be needed in a computer representation of that unsigned
>  integer.  In other words, "uint .size N" is equivalent to
>  "0...BYTES_N", where BYTES_N == 256**N.
>
>    audio_sample = uint .size 3 ; 24-bit, equivalent to 0..16777215
>
>              Figure 9: Control for integer size in bytes

While they're semantically the same, the example is oddly mismatched with the
preceding text. Consider instead:

      audio_sample = uint .size 3 ; 24-bit, equivalent to 0...16777216

---------------------------------------------------------------------------

Appendix B:

>          / "#" "6" ["." uint] "(" S type S ")" ; note no space!

No space where? I see two space productions in that rule (so it clearly
applies to some specific location), and there are several places where spaces
cannot appear.

>    type1 = type2 [S (rangeop / ctlop) S type2]

This rule doesn't seem to properly capture the ambiguity of "a...b". There is a
terribly complex way to address this by defining parallel "type2" and "type3"
rules that differ only in whether a dot is allowed to appear in their value, and
defining type1 as requiring a space after the type that can contain dots -- but
that is probably overkill. It's probably sufficient to reiterate the warning
about requiring a space under such circumstances as a comment on this rule.

>    HEXDIG = DIGIT / "A" / "B" / "C" / "D" / "E" / "F"

It is a common implementor mistake to forget that ABNF is, by default,
case-insensitive. It is probably worth adding a comment here as a reminder.
(The same applies to "0x", "0b", "e", and "p" above, but these seem less likely
to appear in arbitrary case.)

---------------------------------------------------------------------------

Appendix B:

>    SCHAR = %x20-21 / %x23-5B / %x5D-10FFFD / SESC
>    SESC = "\" %x20-10FFFD
...
>    PCHAR = %x20-10FFFD

These almost certainly should be:

      SCHAR = %x20-21 / %x23-5B / %x5D-7E / %x80-10FFFD / SESC
      SESC = "\" %x20-7E / %x80-10FFFD
...
      PCHAR = %x20-7E / %x80-10FFFD

(i.e., exclude the control character %x7F)

---------------------------------------------------------------------------

Appendix C:

>  (It is not an error to extend a rule name
>  that has not yet been defined; this makes the right hand side the
>  first entry in the choice being created.)

Is it an error to redefine a rule name that has already been defined?

[Ballot discuss]
Rich version of this review at:
https://mozphab-ietf.devsvcdev.mozaws.net/D4234


I am marking this document discussed because I have concerns about
whether this document can be interoperably implemented. I have noted a
number of points below.

DETAIL
S 3.5.2.
>      point.)
> 
>  3.5.2.  Tables
> 
>      A table can be specified by defining a map with entries where the
>      keytype is not single-valued, e.g.:

this is the first use of the term single-valued, so I don't know how
to interpret this.

More generally: it seems like:

```
  square-roots = {x => y}
                            x = int
                            y = float
```

Defines a map and yet

```
  square-roots = {x => y}
                            y = float
```

Defines a struct. Is that correct? If so, does that mean that I don't
know whether something is a map or a struct until I ahve parsed the
whole definition?



S 3.5.3.
>                        mynumber = int / float
> 
>  3.5.3.  Non-deterministic order
> 
>      While the way arrays are matched is fully determined by the Parsing
>      Expression Grammar (PEG) algorithm, matching is more complicated for

PEG is an informative reference, and this text seems to create a
normative dependency.


S 3.6.
>                            * tstr => any
>                          }
> 
>  3.6.  Tags
> 
>      A type can make use of a CBOR tag (major type 6) by using the

What happens if I define a type twice? Is that permitted?


S 3.7.
>                              buuid = #6.37(bstr)
> 
>      In the following example, usage of the tag 32 for URIs is optional:
> 
>                          my_uri = #6.32(tstr) / tstr
> 

I am basically unable to make sense of this section. Your previous
example of tags used #7.25 but here you are specifying everything as
using 6.

It seems like the semantics here are something to the effect of:

X = #6.Y(Z)

means: act as if this were a thing of type Z but it's tagged by Y.

Is that correct? But then is this about the wire encoding or the
interpretation or both? And in either case, what if what appears on
the wire has a different tag.




S 3.8.2.
>                          cwr: 15,
>                          ns: 0,
>                        ) / (4..7) ; data offset bits
> 
>                        rwxbits = uint .bits rwx
>                        rwx = &(r: 2, w: 1, x: 0)

What is the scope of the definition for r, w, and x? is it global.


S 3.9.
>              $$tcp-option //= (
>              sack-permitted: true
>              )
> 
>      Names that start with a single "$" are "type sockets", names with a
>      double "$$" are "group sockets".  It is not an error if there is no

what is the difference between these two?


S 7.3.
>      order of the rules given.  (It is not an error to extend a rule name
>      that has not yet been defined; this makes the right hand side the
>      first entry in the choice being created.)
> 
>      genericparm = "<" S id S *("," S id S ) ">"
>      genericarg = "<" S type1 S *("," S type1 S ) ">"

What is the meaning of 

[Ballot comment]
S 1.2.
>      capitals, as shown here.
> 
>  1.2.  Terminology
> 
>      New terms are introduced in _cursive_.  CDDL text in the running text
>      is in "typewriter".

I think you mean that these types of text will be indicated by being
bracketed by _ or ", but this isn't clear from this text. It's
especially not clear when you then quote byte and octet.


S 2.
>      There are a number of more or less atomic elements of a CBOR data
>      model, such as numbers, simple values (false, true, nil), text and
>      byte strings; CDDL does not focus on specifying their structure.
>      CDDL of course also allows adding a CBOR tag to a data item.
> 
>      The more important components of a data structure definition language

More important than what?


S 2.1.
>                                }
> 
>                    Figure 1: Using a group directly in a map
> 
>      The three entries of the group are written between the curly braces
>      that create the map: Here, "age", "name", and "employer" are the

This is kind of a confusing way to introduce this. Its seems like the
relevant point is that:

() makes a group
{()} or {} makes a map.

No?


S 2.1.
>                            }
> 
>                            dog = {
>                              identity,
>                              leash-length: float,
>                            }

So to be clear, this is a mixin, as in Go structs?


S 2.1.2.
>      only one big protocol data unit that has all definitions ad hoc where
>      needed.
> 
>  2.1.2.  Syntax
> 
>      The composition syntax intends to be concise and easy to read:

Syntax does not intend. perhasp you mean it is intended to be.


S 2.2.3.
>      tool when displaying integers that are taken from that choice).
> 
>  2.2.3.  Representation Types
> 
>      CDDL allows the specification of a data item type by referring to the
>      CBOR representation (major and minor numbers).  How this is used

RFC 7049 does not refer to "minor" numbers.


S 2.2.3.
>      in the prelude:
> 
>        my_breakfast = #6.55799(breakfast)  ; cbor-any is too general!
>        breakfast = cereal / porridge
>        cereal = #6.998(tstr)
>        porridge = #6.999([liquid, solid])

What is the parenthetical syntax here?


S 3.1.
>        according to the respective syntactic rules of that definition.
> 
>      o  A name can consist of any of the characters from the set {'A',
>        ..., 'Z', 'a', ..., 'z', '0', ..., '9', '_', '-', '@', '.', '$'},
>        starting with an alphabetic character (including '@', '_', '$')
>        and ending in one or a digit.

This seems to say that names end in a digit, but none of your name
examples do.


S 3.2.
>      An optional _occurrence_ indicator can be given in front of a group
>      entry.  It is either one of the characters '?' (optional), '*' (zero
>      or more), or '+' (one or more), or is of the form n*m, where n and m
>      are optional unsigned integers and n is the lower limit (default 0)
>      and m is the upper limit (default no limit) of occurrences.
> 

isn't bare * then just a degenerate case of n*m


S 3.5.1.
>      written with quoted strings in the member key positions.  More
>      generally, all the types defined can be used in a keytype position by
>      following them with a double arrow -- in particular, the double arrow
>      is necessary if a type is named by an identifier (which would be
>      interpreted as a string before a colon).  A string also is a (single-
>      valued) type, so another form for this example is:

I'm sorry, I'm not following this. Can you give an example of when you
have to use =>


S 3.10.
>        message = {type: t, value: v}
> 
>      When using a generic rule, the formal parameters are bound to the
>      actual arguments supplied (also using angle brackets), within the
>      scope of the generic rule (as if there were a rule of the form
>      parameter = argument).

This looks more like a macro than a generic.


S 7.3.
>      definition with different member names); RFC 7071 could be read to
>      forbid the repetition of ext-value ("A specific reputon-element MUST
>      NOT appear more than once" is ambiguous.)
> 
>      The CDDL tool (which hasn't quite been trained for polite
>      conversation) says:

It seems like it might be a good idea to clean up the examples here so
they are in fact polite.


S 7.3.
>        "Thumbnail": {"Width": 1111, "Height": 176, "Url": 32("scrog")},
>        "IDs": []}}
> 
>  Appendix B.  ABNF grammar
> 
>      The following is a formal definition of the CDDL syntax in Augmented

It's pretty odd to have the formal specification -- which clearly is
needed to understand and implement this document -- in an appendix.


S 7.3.
>        CRLF = %x0A / %x0D.0A
> 
>                              Figure 14: CDDL ABNF
> 
>      Note that this ABNF does not attempt to reflect the detailed rules of
>      what can be in a prefixed byte string.

I am trusting others to have read the ABNF.


S 7.3.
>      for a group expression (production "grpent"), with the intention that
>      the semantics does not change when the name is replaced by its
>      (parenthesized if needed) definition.  Note that whether the name
>      defined by a rule stands for a type or a group isn't always
>      determined by syntax alone: e.g., "a = b" can make "a" a type if "b"
>      is one, or a group if "b" is one.  More subtly, in "a = (b)", "a" may

It would be clearer to say 'if "b" is a type' and 'if "b" is a group'


S 7.3.
>      type = type1 S *("/" S type1 S)
> 
>      A type can be given as a choice between one or more types.  The
>      choice matches a data item if the data item matches any one of the
>      types given in the choice.  The choice uses Parsing Expression
>      Grammar [PEG] semantics: The first choice that matches wins.  (As a

Is this the only information I need?


S 7.3.
>      control operators.
> 
>      group = grpchoice S *("//" S grpchoice S)
> 
>      A group matches any sequence of key/value pairs that matches any of
>      the choices given (again using Parsing Expression Grammar semantics).

This also seems not fully defined.

[Ballot comment]
Thanks for updating the editor's copy pursuant to the secdir review!

As I was reading, I wondered about potential confusion between a numerical
value and the corresponding text string when used as a keytype, especially
for barewords.  The bareword ABNF requires a leading EALPHA, which should
force the right parsing, while the memberkey ABNF still allows literal
values to be used as keys.  I do wonder, though, if the 'id' ABNF's
limitations on textual names (i.e., strings that could be interpreted as
numbers are disallowed) should be mentioned in the main text as how
disambiguation is enforced in general.

It's a little weird to use PersonalData as an example, given the privacy
considerations inherent in storing personal data, but I guess this is not
really a flaw in the spec.

Section 1

Nit: bullet (G3) lacks grammatical parallelism with its sibling bullets;
something like "Be able to" would restore parity.

Section 2

  1.  Instead of defining all four types of composition in CDDL
      separately, or even defining one kind for arrays (vectors and
      records) and one kind for maps (tables and structs), there is
      only one kind of composition in CDDL: the _group_ (Section 2.1).

This perhaps reads a bit strongly, as we do go on to define syntactic sugar
for arrays and maps, even though they build on the shared group
abstraction.

Section 2.1

  Note that the (curly) braces signify the creation of a map; the
  groups themselves are neutral as to whether they will be used in a
  map or an array.
[...]
  Note that the lists inside the braces in the above definitions
  constitute (anonymous) groups, while "identity" is a named group.

I might add another sentence in one of these places foreshadowing the
behavior that groups are "macro-like" the sense that when used in the
description of another group, their contents are siblings of the elements
that are new in the other group, as opposed to being part of a nested
structure.

Section 3.1

  o  CDDL uses UTF-8 [RFC3629] for its encoding.

It's pretty rare for it to be sufficient to just say "UTF-8" in a technical
spec; what kind of internationalization review has been done?  Do we need
to specify anything about normalization or canonicalization?

Section 3.5.1

  The "struct" usage of maps is similar to the way JSON objects are
  used in many JSON applications.

  A map is defined in the same way as defining an array (see
  Section 3.4), except for using curly braces "{}" instead of square
  brackets "[]".

Taken together, these paragraphs read as if (1) a struct is a type of map,
and (2) a map uses curly brackets.  But the following example shows a struct
as enclosed within square brackets.  Where am I going wrong?

        GpsCoordinates = {
          longitude      : uint,            ; multiplied by 10^7
          latitude      : uint,            ; multiplied by 10^7
        }

It is perhaps irresponsible to include an example that does not specify the
units of the measurement (e.g., degrees or radians).

Section 3.8.6

  value from being sent over the wire.  This control is only meaningful
  when the control type is used in an optional context; otherwise there
  would be no way to express the default value.

Maybe s/express/utilize/?  That is, the ".default" control still expresses
what the default value would be, but that information would never be used.

Section 5

  o  Where the CDDL includes extension points, the impact of extensions
      on the security of the system needs to be carefully considered.

Would it make sense to also add guidance for judicious use of .within to
constrain extension points?

  Writers of CDDL specifications are strongly encouraged to value
  simplicity and transparency of the specification over its elegance.
  Keep it as simple as possible while still expressing the needed data
  model.

Perhaps "simplicity of [type] constructions", since some readers may equate
simplicity [of design] and elegance.

Section 6.1

I don't really understand why there's a need for distinctions based on the
presence of an internal dot, especially given that this document does not
define any such operators.  What would such a control operator look like?

Section 7.2

It seems that RFC 4648 might need to be a normative reference given that it
specifies how some byte string literals are interpreted in EDN.

Appendix B

On first glance I wonder if some of the S should be 1*WS to avoid parsing
ambiguities, but I did not think about it very hard.

  Note that this ABNF does not attempt to reflect the detailed rules of
  what can be in a prefixed byte string.

Before I made it this far, I was going to note that the "bytes" definition
seems to allow me to use a "h" or b64" prefix with "arbitrary" contents; it
seems that an alternate construction could embody the semantic restrictions
for such strings into the ABNF.  How bad would it be if a future update to
this document attempted to actually reflect the "detailed rules of what can
be in a prefixed byte string"?

Appendix D

I can't decide if most of the "#" entries need double-quotes around them to
parse properly as ABNF.  Is it best to think about this CBOR major/minor
notation as an extension to standard ABNF?

(Via drafts-lastcall@iana.org): IESG/Authors/WG Chairs:

The IANA Functions Operator has completed its review of draft-ietf-cbor-cddl-05. If any part of this review is inaccurate, please let us know.

The IANA Functions Operator has a question about one of the actions requested in the IANA Considerations section of this document.

The IANA Functions Operator understands that, upon approval of this document, there is a single action which we must complete.

IANA is to create a new registry called the CDDL Control Operators registry. The new registry is to be maintained via Specification Required as defined by RFC 8126.

IANA Question --> Where should this new registry be located? Should it be added to an existing registry page? If not, does it belong in an existing category at http://www.iana.org/protocols?

There are initial registrations in the new registry as follows:

+----------+---------------+
| name | Reference |
+----------+---------------+
| .size | [ RFC-to-be ] |
| .bits | [ RFC-to-be ] |
| .regexp | [ RFC-to-be ] |
| .cbor | [ RFC-to-be ] |
| .cborseq | [ RFC-to-be ] |
| .within | [ RFC-to-be ] |
| .and | [ RFC-to-be ] |
| .lt | [ RFC-to-be ] |
| .le | [ RFC-to-be ] |
| .gt | [ RFC-to-be ] |
| .ge | [ RFC-to-be ] |
| .eq | [ RFC-to-be ] |
| .ne | [ RFC-to-be ] |
| .default | [ RFC-to-be ] |
+----------+---------------+

IANA Question --> Should the entries in this new registry be alphabetized?

The IANA Functions Operator understands that this is the only action required to be completed upon approval of this document.

Note:  The actions requested in this document will not be completed until the document has been approved for publication as an RFC. This message is meant only to confirm the list of actions that will be performed.

Thank you,

Sabrina Tanamal
Senior IANA Services Specialist

The following Last Call announcement was sent out (ends 2018-10-04):

From: The IESG
To: IETF-Announce
CC: Barry Leiba , cbor@ietf.org, alexey.melnikov@isode.com, barryleiba@computer.org, cbor-chairs@ietf.org, draft-ietf-cbor-cddl@ietf.org
Reply-To: ietf@ietf.org
Sender:
Subject: Last Call:  (Concise data definition language (CDDL): a notational convention to express CBOR and JSON data structures) to Proposed Standard


The IESG has received a request from the Concise Binary Object Representation
Maintenance and Extensions WG (cbor) to consider the following document: -
'Concise data definition language (CDDL): a notational convention to
  express CBOR and JSON data structures'
  as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
ietf@ietf.org mailing lists by 2018-10-04. Exceptionally, comments may be
sent to iesg@ietf.org instead. In either case, please retain the beginning of
the Subject line to allow automated sorting.

Abstract


  This document proposes a notational convention to express CBOR data
  structures (RFC 7049).  Its main goal is to provide an easy and
  unambiguous way to express structures for protocol messages and data
  formats that use CBOR or JSON.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-cbor-cddl/

IESG discussion can be tracked via
https://datatracker.ietf.org/doc/draft-ietf-cbor-cddl/ballot/


No IPR declarations have been submitted directly on this I-D.

I still have a list of comments that I need to write down and discuss with the WG, but I think this can wait till after IETF LC.

1. Summary

This document proposes a standard notational convention to express CBOR
data structures.  Its main goal is to provide an easy and unambiguous way
to express structures for protocol messages and data formats that use
CBOR or JSON.  As it's proposing a standard, its target status is
Proposed Standard.

Barry Leiba is the document shepherd and Alexey Melnikov is the
responsible AD.

2. Review and Consensus

The CBOR working group has been working on the CDDL definition for about
a year, and has had productive, healthy discussion that's led to the
current document.  There is quite wide deployment of CBOR and a lot of
interesting in the definition language that's proposed here.  As is
typical, we had a core set of maybe half a dozen very active
participants, with quite a few others chiming in occasionally.  The
document shepherd thinks the interest and contribution has been robust.

There are no significant disagreements that remain, and there's solid
working group consensus on what's here now.  There have been
disagreements about how to represent particular things, but they have
been cleanly resolved and none are important to note here.  I'll call out
the latest one, as it's just come up: at the end of working group last
call, Jim Schaad raised an issue on the mailing list about an ambiguity
that affects automated parser generation.  After discussion on the
working group telechat, Carsten proposed text that clarifies that syntax
alone may not always be sufficient to understand the meaning of a name,
and that semantics of the name must be understood.

3. Intellectual Property

The authors are in full compliance with BCPs 78 and 79 and there is no
known IPR directly related to this document.

4. Other Points

There is a normative reference to an ISO document, ISO 6093.

The document creates a new IANA registry that uses "specification
required", and the instructions to IANA and to the designated expert are
clear and sufficient.  An expert will need to be designated for this
registry.  It would be appropriate to use the same expert(s) as for the
CBOR Simple Values and CBOR Tags registries (currently, Carsten).