Data At Rest Encryption: DARE Container
draft-hallambaker-dare-container-00
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft whose latest revision state is "Expired".
|
|
|---|---|---|---|
| Author | Phillip Hallam-Baker | ||
| Last updated | 2018-05-24 | ||
| RFC stream | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-hallambaker-dare-container-00
Network Working Group P. Hallam-Baker
Internet-Draft Comodo Group Inc.
Intended status: Informational May 24, 2018
Expires: November 25, 2018
Data At Rest Encryption: DARE Container
draft-hallambaker-dare-container-00
Abstract
This document describes DARE Container, a message and file syntax
that allows a sequence of data frames to be represented with
cryptographic integrity, signature and encryption enhancements to be
constructed in an append only format. The format supports data
integrity checks using digest chains and Merkle trees. The simplest
supports efficient append only write operations and efficient read
operations in either the forward or reverse direction. Support for
efficient random-access reads may be provided through the use of
binary trees or index records appended to the end of the file.
This document is also available online at
http://mathmesh.com/Documents/draft-hallambaker-dare-container.html
[1] .
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on November 25, 2018.
Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved.
Hallam-Baker Expires November 25, 2018 [Page 1]
Internet-Draft DARE Data Container May 2018
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1. Related Specifications . . . . . . . . . . . . . . . . . 5
2.2. Requirements Language . . . . . . . . . . . . . . . . . . 5
3. Container Format . . . . . . . . . . . . . . . . . . . . . . 5
3.1. Container Profile . . . . . . . . . . . . . . . . . . . . 6
3.1.1. Index Profiles . . . . . . . . . . . . . . . . . . . 7
3.1.2. Content Profiles . . . . . . . . . . . . . . . . . . 7
3.2. Payload Encryption . . . . . . . . . . . . . . . . . . . 8
4. Index Mechanisms . . . . . . . . . . . . . . . . . . . . . . 8
4.1. Tree . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.2. Position Index . . . . . . . . . . . . . . . . . . . . . 9
4.3. Metadata Index . . . . . . . . . . . . . . . . . . . . . 9
5. Integrity Mechanisms . . . . . . . . . . . . . . . . . . . . 9
5.1. Digest Chain calculation . . . . . . . . . . . . . . . . 9
5.2. Binary Merkle tree calculation . . . . . . . . . . . . . 10
6. Reference . . . . . . . . . . . . . . . . . . . . . . . . . . 10
6.1. Container Headers . . . . . . . . . . . . . . . . . . . . 10
6.1.1. Structure: ContainerHeaderFirst . . . . . . . . . . . 10
6.1.2. Structure: ContainerHeader . . . . . . . . . . . . . 11
6.2. Content Metadata Structure . . . . . . . . . . . . . . . 12
6.2.1. Structure: ContentMeta . . . . . . . . . . . . . . . 12
6.3. Index Structures . . . . . . . . . . . . . . . . . . . . 12
6.3.1. Structure: ContainerIndex . . . . . . . . . . . . . . 12
6.3.2. Structure: IndexPosition . . . . . . . . . . . . . . 13
6.3.3. Structure: KeyValue . . . . . . . . . . . . . . . . . 13
6.3.4. Structure: IndexMeta . . . . . . . . . . . . . . . . 13
6.4. Signature . . . . . . . . . . . . . . . . . . . . . . . . 13
7. Further Work . . . . . . . . . . . . . . . . . . . . . . . . 14
7.1. Fast open with random access . . . . . . . . . . . . . . 14
7.2. Partitioning of very large data sets across hosts . . . . 15
7.3. Filtering and redaction . . . . . . . . . . . . . . . . . 15
7.4. Encryption of large data blocks . . . . . . . . . . . . . 15
7.5. Concurrent Writes . . . . . . . . . . . . . . . . . . . . 16
8. Security Considerations . . . . . . . . . . . . . . . . . . . 16
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16
Hallam-Baker Expires November 25, 2018 [Page 2]
Internet-Draft DARE Data Container May 2018
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 16
11. Appendix A: Examples and Test Vectors . . . . . . . . . . . . 16
11.1. Simple container . . . . . . . . . . . . . . . . . . . . 16
11.2. Payload and chain digests . . . . . . . . . . . . . . . 18
11.3. Merkle Tree . . . . . . . . . . . . . . . . . . . . . . 19
11.4. Signed container . . . . . . . . . . . . . . . . . . . . 21
11.5. Encrypted container . . . . . . . . . . . . . . . . . . 21
12. Appendix B . . . . . . . . . . . . . . . . . . . . . . . . . 24
13. References . . . . . . . . . . . . . . . . . . . . . . . . . 25
13.1. Normative References . . . . . . . . . . . . . . . . . . 25
13.2. Informative References . . . . . . . . . . . . . . . . . 25
13.3. URIs . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 26
1. Introduction
DARE Container is a message and file syntax that allows a sequence of
data frames to be represented with cryptographic integrity,
signature, and encryption enhancements to be constructed in an append
only format. DARE Container was developed in response to needs that
arose out of the design of the Mathematical Mesh
[draft-hallambaker-mesh-architecture] . It is built on the binary
encodings of JSON data objects, JSON-B and JSON-C
[draft-hallambaker-jsonbcd] and the DARE Message format
[draft-hallambaker-dare-message] .
The high level requirements supported include:
o Recording Mesh transactions in persistent storage.
o Synchronizing transaction logs between hosts.
o Representing message archives (aka mail spool)
o Signing and encrypting single data items.
The features supported by DARE Container include:
o The format is append only, thus providing for rapid write
operations and enabling the use of technologies that provide
atomic transactions.
o All length and index values support the use of integers of at
least 64 bits.
o Data frames may be of variable length.
Hallam-Baker Expires November 25, 2018 [Page 3]
Internet-Draft DARE Data Container May 2018
o Data frames may be read in either direction. This allows the last
n frames to be read as efficiently as the first n frames.
o Appending a data frame to an existing file is efficient taking no
more than log2 (n) operations.
o A binary tree index MAY be constructed on an incremental basis,
allowing random access to the nth record in the file in log2 (n)
operations.
o An index MAY be appended to an existing container to allow random
access to the nth record in the file in log2 (n) operations
o Permits the use of modern data encodings (e.g. JSON [RFC7159] ).
o Supports digital signature and public key operations on the
payloads of individual data frames.
o Data frame content (i.e. payload data) may be overwritten without
invalidating the integrity of any other frame. This allows
content to be expunged in exigent circumstances (court order,
regulatory, confidentiality breach, etc.) without compromising the
integrity of the rest of the data in the container.
Many file proprietary formats are in use that support some or all of
these capabilities but only a handful have public, let alone open,
standards. DARE Container is designed to provide a superset of the
capabilities of existing message and file syntaxes, including:
o Cryptographic Message Syntax [RFC5652] defines a syntax used to
digitally sign, digest, authenticate, or encrypt arbitrary message
content.
o The.ZIP File Format specification [ZIPFILE] developed by Phil
Katz.
o The BitCoin Block chain [BLOCKCHAIN] .
o JSON Web Encryption and JSON Web Signature
Attempting to make use of these specifications in a layered fashion
would require at least three separate encoders and introduce
unnecessary complexity.
Every data format represents a compromise between different concerns,
in particular:
Data Storage The space required to record data in the encoding.
Hallam-Baker Expires November 25, 2018 [Page 4]
Internet-Draft DARE Data Container May 2018
Memory Overhead The additional volatile storage (RAM) required to
maintain indexes etc. to support efficient retrieval operations.
Number of Operations The number of operations required to retrieve
data from or append data to an existing encoded sequence.
Number of Disk Seek Operations Optimizing the response time of
magnetic storage media to random access read requests has
traditionally been one of the central concerns of database design.
The DARE Container format is designed to the assumption that this
will cease to be a concern as solid state media replaces magnetic.
While the cost of storage of all types has declined rapidly over the
past decades, so has the amount of data to be stored. DARE Container
represents a pragmatic balance of these considerations for current
technology. In particular, since payload volumes are likely to be
very large, memory and operational efficiency are considered higher
priorities than data volume.
2. Definitions
2.1. Related Specifications
DARE Container makes use of the following related standards and
specifications.
Encoding Content frame headers are encoded using JavaScript Object
Notation (JSON) [RFC7159] , JSON-B or JSON-C
[draft-hallambaker-jsonbcd] .
Cryptography The encryption and signature schemes used are based on
JSON Web Signature [RFC7515] and JSON Web Encryption [RFC7516] .
2.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119] .
3. Container Format
A DARE Container consists of a sequence of JBCD Frames containing up
to three ordered JBCD records as follows:
Header Record [required] Metadata of any type including container
metadata, payload metadata and DARE message headers.
Hallam-Baker Expires November 25, 2018 [Page 5]
Internet-Draft DARE Data Container May 2018
Payload Record [optional] The frame data. Payload records are
either complete or incremental. A complete payload contains a
complete unit of whatever data is being written to the log. An
incremental payload contains a part of a unit that has been split
across multiple records.
Guard Record [optional] An opaque data record with a known value
written to the end of a frame to allow a writer to avoid
corruption of the container framing data by detecting an
incomplete append state.
A DARE frame consists of a forward length indicator, the framed data
and a reverse length indicator. The reverse length indicator is
written out backwards to allow the frame to be read in the reverse
direction:
[[This figure is not viewable in this format. The figure is
available at http://mathmesh.com/Documents/draft-hallambaker-dare-
container.html [2].]]
JBCD Bidirectional Frame
When first reading an existing file, an application will typically
read the first frame and the last frame (if the container has more
than one frame). This allows the reader to quickly determine the
format(s) used by the container, the number of frames in the
container and the location of any index frames (if present).
The container format is designed to support creation of write-once
and append-only file formats. Each frame SHOULD be written as an
atomic operation.
The first frame in a container and the first record in a frame have
special roles that are described in this document.
o The first frame in a container describes the container format
options and defaults. These include the range of encoding options
for frame metadata supported and the container profiles to which
the container conforms.
o The first record in a frame MUST NOT contain payload data
3.1. Container Profile
A key objective of the DARE Container format is that the simplest
possible reader be capable of reading any container file albeit with
possibly reduced performance.
Hallam-Baker Expires November 25, 2018 [Page 6]
Internet-Draft DARE Data Container May 2018
A Container MAY conform to one or more profiles. Conforming to a
profile typically requires a writer to provide additional information
when writing a file but does not require a reader to interpret it
unless use of a feature (e.g. authentication) that depends on the
additional information is required.
3.1.1. Index Profiles
The following profiles are currently defined:
Tree Frame headers contain IndexPosition entries that specify the
start position of previous frames. This enables efficient random
access to any frame in the file.
Digest Frame headers contain PayloadDigest entries that specify the
digest value of the corresponding payload data in that frame.
Chain Frame headers contain ChainDigest entries that link each frame
to the preceding frame.
Merkle Frame headers contain TreeDigestPartial and TreeDigestFinal
entries linking all the frames in the container in a binary Merkle
Tree.
The use of Chain and Merkle Trees for integrity checks is described
below.
The use of Tree and Index frames is described below.
3.1.2. Content Profiles
The following profiles are currently defined:
Singleton A container with exactly one content frame. A container
declared as a singleton frame cannot have additional content
frames appended (but metadata frames may be)
Multi A container whose payload data is limited to content frames.
A container declared as a multi container may contain 0, 1 or more
content frames.
Archive A multi-container whose payload data is limited to content
frames whose last frame contains a metadata index for the content
frames in the container.
Unitary A multi-container in which each frame represents exactly one
payload object.
Hallam-Baker Expires November 25, 2018 [Page 7]
Internet-Draft DARE Data Container May 2018
Serial A multi-container in which payload objects MAY be split
across multiple consecutive frames.
Interleaved A multi-container in which payload objects MAY be split
across multiple frames which may in turn be interleaved with
frames containing other payload objects in complete or partial
form.
3.2. Payload Encryption
DARE container payloads MAY be encrypted as DARE Message Enhanced
Data Sequences [draft-hallambaker-dare-message] . This specification
builds on JSON Web Encryption (JWE) [RFC7516] to provide a flexible
framework allowing a single key exchange to be applied to encrypt
multiple data sequences.
The DARE Container and DARE Message format are designed to compliment
each other:
o DARE Message Format allows a Master Key established in a single
key exchange to be applied to multiple DARE related messages.
o DARE Container Format allows sets of DARE related messages to be
organized so that individual messages and the keying material
required to interpret them can be efficiently retrieved.
4. Index Mechanisms
An index may be appended to an existing file at any time. Since the
use of bidirectional frames makes reading the last record is as
efficient as reading the first, the last record in an indexed file is
usually either the index itself or a pointer to the last index.
An index frame consists of a frame header
Use of index frames provides read access to any record in the file in
O(1) operations but attempting to compiling a complete index with
every write incurs an O(n) penalty on write for both operations and
storage. Accordingly, random read access to a file while it is being
written is better supported using an index tree.
4.1. Tree
Binary search is supported by means of the TreePosition parameter
specified in the FrameHeader. This parameter specifies the value of
the immediately preceding apex.
Hallam-Baker Expires November 25, 2018 [Page 8]
Internet-Draft DARE Data Container May 2018
Calculation of the immediately preceding apex is most easily
described by representing the array index in binary with base of 1
(rather than 0). An array index that is a power of 2 (2, 4, 8, 16,
etc.) will be the apex of a complete tree. Every other array index
has the value of the sum of a set of powers of 2 and the immediately
preceding apex will be the value of the next smallest power of 2 in
the sum.
For example, to find the immediately preceding apex for frame 5, we
add 1 to get 6. 6 = 4 + 2, so we ignore the 2 and the preceding frame
is 4.
The values of Tree Position are shown for the first 8 frames in
figure xx below:
[[This figure is not viewable in this format. The figure is
available at http://mathmesh.com/Documents/draft-hallambaker-dare-
container.html [3].]]
Merkle Tree Integrity check
An algorithm for efficiently calculating the immediately preceding
apex is provided in Appendix C.
4.2. Position Index
Contains a table of index, position pairs pointing to prior locations
in the file.
4.3. Metadata Index
Contains a list of IndexMeta entries. Each entry contains a metadata
description and a list of frame indexes (not positions) of frames
that match the description.
5. Integrity Mechanisms
Frame sequences in a DARE container MAY be protected against a frame
insertion attack by means of a digest chain, a binary Merkle tree or
both.
5.1. Digest Chain calculation
A digest chain is simple to implement but can only be verified if the
full chain of values is known. Appending a frame to the chain has
O(1) complexity but verification has O(n) complexity:
Hallam-Baker Expires November 25, 2018 [Page 9]
Internet-Draft DARE Data Container May 2018
[[This figure is not viewable in this format. The figure is
available at http://mathmesh.com/Documents/draft-hallambaker-dare-
container.html [4].]]
Hash chain integrity check
The value of the chain digest for the the first frame (frame 0) is
H(IV+H(Payload0)), where IV is an initialization vector consisting of
a string of zero bytes and payloadn is the sequence of payload data
bytes for frame n
The value of the chain digest for frame n is H(H(Payloadn-1
+H(Payloadn)), where A+B stands for concatenation of the byte
sequences A and B.
5.2. Binary Merkle tree calculation
The tree index mechanism describe earlier may be used to implement a
binary Merkle tree. The value TreeDigest specifies the apex value of
the tree for that node.
Appending a frame to the chain has O(log2n) complexity provided that
the container format supports at least the binary tree index.
Verifying a chain has O(log2 n) complexity, provided that the set of
necessary digest inputs is known.
To calculate the value of the tree digest for a node, we first
calculate the values of all the sub trees that have their apex at
that node and then calculate the digest of that value and the
immediately preceding local apex.
6. Reference
TBS stuff
6.1. Container Headers
TBS stuff
6.1.1. Structure: ContainerHeaderFirst
Inherits: ContainerHeader
Inherits: ContainerHeader
Hallam-Baker Expires November 25, 2018 [Page 10]
Internet-Draft DARE Data Container May 2018
DataEncoding: String (Optional) Specifies the data encoding for the
header section of for the following frames. This value is ONLY
valid in Frame 0 which MUST have a header encoded in JSON.
6.1.2. Structure: ContainerHeader
Inherits: DAREHeader
Describes a container header. A container header MAY contain any
DARE Message header.
Index: Integer (Optional) The record index within the file. This
MUST be unique and satisfy any additional requirements determined
by the ContainerType.
ContainerType: String (Optional) Specifies the container type for
the following records.
IsMeta: Boolean (Optional) If true, the current frame is a meta
frame and does not contain a payload.
Note: Meta frames MAY be present in any container. Applications
MUST accept containers that contain meta frames at any position in
the file. Applications MUST NOT interpret a meta frame as a data
frame with an enpty payload.
UniqueID: String (Optional) Unique object identifier
ContentMeta: ContentMeta (Optional) Content meta data.
TreePosition: Integer (Optional) Position of the frame containing
the apex of the preceding sub-tree.
IndexPosition: Integer (Optional) Specifies the position in the file
at which the last index entry is to be found
ExchangePosition: Integer (Optional) Specifies the position in the
file at which the key exchange data is to be found
ContainerIndex: ContainerIndex (Optional) An index of records in the
current container up to but not including this one.
PayloadDigest: Binary (Optional) If present, contains the digest of
the Payload.
ChainDigest: Binary (Optional) If present, contains the digest of
the PayloadDigest values of this frame and the frame immediately
preceding.
Hallam-Baker Expires November 25, 2018 [Page 11]
Internet-Draft DARE Data Container May 2018
TreeDigest: Binary (Optional) If present, contains the Binary Merkle
Tree digest value.
Event: String (Optional) Unique object identifier
Labels: String [0..Many] List of labels that are applied to the
payload of the frame.
KeyValues: KeyValue [0..Many] List of key/value pairs describing the
payload of the frame.
First: Integer (Optional) Frame number of the first object instance
value.
Previous: Integer (Optional) Frame number of the immediately prior
object instance value
6.2. Content Metadata Structure
TBS stuff
6.2.1. Structure: ContentMeta
Information describing the object instance
ContentType: String (Optional) The content type field as specified
in JWE
Paths: String [0..Many] List of filename paths for the payload of
the frame.
UniqueID: String (Optional) Unique object identifier
Created: DateTime (Optional) Initial creation date.
Modified: DateTime (Optional) Date of last modification.
6.3. Index Structures
TBS stuff
6.3.1. Structure: ContainerIndex
A container index
Full: Boolean (Optional) If true, the index is complete and contains
position entries for all the frames in the file. If absent or
false, the index is incremental and only contains position entries
Hallam-Baker Expires November 25, 2018 [Page 12]
Internet-Draft DARE Data Container May 2018
for records added since the last frame containing a
ContainerIndex.
Positions: IndexPosition [0..Many] List of container position
entries
Metas: IndexMeta [0..Many] List of container position entries
6.3.2. Structure: IndexPosition
Specifies the position in a file at which a specified record index is
found
Index: Integer (Optional) The record index within the file.
Position: Integer (Optional) The record position within the file
relative to the index base.
6.3.3. Structure: KeyValue
Specifies a key/value entry
Key: String (Optional) The key
Value: String (Optional) The value corresponding to the key
6.3.4. Structure: IndexMeta
Specifies the list of index entries at which a record with the
specified metadata occurrs.
Index: Integer [0..Many] List of record indicies within the file
where frames matching the specified criteria are found.
ContentType: String (Optional) Content type parameter
Paths: String [0..Many] List of filename paths for the current
frame.
Labels: String [0..Many] List of labels that are applied to the
current frame.
6.4. Signature
Payload data MAY be signed using a JWS [RFC7515] as applied in the
DARE Message format [draft-hallambaker-dare-message] .
Hallam-Baker Expires November 25, 2018 [Page 13]
Internet-Draft DARE Data Container May 2018
Signatures are specified by the Signatures parameter in the content
header. The data that the signature is calculated over is defined by
the typ parameter of the Signature as follows.
Payload The frame payload data.
PayloadDigest The value of the PayloadDigest parameter
ChainDigest The value of the ChainDigest parameter
TreeDigestFinal The value of the TreeDigestFinal parameter
If the typ parameter is absent, the value Payload is implied.
A frame MAY contain multiple signatures created with the same signing
key and different typ values.
The use of signatures over chain and tree digest values permit
multiple frames to be validated using a single signature verification
operation.
7. Further Work
The container format is intended to be the basis of future work to
support:
o Very large container sizes (larger than the size of the host's
memory).
o Partitioning of very large data sets across multiple hosts with
parallel append.
o Fault tolerance
7.1. Fast open with random access
The container format is designed to be capable of supporting
efficient random access to frames in containers considerably larger
than the processing memory of the host computer without the need to
pre-load indexes.
A combination of the following strategies is being considered:
o Use memory mapped file views to container data to optimize random
access times while controlling memory use and time taken to
construct memory views.
Hallam-Baker Expires November 25, 2018 [Page 14]
Internet-Draft DARE Data Container May 2018
o When the container is first bound, use the binary tree index data
in TreePosition parameters to support random access operations
until index building is complete.
o Perform Index building operations as a non-blocking background
task.
7.2. Partitioning of very large data sets across hosts
While storage devices capable of storing tends of Tb of data with
RAID redundancy are commonplace, it is generally desirable that there
be at least as many CPU cores as disks. Thus, partitioning of data
sets across multiple hosts becomes desirable for throughput even if a
single host could handle the storage requirement.
7.3. Filtering and redaction
In the types of applications envisaged in the Mesh, almost every data
set may be reduced to collections that are bound to a single account.
While it is obviously desirable that a user's mail messages (for
example) be replicated across multiple machines to provide fault
tolerance, fragmenting the copies of this data set across multiple
machines should be avoided unless the data volumes are so large as to
require it.
7.4. Encryption of large data blocks
The encoding scheme is 64-bit clean throughout and thus supports
containers and frames as large as 18 petabytes. Larger data volumes
could be supported through use of 128-bit integer pointers but even
if the technology to support such data volumes were developed, it is
highly unlikely anyone would want to represent data sets anywhere
near this size in a serial format.
Due to limitations in the design of the encryption schemes that may
be used (e.g. AES-GCM), the maximum encrypted frame size is 64GB.
While this is not currently a major concern for encryption of
individual data files, it is easy to see situations in which an
archive of encrypted files could exceed that amount. One possibility
would be to define a modification to AES -GCM which caused the
encryption key to be incremented by a fixed amount after encrypting a
certain amount of data, though this might well present implementation
challenges unless the maximum data block size was chosen to be
deliberately small so as to force code paths to be exercised.
Another possibility would be to limit the size of encrypted data
frames by requiring the frame pointer to be no larger than 32 bits
and require larger data items to be represented as a sequence of
frames.
Hallam-Baker Expires November 25, 2018 [Page 15]
Internet-Draft DARE Data Container May 2018
7.5. Concurrent Writes
The container format deliberately avoids support for concurrent write
operations. Should this be desirable, some mechanism must be
provided to cache write fragments to an intermediate file and then
consolidate them for writing to the master log.
8. Security Considerations
9. IANA Considerations
10. Acknowledgements
11. Appendix A: Examples and Test Vectors
The data payloads in all the following examples are identical, only
the authentication and/or encryption is different.
o Frame 1..n consists of 300 bytes being the byte sequence 00, 01,
02, etc. repeating after 256 bytes.
For conciseness, the wire format is omitted for examples after the
first, except where the data payload has been transformed, (i.e.
encrypted).
11.1. Simple container
Here the simple container:
Hallam-Baker Expires November 25, 2018 [Page 16]
Internet-Draft DARE Data Container May 2018
f4 5b
f0 59
7b 0a 20 20 22 49 6e 64 65 78 22 3a 20 30 2c 0a
20 20 22 43 6f 6e 74 61 69 6e 65 72
...
7d 2c 0a 20 20 22 44 61 74 61 45 6e 63 6f 64 69
6e 67 22 3a 20 22 4a 53 4f 4e 22 7d
5b f4
f5 01 40
f0 0f
7b 0a 20 20 22 49 6e 64 65 78 22 3a 20 31 7d
f1 01 2c
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f
40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f
50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f
60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f
70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f
80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f
90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f
a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af
b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf
c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf
d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df
e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef
f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
20 21 22 23 24 25 26 27 28 29 2a 2b
40 01 f5
Figure 1
The header values are:
Frame 0
{
"Index": 0,
"ContainerType": "List",
"ContentMeta": {},
"DataEncoding": "JSON"}
Figure 2
Hallam-Baker Expires November 25, 2018 [Page 17]
Internet-Draft DARE Data Container May 2018
Frame 1
{
"Index": 1}
Figure 3
11.2. Payload and chain digests
Frame 0
{
"Index": 0,
"ContainerType": "Chain",
"ContentMeta": {},
"PayloadDigest": "z4PhNX7vuL3xVChQ1m2AB9Yg5AULVxXcg_SpIdNs6c5H0
NE8XYXysP-DGNKHfuwvY7kxvUdBeoGlODJ6-SfaPg",
"ChainDigest": "FEHy24Y6cLModDXWH31kVc2a3TdhjXPooKHpLAb2JbsO1YQ
nJolmowXAYHhkOGY0kg3jrKNTjds0myf4Dw1sdg",
"DataEncoding": "JSON"}
Figure 4
Frame 1
{
"Index": 1,
"PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
"ChainDigest": "7JaijhBvQUOjBiO1_Zt6NtJil8iB0rW9HeM_4iYooc_AaAf
utlF0LLVY6PO7INB-eztypyEqVzgMil9JkjtRGQ"}
Figure 5
Frame 2
{
"Index": 2,
"PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
"ChainDigest": "wJZFYd61nntCJ0Bv80l6-Cn-sR2u3iD0zCRjOLxje8dsKIu
UnP4X1mgeNenNDBdXysrFs3vVAqkC-hfSAPF0Aw"}
Figure 6
Frame 3
Hallam-Baker Expires November 25, 2018 [Page 18]
Internet-Draft DARE Data Container May 2018
{
"Index": 3,
"PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
"ChainDigest": "RORNZxIcM23cZtXPh9vuHhkgiGa_O4a0ZiU0ku2OK4dB974
clvh5F0VZsX7IwVBayAG2nDTdqhyZ-qOnTRiumA"}
Figure 7
11.3. Merkle Tree
Frame 0
{
"Index": 0,
"ContainerType": "Merkle",
"ContentMeta": {},
"PayloadDigest": "z4PhNX7vuL3xVChQ1m2AB9Yg5AULVxXcg_SpIdNs6c5H0
NE8XYXysP-DGNKHfuwvY7kxvUdBeoGlODJ6-SfaPg",
"TreeDigest": "FEHy24Y6cLModDXWH31kVc2a3TdhjXPooKHpLAb2JbsO1YQn
JolmowXAYHhkOGY0kg3jrKNTjds0myf4Dw1sdg",
"DataEncoding": "JSON"}
Figure 8
Frame 1
{
"Index": 1,
"TreePosition": 0,
"PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
"TreeDigest": "fPTYagAvSDP_755jpFUs-Wq6cgvtr5vrFwW-E12vsrbq1ReN
sGzp-V2XqzFPiWaUckACPjegD7ioe1bGzxoWQQ"}
Figure 9
Frame 2
{
"Index": 2,
"TreePosition": 315,
"PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
"TreeDigest": "7fyKKQNLGEeHX1oCsV8NtOdPm615SkDnM1vkcexx2tOuVd5k
kZIdLdsWRCLic9luTSsUN6D6_-c-8ftbhL9dJg"}
Figure 10
Hallam-Baker Expires November 25, 2018 [Page 19]
Internet-Draft DARE Data Container May 2018
Frame 3
{
"Index": 3,
"TreePosition": 315,
"PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
"TreeDigest": "b9ca9Pv-6fxUg-V3ulOhhRngxebkZCxyDmWhQUYeADmSvvPb
jMcNTUJxdDpKlMPrDBInSWMChinsc5s9Tv4byw"}
Figure 11
Frame 4
{
"Index": 4,
"TreePosition": 1441,
"PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
"TreeDigest": "g1hQeWJgDlNoTSGfMb6NhQk5-p6iaAI2_GiAhBM-F2Cp3UvJ
7AR_bC2Drp5YElGXAzC2K5qZ30l7j2D-jqykFw"}
Figure 12
Frame 5
{
"Index": 5,
"TreePosition": 1441,
"PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
"TreeDigest": "p89BhjJAgMMoSrOmot6oaBGa6Dgz-zogZjZ9mm1Iz4yLHxm9
7nWAIBaZFiC1XkuCoP-tr3tag_rHoZhgQV8_PQ"}
Figure 13
Frame 6
{
"Index": 6,
"TreePosition": 2570,
"PayloadDigest": "8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZD
lZeaWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
"TreeDigest": "HEA7EeUGfSjZqjmN3PDp0FVbnixBBXfSQAYm_rNPHVWJVMDu
3SfmxKvN_yBTtMXk-Jad9cyXDKsecLNHLyoQWg"}
Figure 14
Hallam-Baker Expires November 25, 2018 [Page 20]
Internet-Draft DARE Data Container May 2018
11.4. Signed container
11.5. Encrypted container
The following example shows a container in which all the frame
payloads are encrypted under the same master secret established in a
key agreement specified in the first frame.
Frame 0
{
"enc": "A256CBC",
"recipients": [{
"kid": "MCRW5-YRWQK-M4HE3-D4JE6-IYQLM-22ITA-A",
"epk": {
"PublicKeyDH": {
"kid": "MBSNR-T4GZ6-LBZTD-G2CP3-MGT4K-J4VGF-A",
"Domain": "YE6bnq1MlX5ojaJto6PLP_PEwA",
"Public": "Cmh_MVvRHNDOH6GB2Gjke8n8z3UvqNXmNcYQRKPMqEvSsa_zagAw
ChVd3EX1KZ7D8mQYiCbh-F5n2vvydzJ6UOqa4Ur-VzrGCaorB_zdSWoy8DxlHIcSL
iCoE24lf_hHdv25lCu9-wPRXAlr2f3aAPMd9Mq_iqpkSoonXjMdN16uhUxkC-dX70
2YVq2z9S5_Q_sWeYAgdg-9nVVdMQcY3I9mWu5HvbIo_z-lnTUP5mmRD4Otii67EpG
K02QooxPJb0u8UEKFp665ZMu0Snd6nCfBwlxeuVoUgoXlB6vPAc_65c7AdyY3obnF
ai6qZ8wiwV8NFyyyRL3jUuT8mGba1AA"}},
"wmk": "PkGqGm7GBjO1sTgQbBdpccPNqoVq76b0OVxLweTerdzYyx5iRdIXXA"}],
"Index": 0,
"ContainerType": "List",
"ContentMeta": {},
"DataEncoding": "JSON"}
Figure 15
Frame 1
{
"Index": 1,
"ExchangePosition": 0}
Figure 16
Frame 2
{
"Index": 2,
"ExchangePosition": 0}
Figure 17
Hallam-Baker Expires November 25, 2018 [Page 21]
Internet-Draft DARE Data Container May 2018
Here are the container bytes. Note that the content is now encrypted
and has expanded by 25 bytes. These are the salt (16 bytes), the AES
padding (4 bytes) and the JSON-B framing (5 bytes).
f5 03 0c
f1 03 04
7b 0a 20 20 22 65 6e 63 22 3a 20 22 41 32 35 36
43 42 43 22 2c 0a 20 20 22 72 65 63
...
7d 2c 0a 20 20 22 44 61 74 61 45 6e 63 6f 64 69
6e 67 22 3a 20 22 4a 53 4f 4e 22 7d
f0 03
88 01 00
0c 03 f5
f5 01 72
f0 28
7b 0a 20 20 22 49 6e 64 65 78 22 3a 20 31 2c 0a
20 20 22 45 78 63 68 61 6e 67 65 50 6f 73 69 74
69 6f 6e 22 3a 20 30 7d
f1 01 45
88 10 a8 9e 8c f3 f7 d6 fa c1 b7 e7 a0 30 71 bc
ae 7a 89 01 30 04 90 ce 1d 1a 79 f3 92 31 ae 2b
a5 b3 13 29 75 9c 48 07 b1 9b f1 46 d3 95 d2 81
3e 60 4f 76 16 fe 6d 2f 23 99 40 13 84 91 ca 1f
53 30 ec fe 66 b4 e1 38 95 6c 64 9f f7 59 30 5b
b7 d0 b5 1d 7d 29 77 61 21 e2 2e 29 30 66 ed 7a
92 4f 63 d4 7f 9d 3c 7f 12 59 8e 3f d9 f2 5a 63
8e ab 0a 3b 95 fa fc 16 1f b7 43 86 c5 f1 ef 6b
d8 fc 8d 43 44 2f 55 4c 06 e4 19 08 2e ba 28 21
af 04 88 e8 0e 5c 5f 25 7e 5a af 7a d5 44 aa de
2e ed e2 cd 9c df 58 95 b2 d2 05 d8 64 90 0c 65
b6 0b be 7b 86 99 90 e3 4d 73 f9 91 74 bf 07 7f
29 0b 0e 9d 5f bf ce 60 90 4a fb ef 37 9c 62 85
e8 01 2d 87 c8 a4 5c 07 ea dc c0 ec e4 10 d7 48
ee 57 e2 35 aa c0 e7 65 5f 5a 16 b3 cb e0 96 6b
ee 85 86 ff 71 d1 37 7d 65 81 6d 67 88 97 81 58
5d 30 1d 6c 8d 1a 2f 08 b4 d5 d4 80 17 fe 22 54
cd 4c 02 34 fa 4a e4 9c c0 c9 2f a2 e0 3e 58 f6
14 ef a8 53 9b f2 69 f1 d0 10 26 93 45 13 d8 b5
86 57 2e 6c b5 fb 74 eb ee b7 b5 a5 87 f6 ec b8
2d 55 48 31 ac
72 01 f5
f5 01 72
f0 28
7b 0a 20 20 22 49 6e 64 65 78 22 3a 20 32 2c 0a
20 20 22 45 78 63 68 61 6e 67 65 50 6f 73 69 74
69 6f 6e 22 3a 20 30 7d
f1 01 45
Hallam-Baker Expires November 25, 2018 [Page 22]
Internet-Draft DARE Data Container May 2018
88 10 2d 8f 69 b8 89 72 a5 f0 65 76 9d b6 7f 52
bd b2 89 01 30 11 1d 1b a4 ee 8d bc d7 2a 02 08
82 d2 3a 96 a3 36 b3 4a ad 67 d5 31 4c 5e 12 b0
ac 16 d3 ee 9d 34 d2 0a f7 70 6b e7 63 d2 63 21
8d 65 4f 60 13 78 51 11 e9 53 f6 e2 70 79 5d 8f
a9 66 8c 7f 0c 55 d1 8f 9a 4b e7 2c dc b5 85 9e
9c aa a8 2f ba 79 1f b6 c5 3b e8 2d 72 04 74 d8
f6 a6 ca 8e 67 68 d4 64 59 1c be 65 a2 51 02 22
af 94 bd fe ac 01 bc 65 23 e8 fb 4f bb 62 21 f7
77 7d e2 8a 4a a7 86 09 9e b0 b3 3c 44 fa 91 d0
95 22 e0 78 46 ee ab 8b 7c 84 83 c9 e2 67 ae 2e
f6 24 42 2b 42 3e 70 c6 e1 91 ec 12 83 c2 95 ff
a5 ea 09 70 8d 93 35 e8 1d 11 d6 90 6c d8 be 9e
bc d7 98 ad 7f c7 58 cc e3 0d ce 51 d7 5d 05 5b
23 e8 42 ba ea 79 81 98 2a 9b a9 5d e8 2e c9 7e
14 d4 7a 7f 0f f8 aa 5c 2c b2 8c 0a f4 c6 b3 89
52 e4 c8 16 69 aa c2 a3 f8 6e 05 ff c4 8d 04 92
9e 83 c5 b2 8f 8a 1f 19 72 e2 f4 bc 8c 48 4f fc
75 a2 17 73 d5 8a 6e 22 9a b7 f8 a9 86 98 33 a4
4b 57 9a 60 59 a2 19 7f e2 35 f1 e0 63 5d bf 25
c5 8f c6 90 f6
72 01 f5
Figure 18
The following example shows a container in which all the frame
payloads are encrypted under separate key agreements specified in the
payload frames.
Frame 0
{
"Index": 0,
"ContainerType": "List",
"ContentMeta": {},
"DataEncoding": "JSON"}
Figure 19
Frame 1
Hallam-Baker Expires November 25, 2018 [Page 23]
Internet-Draft DARE Data Container May 2018
{
"enc": "A256CBC",
"recipients": [{
"kid": "MCRW5-YRWQK-M4HE3-D4JE6-IYQLM-22ITA-A",
"epk": {
"PublicKeyDH": {
"kid": "MCHK5-FYGDU-J3LE2-H427C-OPD6G-HD4QY-A",
"Domain": "YE6bnq1MlX5ojaJto6PLP_PEwA",
"Public": "kRBM44hYJidVQYULAtk8SHrYUMVjmBxLnPVwEwmi7tVK4xBipCAH
rx06dlbO-gtd9f6V984yEtgXs1GJibo4pCvwXyWq70CoPosc9Rymingo2c6jjLFQP
NvPks0e9kYSVLVTSM34XaTzQKjX4S9AkG4zI4IIrENDPQN6gfRe0Px-NCB-tR3pCW
xJPPxpK2cYopHJX1jYamN69aF7r4ALx2jQSTiwv1558bU-atJNRqYv6dXVseiLpTo
TUVpwG838VGPtjP1DeL_Tuk-di7uxj_n-EUVARr1N0HJwVcjJUHtrOt37jKtM5-RQ
5XKZ3WRvX23N0ae50WjZV-aHmWjvIw"}},
"wmk": "iU--JG-ERFbIiIlpm9WYlOCg9K2yUEefcTFXWnzgkIqsEt84dqP_FA"}],
"Index": 1}
Figure 20
Frame 2
{
"enc": "A256CBC",
"recipients": [{
"kid": "MCRW5-YRWQK-M4HE3-D4JE6-IYQLM-22ITA-A",
"epk": {
"PublicKeyDH": {
"kid": "MAK46-P3CXG-TWHTU-KX7AV-EKBKZ-JIZW4-A",
"Domain": "YE6bnq1MlX5ojaJto6PLP_PEwA",
"Public": "HvdTPFfrFM9x6lzaXC6VqCX_QKxg1KRCirshm4lYs9dmZgvTJTep
QLejdDzuT4IUpCcdDq_lZloCbQVPtl-lHxm60UWYzeJeoBHcxpffXhJVEjFU7Ieet
vBAZP4IHnT_uhyxHomf6oHAo-VvtUa_nPRUpA4PyKApEL6bZPwwwU0WjoCfX25ezs
M8ODZDA2BF4-TQlVHmfzpHIIEZ0oscpcHEcjYBAOjoYrYxdyORUa56E0aXifW5Hb1
R00ZqXPB9dg3j-TEG6Bt1lMk1GSPe3zw339j-9bFl9VLTAlhX5WHvZfDSHQdJev1T
bFV14t_tCaIg0jfKgcZSLzVNN1GwhQA"}},
"wmk": "G69T62RWbpRGzv1IDAKSAB_ETERF3ZGHNx5Jqv-zj-3EPpgJsJZizA"}],
"Index": 2}
Figure 21
12. Appendix B
Hallam-Baker Expires November 25, 2018 [Page 24]
Internet-Draft DARE Data Container May 2018
public long PreviousFrame (long Frame) {
long x2 = Frame + 1;
long d = 1;
while (x2 > 0) {
if ((x2 & 1) == 1) {
return x2 == 1 ? (d / 2) - 1 : Frame - d;
}
d = d * 2;
x2 = x2 / 2;
}
return 0;
}
Figure 22
13. References
13.1. Normative References
[draft-hallambaker-dare-message]
"[Reference Not Found!]".
[draft-hallambaker-jsonbcd]
Hallam-Baker, P., "Binary Encodings for JavaScript Object
Notation: JSON-B, JSON-C, JSON-D", draft-hallambaker-
jsonbcd-12 (work in progress), May 2018.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997.
[RFC7159] Bray, T., "The JavaScript Object Notation (JSON) Data
Interchange Format", RFC 7159, DOI 10.17487/RFC7159, March
2014.
[RFC7515] Jones, M., Bradley, J., and N. Sakimura, "JSON Web
Signature (JWS)", RFC 7515, DOI 10.17487/RFC7515, May
2015.
[RFC7516] Jones, M. and J. Hildebrand, "JSON Web Encryption (JWE)",
RFC 7516, DOI 10.17487/RFC7516, May 2015.
13.2. Informative References
[BLOCKCHAIN]
Chain.com, "Blockchain Specification".
Hallam-Baker Expires November 25, 2018 [Page 25]
Internet-Draft DARE Data Container May 2018
[draft-hallambaker-mesh-architecture]
Hallam-Baker, P., "Mathematical Mesh: Architecture",
draft-hallambaker-mesh-architecture-04 (work in progress),
September 2017.
[RFC5652] Housley, R., "Cryptographic Message Syntax (CMS)", STD 70,
RFC 5652, DOI 10.17487/RFC5652, September 2009.
[ZIPFILE] PKWARE Inc, "APPNOTE.TXT - .ZIP File Format
Specification", October 2014.
13.3. URIs
[1] http://mathmesh.com/Documents/draft-hallambaker-dare-
container.html
[2] http://mathmesh.com/Documents/draft-hallambaker-dare-
container.html
[3] http://mathmesh.com/Documents/draft-hallambaker-dare-
container.html
[4] http://mathmesh.com/Documents/draft-hallambaker-dare-
container.html
Author's Address
Phillip Hallam-Baker
Comodo Group Inc.
Email: philliph@comodo.com
Hallam-Baker Expires November 25, 2018 [Page 26]