Individual Submission to ldup working group R. Harrison Internet Draft J. Sermersheim Document: draft-rharrison-lburp-00.txt Novell, Inc. Category: Proposed Standard October, 1999LDAP Bulk Update/Replication Protocol
Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.1. Abstract
The protocol described in this document allows an LDAP client (a genuine client or an LDAP server acting as a client) to perform a bulk update to a replica on an LDAP server using standard LDAP update operations encapsulated within the LDUP Update Transfer Protocols described in [LDUP MODEL] and [LDUP RUP]. The protocol may be used to initialize all of the entries in an LDAP replica or to incrementally change the existing entries in an LDAP replica. It is suitable for client utilities that need to efficiently initialize a replica with many entries or efficiently make a substantial set of update changes to a replica. It is also suitable as a protocol for replication between a single master replica and its slave replicas.2. Conventions used in this document
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [ReqsKeywords].3. Motivation for protocol
Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 1 LDAP Bulk Update/Replication Protocol October, 1999 This protocol arose from the need to allow LDAP clients to efficiently present large quantities of updates to an LDAP server and have the LDAP server efficiently process them. This protocol introduces a minimum of new operational functionality to the LDAP protocol since the updates discussed in this draft are done with standard LDAP [LDAPv3] update operations. However, this protocol greatly facilitates this process by allowing the server to recognize the clientÆs intent to perform a potentially large set of update operations and then to change its processing strategy to be more efficient than it otherwise could be. In effect, this protocol gives a hint to the server that the LDAP operations bracketed within it can be treated in a special way because they are related to each other. The server may then take actions that would not otherwise be practical to speed the processing of the updates. Examples of such actions include refusing to perform operations for other clients during the update sequence and grouping update operations. Additionally, this protocol deals with a common interoperability problem in this space caused by implementations having different requirements and abilities for handling linear and circular dependencies as entries are created. A common application of this protocol is anticipated to be the initialization or update of an LDAP replica from a set of records represented in an LDIF [LDIF] file. Due to the abilities of various implementations, these files often contain ôout-of-orderö records where an entry is created before its parent or it is made a member of a group before the memberÆs entry has been created. Some implementations create temporary holding objects to deal with these issues, but others do not. This protocol allows the server to reorder update operations in a limited way to deal with such cases.4. Overview of protocol
This replication update protocol utilizes standard LDAP update operations for transmitting replication information between client and server. The client brackets a stream of asynchronous LDAP update operations between two LDAP v3 extended requests: the first notifies the server that the following stream of LDAP update operations are to be treated as a unit of replication information and the second marks the end of the replication stream. This allows the LDAP server to make certain assumptions about the content of the LDAP requests in the replication stream that can be used to facilitate the rapid and efficient processing of the requests. Two styles of update--full and incremental--are defined. Full update creates a completely new set of entries in the target replica; any existing entries in that replica at the start of the replication operation are removed before the LDAP requests in the Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 2 LDAP Bulk Update/Replication Protocol October, 1999 replication stream are processed. The only LDAP operation allowed in a full update replication stream is add. After a server receives and acknowledges a full replication request, the LDAP server MUST not service LDAP requests other than those contained in the replication stream for the replica which is being initialized until the replication stream is completely processed. Incremental update performs a series of incremental changes to the replica; existing entries in the replica are only affected if explicitly modified in some way by the update operations contained in the replication stream. All LDAP v3 update operations defined in [LDAPv3]--add, modify, delete, and moddn--are allowed in the incremental replication stream. When a server is processing an incremental replication request, the replica which is being modified MAY service LDAP requests received on connections other than the connection on which the replication stream is being sent. The LDAP server processing the replication update operation is responsible to send a synchronous extended operation response to the client, signaling the beginning of the replication sequence. It sends an asynchronous response signifying the completion status of each LDAP operation in the replication stream. After the server has completed processing all update operations in the replication stream and has responded to each, it sends a final extended operation response to the client, which signals the end of the replication operation. No attempt is made to deal with the issues associated with multiple- master replicas or to maintain state information such as modification times of entries or attribute values in such a way that updates to the same entry on multiple master replicas can be correctly ordered. For this reason convergence of data between all replicas can only be assured in a single-master replication environment.5. High-level Description of Protocol Flow
AUTHOR'S NOTE: This section is taken, almost verbatim, from section 4 of [LDUP RUP]. It is expected that at some future point, the text in section 4 of [LDUP RUP] will be generalized in such a way that most or all of the overlapping text can be removed from this document. The following section provides a high-level overview of the replication protocol. Throughout this section, the client or server acting as supplier is indicated by the letter "S", and the server acting as consumer is indicated by the letter "C". The construct "S -> C" indicates that the supplier is sending an LDAPv3 operation to the consumer, and "C -> S" indicates that the consumer is sending an LDAPv3 operation to the supplier. Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 3 LDAP Bulk Update/Replication Protocol October, 19995.1 Supplier-initiated replication protocol
S -> C: LDAP bind operation (identity and credentials used are implementation-defined) C -> S: Bind response S -> C: StartReplicationRequest LDAPv3 extended operation. The parameters are: 1) Root of replicated area (unambiguously identifies the replicated area) 2) Supplier's replicaID 3) OID of incremental or full update replication protocol as defined in this document. 4) The protocol initiation type - Supplier-Initiated C -> S: StartReplicationResponse LDAPv3 extended operation. The parameters are: 1) A response code (see section 7) 2) An optional update vector that is included if and only if the response code is REPL_SUCCESS. AUTHOR'S NOTE: The RUP protocol needs to be changed here to accommodate the Bulk Update/Replication Protocol. The client will ignore any update vector sent, so I suggest making the update vector optional for this protocol as well as in cases of errors. S -> C: The supplier may send zero or more asynchronous LDAP v3 update operations. C -> S: Upon completion (successful or otherwise) of each LDAP v3 update operation, the consumer sends the appropriate LDAP response for that operation as defined in [LDAPv3]. S -> C: After all required updates have been sent to the consumer, the supplier sends an EndReplicationRequest LDAPv3 extended operation. C -> S: The consumer responds by sending an EndReplicationRequest LDAPv3 extended operation and then closes the connection. Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 4 LDAP Bulk Update/Replication Protocol October, 19995.2. Consumer-initiated replication protocol
AUTHOR'S NOTE: This section has been copied from section 4.2 of [LDUP RUP]. References to "this document" in this section thus actually refer to [LDUP RUP]. It is anticipated that at some future point that this section will simply refer to section 4.2 of [LDUP RUP]. C -> S: LDAP bind operation (identity and credentials used are implementation-defined) S -> C: Bind response C -> S: StartReplicationRequest LDAPv3 extended operation. The parameters are: 1) Root of replicated area (unambiguously identifies the replicated area) 2) Consumer's replicaID 3) OID of replication protocol to be used (this document defines IETF-LDUP incremental and IETF-LDUP total update protocols) 4) The protocol initiation type - Consumer-Initiated in this case S -> C: StartReplicationResponse LDAPv3 extended operation. The parameters are: 1) A response code (see section 7) S -> C: The supplier server disconnects from the consumer server, and then connects to the consumer, beginning a Supplier- Initiated protocol session (see section 4.1).5.3 Client-initiated replication protocol
There are times when an LDAP client that doesn't have an outstanding replication agreement with a consumer server wants to initiate an update replication session as a supplier using this protocol. In this case, the following changes apply to the Supplier-initiated replication protocol: 1) The client, acting as supplier, sends the special value of [value to be determined] as the supplier's replicaID in the StartReplicationRequest. 2) The consumer, recognizing the supplier as a client, omits the update vector in the StartReplicationResponse, for every value of the response code, including REPL_SUCCESS. Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 5 LDAP Bulk Update/Replication Protocol October, 19996. Elements of Protocol
The LDAP Bulk Update/Replication protocol works within the framework of the Replication Update Protocol [LDUP RUP]. Unless otherwise stated, those elements of protocol taken from the Replication Update Protocol framework are to be used precisely as described there.6.1 StartReplicationRequest Extended Operation
The StartReplicationRequest Extended Operation is defined in Section 5.1 of [LDUP RUP]. Clarification of the use of the replicaID is required for this protocol, as is the specification of the replicationProtocolOID. Other parameters of the StartReplicationRequest are as defined in Section 5.1 of [LDUP RUP].6.1.1 replicaID
In the case where a replication agreement exists between the supplier and consumer, the replicaID is as specified in Section 5.1 of [LDUP RUP]. When a client, acting as supplier, is initiating a replication request, it MUST send the special value of [value to be determined] as the supplier's replicaID in the StartReplicationRequest.6.1.2 replicationProtocolOID
The OID for the Full Update Protocol is 2.16.840.1.113719.1.27.1.4.1. The OID for the Incremental Update Protocol is 2.16.840.1.113719.1.27.1.4.2.6.2 StartReplicationResponse
In the case where a replication agreement exists between the supplier and consumer, the updateVector is as specified in Section 5.1 of [LDUP RUP]. When a client, acting as supplier, has initiated a replication request, the updateVector SHALL NOT be returned in the StartReplicationResponse.6.3 ReplicationUpdateOperation
The ReplicationUpdateOperation is not utilized in the LDAP Bulk Update/Replication protocol. Standard LDAP update requests as defined in sections 4.6, 4.7, 4.8, and 4.9 of [LDAPv3] are used to send all update requests to the consumer server.6.4 EndReplicationRequest
Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 6 LDAP Bulk Update/Replication Protocol October, 1999 The EndReplicationRequest is sent as specified in section 5.4 of [LDUP RUP]. The replicaUpdateVector MUST NOT be sent by the supplier, and the returnConsumerUpdateVector MUST be set to FALSE.6.5 EndReplicationResponse
The EndReplicat1ionResponse is sent as specified in section 5.4 of [LDUP RUP]. No replicaUpdateVector will be returned due to the requirement that the supplier not request it.7. Semantics of Full and Incremental Update Protocols
Since both the full and incremental update protocols use standard LDAP operations to transmit update information, no attempt is made by the protocol to include the information needed to support full multi-master replication as defined by [LDUP RUP]. Although entries and their associated attributes and attribute values can be synchronized using this protocol, no CSNs are included in the update stream; updates made using this replication protocol are treated the same as other LDAP operations wherein they are deemed to occur at the present. Operational attributes such as modifiersTimeStamp MAY be included in the replication stream, and if they are, they SHOULD be put into the entry. The LDAP update operations that form the replication stream may be sent asynchronously by the supplier to the consumer. This means that the supplier need not wait for a response from one LDAP update operation before sending the next. Suppliers MUST NOT send the replication stream over multiple connections or do other things that could change the order of the operations in the replication sequence. The LDAP operations in the replication stream, plus the initial state of entries in the consumer replica in the case of incremental update, collectively represent the desired final state of the consumer replica. The consumer server may take any action required to efficiently achieve this desired final state of the replica. Examples of such actions include reordering LDAP requests to ensure that linear dependencies within the replication stream are correctly ordered, breaking an LDAP modification request into two or more separate modification requests to adequately resolve a circular dependency within the replication stream, and aggregating LDAP update operations to process them more efficiently. When taking actions of this sort, the server MUST NOT reorder LDAP requests in such a way that the desired final state is different from that which would have been achieved before the action was taken.7.1 Semantics of the Full Replication Protocol
Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 7 LDAP Bulk Update/Replication Protocol October, 1999 Full replication creates a completely new set of entries in that replica; any existing entries in that replica at the start of the replication operation are removed before the LDAP requests in the replication stream are processed. The only LDAP operation allowed in a full replication stream is add. If any other LDAP operation is received as part of the replication stream, the server MUST return unwillingToPerform as the response for the operation. While a server is processing a full replication request, the consumer server MUST not service LDAP requests for the replica involved in the replication operation other than those contained in the replication stream including LDAP requests on other connections. This requirement is designed to allow implementers the freedom to implement highly-efficient methods of handling the replication stream without being constrained by the need to maintain a live, working database while doing so. If the full update sequence fails for some reason (e.g. a connection is abruptly broken in the middle of an otherwise-successful update sequence) the server MUST remove any entries created during the update sequence, thereby leaving the replica in a known state.7.2 Semantics of the Incremental Replication Protocol
Incremental replication performs a series of incremental changes to the replica; existing entries in the replica are only affected if explicitly modified in some way by the update operations contained in the replication stream. All LDAP v3 update operations--add, modify, delete, and moddn--are allowed in the incremental replication stream. If a non-update operation is received as part of the replication stream, the server MUST return unwillingToPerform as the response for the operation. When a server is processing an incremental replication request, the consumer server MAY service LDAP requests for the replica involved in the replication operation on connections other than that being used to transmit the replication stream.8. Notes To Implementers
It is RECOMMENDED that, if possible, the consumer LDAP server refer non-replication requests for the replica being updated to another server containing that replica during the time that consumer LDAP server has the replica off-line for replication. Implementations MAY choose to perform the operations in the replication stream with special permissions to improve performance. Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 8 LDAP Bulk Update/Replication Protocol October, 19999. Security Considerations
Implementations should ensure that a supplier making a Bulk Update/Replication request is bound with appropriate permissions. There is a potential for loss of data, especially with the full update protocol, which removes all entries in a replica as part of its operation. Second, forcing the removal of all entries in a replica may consume large amounts of server resources. Third, unlike other replication protocols, no existing replication agreement between supplier and consumer is required. These risks increase if the consumer server also processes the replication stream with special permissions to improve performance. For these reasons, implementers should carefully consider which identities or permissions are required to perform Bulk Update/Replication Protocol operations and take steps to ensure that only connections bound with appropriate permissions are allowed to perform them. The data contained in the replication stream may contain passwords and other sensitive data. Care should be taken to properly safeguard this information while in transit between supplier and consumer. 10. References [LDAPv3] Wahl, M., Howes, T., and S. Kille, "Lightweight Directory Access Protocol (v3)", RFC 2251, December 1997. [LDIF] Gordon Good, "LDAP Data Interchange Format (LDIF)", INTERNET- DRAFT: draft-good-ldap-ldif-04.txt, 22 June 1999. [LDUP MODEL] John Merrells, Ed Reed, and Uppili Srinivasan, "LDAP Replication Architecture", INTERNET-DRAFT: draft-ietf-ldup- model-01.txt, June 25, 1999. [LDUP RUP] Ellen Stokes and Gordon Good, "The LDUP Replication Update Protocol", UNPUBLISHED INTERNET-DRAFT: draft-ietf-ldup- protocol-00.txt. [ReqsKeywords] Scott Bradner. "Key Words for use in RFCs to Indicate Requirement Levels". RFC 2119.11. Author's Addresses
Roger Harrison Novell, Inc. Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 9 LDAP Bulk Update/Replication Protocol October, 1999 122 E. 1700 S. Provo, UT 84606 +1 801 861 2642 roger_harrison@novell.com Jim Sermersheim Novell, Inc. 122 E. 1700 S. Provo, UT 84606 +1 801 861 3088 jimse@novell.com Full Copyright Statement "Copyright (C) The Internet Society (date). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implmentation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Individual Submission - Expires April 14, 2000 Harrison & Sermersheim 10
Datatracker
draft-rharrison-lburp-00
This is an older version of an Internet-Draft that was ultimately published as RFC 4373.
