Internet-Draft Richard V. Huber
LDAP Duplication/Replication/Update Gerald F. Maziarski
Protocols WG AT&T Laboratories
Intended Category: TBD Ryan D. Moats
Expires: August 2001 Coreon, Inc.
February 2001
General Usage Profile for LDAPv3 Replication
draft-ietf-ldup-usage-profile-00.txt
Status of This Memo
This document is an Internet-Draft and is in full conformance with all
provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering Task
Force (IETF), its areas, and its working groups. Note that other
groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference material
or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/lid-abstracts.txt.
The list of Internet-Drafts Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Copyright Notice
Copyright (C) The Internet Society (2001). All Rights Reserved.
Abstract
Support for replication in LDAP directory systems is often one of the
key factors in the decision to deploy them. But replication brings
design constraints along with its benefits.
We discuss some of the factors that should be taken into consideration
when designing a replicated directory system. Both programming and
architectural/operational concerns are addressed.
Huber, et al Expires August 2001 [Page 1]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
Huber, et al Expires August 2001 [Page 2]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
Table of Contents
1 Introduction.......................................................4
2 Meta-data Considerations...........................................4
2.1 Schema Considerations...........................................4
2.2 Replication Agreements..........................................5
2.3 Access Control..................................................6
2.4 Change Logs.....................................................7
3 Naming Considerations..............................................7
4 Conflict Resolution Considerations.................................7
4.1 Consistent Access after Changes.................................7
4.2 Conflict Resolution in Single-Master Systems....................8
4.3 Problem Cases...................................................9
4.4 General Principles.............................................10
5 Failover Considerations...........................................11
5.1 Common Issues..................................................11
5.2 Single Master Issues...........................................11
5.3 Multi-Master Issues............................................12
6 Impact of Non-LDAP Changes/Constraints............................12
6.1 Changes Outside of LDAP........................................12
6.2 Application Triggers...........................................13
7 Security Considerations...........................................13
8 Acknowledgements..................................................13
9 References........................................................14
Authors' Addresses...................................................14
Full Copyright Statement.............................................15
Huber, et al Expires August 2001 [Page 3]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
1 Introduction
As LDAP directories become part of critical infrastructure in many
applications, the need for extremely high reliability and availability
becomes significant. And as directories are more widely used, the load
on individual directory servers gets higher.
Distributed, replicated directories can reduce the problems of
reliability and capacity. But applications that work well with a
single, standalone directory will develop problems in a distributed
environment unless both the application and the environment are
carefully designed.
The particular areas of concern depend partly on whether the
distributed directory is a single- or multi-master system, but there
are many concerns that are common to both. In the remainder of the
document, we have flagged some sections as being specific to single- or
multi-master directories. Unflagged sections pertain to both.
This document addresses general issues. There may be additional drafts
in the future that address specific applications.
2 Meta-data Considerations
Any LDAP directory contains meta-data as well as the user data in the
directory. Examples of this meta-data include descriptions of the data
in the directory (e.g. schema), policies for use of the data (e.g.
access controls), and configuration/status information (e.g.
replication agreements); this is not an exhaustive list.
Many types of meta-data are stored in the directory itself; accessible
as regular data or as operational attributes. Since meta-data can
appear in the directory, issues arise when it is replicated.
2.1 Schema Considerations
If the schema of one or more of the copies of a replica differs from
the schema of the other replicas, then there exists the possibility of
schema mismatch when data is exchanged between them. The extensibility
feature of LDAP schema nearly guarantees that replica rings comprised
of a heterogeneous mix of systems will not contain homogeneous schema
because of directory vendors' built-in extensions. Because a given
directory may not utilize its complete schema, not all of the potential
Huber, et al Expires August 2001 [Page 4]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
schema differences in a ring will result in a schema mismatch under
replication.
Schema mismatch issues are further complicated by the possibility of
replicating the "subschemaSubentry" itself. Some directories
distribute schema changes through that mechanism. Currently there is
no standard for LDAP schema representation. In the absence of such a
standard, schema interoperability is not possible in the IETF sense.
Designers should establish common schema on all servers holding a
replica.
It is useful to separate out two classes of schema mismatch: "latent",
when the differences do not intersect the collection of data in the
ring; and "salient", when they do.
Schema mismatches that can cause data corruption in one or more of the
replicas must result in meta-data (e.g. log entries) to comply with
Requirement MM5 of [ReplicaReq]. However, not all schema differences
produce corruption in all circumstances. Some schema differences may
have little or nor impact on the proper storage of replicated data.
However, any time data is added to the directory it could change a
latent mismatch into a salient mismatch. It is therefore recommended
that all schema mismatches be removed from, or otherwise rationalized
among, the replica ring schemas, if possible. The tool described by
requirement AM8 of [ReplicaReq] would help designers detect schema
conflicts as early as possible. The other option that can be
considered in this situation is the use of fractional replication to
replicate only those attributes that do not have differences.
Examples of mismatches:
1. Base syntax of an attribute type
2. Structure Rule of an object class
3. Optional vs. Mandatory attribute in an object class
4. Structural vs. Auxiliary in an object class
5. Object class not defined
6. Attribute type not defined
7. Object identifiers differs on an attribute type or on an
object class
8. Type and number of attributes defined in a class
9. Multi-valued vs. Single valued attribute types
10. ACL format (and consequently, ACL calculus)
11. Matching rule of an attribute type
12. Naming collisions of attribute type names
13. Attribute name aliasing ("street" vs. "streetAddress")
2.2 Replication Agreements
Replication Agreements are central to replication, they allow
configuration of most of the aspects of the replication process,
Huber, et al Expires August 2001 [Page 5]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
including the triggers for replica cycles (from Requirement M1 in
[ReplicaReq]), critical OID information (from Requirement M6 in
[ReplicaReq]), and replication process parameters (Requirement M7 in
[ReplicaReq]). Through the use of a standard schema (Requirement S2)
it is possible to replicate the replication agreement.
If a replication agreement includes replication credentials, the
agreement SHOULD be read protected in the directory and transport of
the replication agreement SHOULD be encrypted. Further, the directory
administrator should consider and understand the possibility of the
replication of the agreement impacting replication processes.
2.3 Access Control
- Access Control Information (ACI) is treated as though it were stored
as attributes in the directory [ACModel]
- LDAP declares [RFC2251] that changes resulting from a single LDAP
MODIFY are atomic (but see caveats in multi-master case)
- ACI affecting a given entry may not be part of that entry (it could
be part of a group entry or part of an ancestor of the entry in
question)
- ACI cannot always be changed atomically with associated data changes
- If you aren't careful, you can leave windows where data is
unprotected
- Access control changes SHOULD be made before adds and after deletes
- Access control changes and the associated data changes SHOULD be
made on same system.
- Life gets interesting when access control policy is inherited from a
superior partition, but that is mostly a partitioning issue, not a
replication issue (once we figure out how partitioning works we may
find out it is also a replication issue).
Even when ACL information is faithfully replicated among heterogeneous
members of a replica ring that agree on transfer format, there is no
guarantee that an ACI change is expressed similarly everywhere in the
ring. This caveat is partly due to partitioning effects mentioned
above, and partly due to vendor differences with regard to the
expression of security policy. The access control mechanisms specified
in [ACModel] are neutral with respect to policy inheritance mechanisms,
explicit vs. implicit denial, and group nesting; all of which may
differ between vendors, and will affect or modify the expression of
access control information.
Huber, et al Expires August 2001 [Page 6]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
2.4 Change Logs
Requirement G4 of [ReplicaReq] states that meta-data must not grow
without bound. Since it is unrealistic to assume that meta-data won't
be needed during replication, how and when meta-data can be purged must
be considered.
Replicas that use connections with intermittent quality SHOULD use
explicit replica cycle scheduling. This allows notification of a
delayed replica and manual intervention before the meta-data grows
without bound. In extreme cases, it may be necessary to remove these
replicas from the replication ring and re-add them add once better
connectivity is available.
Second, it is also possible for a consumer to receive changes that
cannot be applied. The replication agreement SHOULD include
information for when the consumer should stop deferring the change and
declare the changes as inapplicable, thus invoking whatever procedure
is in place to comply with requirement MM5 of [ReplicaReq] during
multi-master replication.
3 Naming Considerations
A number of naming models have been proposed for directories
([RFC1255], [RFC2377], [CIMNames]), and many others have been
implemented on an ad hoc basis. Each of these models specifies the
naming attributes to be used and provides rules for using them. The
naming scheme may also specify containment rules.
The naming plan applies to the directory as a whole, not the individual
servers holding replicas. In a heterogeneous replicated environment,
all of the replicating servers must be capable of supporting all of the
rules for the naming plan in use for that directory.
Some directory implementations have naming constraints. If one of them
is part of a replicated directory, those constraints will have to be
observed by all participating directories. If the environment contains
implementations with incompatible constraints there is a major problem.
This SHOULD be checked as early in the design phase as possible to
avoid costly problems.
4 Conflict Resolution Considerations
4.1 Consistent Access after Changes
Many operations on a directory are done as a set of steps. For
example, a new object may be created in one operation, and its values
may be filled in as part of a separate LDAP operation. An
Huber, et al Expires August 2001 [Page 7]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
administrator may add a user to a directory, and that user may then try
to log in using the new entry.
Replicated LDAP directories provide loose consistency [ReplicaReq]. A
new entry or a change to an existing entry will not reach all replicas
immediately; there will be some delay before changes are available on
all replicas. Changes made (e.g. adding a new user) on one physical
system may appear to be "lost" if checked on another physical system
before replication is complete.
In general, LDAP applications SHOULD be prepared to operate correctly
in the face of replication delays. In some cases, this means designing
to allow for delay. In the case of the newly created user, it SHOULD
be standard practice to ask the user to wait a while before trying to
use the entry. In the case where the new object must be filled in, the
application SHOULD make appropriate use of LDAP sessions to make sure
that the same server is reached for both operations.
As a general rule, an LDAP application SHOULD bind once and not unbind
until a complete set of related operations have been performed.
In the single-master case, all write requests go to one server. If a
set of related reads and writes are done, they SHOULD all be done on
the master if possible. Ideally, only sets of related operations that
can not include a write SHOULD go to one of the slave servers. But
load balancing concerns may make this impractical.
In some cases, related requests will deal with data in different
partitions which are not all available on a single server. In this
case, it is safer to keep sessions open to all servers rather than
closing the session with one server and opening one with another
server.
It may not always be obvious to clients that they are using different
servers. If a load distribution system is used between the client and
the server, the client may find that a change request and a subsequent
lookup are directed to different physical servers even though the
original requests were sent to the same server name and/or address.
Since LDAP is session oriented, any load distribution system used
should take sessions into account. Thus, keeping all related read and
write requests within a single bind/unbind session SHOULD be the goal
in this situation as well.
4.2 Conflict Resolution in Single-Master Systems
It is possible that resolution conflicts could occur in a single master
replication system. Because requirement SM2 of [ReplicaReq] is a
Huber, et al Expires August 2001 [Page 8]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
SHOULD and not a MUST, it is possible for implementers to reorder
changes. If changes are reordered, it is quite possible for a conflict
to occur. If schema changes are critical and must be moved to the
front of the replication queue, then a schema change that deletes an
attribute for some object class that follows some changes that delete
values of the soon-to-be-deleted attribute can cause conflict. If the
schema change moves to the head of the queue, the consumer servers
might have to delete an attribute that still has values, and then
receive requests to delete the values of an attribute that is no longer
defined.
However, directory administrators may have scenarios where re-ordering
of replication information is desirable. On a case-by-case basis, the
directory administrator should make such decisions.
4.3 Problem Cases
4.3.1 Atomicity
The fact that replication does not guarantee the time order arrival of
changes at a consumer allows situations where changes that were applied
successfully at the supplier may fail in part when an attempt is made
to apply the same change at the consumer. Examples appear below.
4.3.1.1 Locking
There is an entry with distinguished name DN that contains attributes
X, Y, and Z. The value of X is 1. On replica A, a ModifyRequest is
processed which includes modifications to change that value of X from 1
to 0 and to set the value of Y to "USER1". At the same time, replica B
process a ModifyRequest which includes modifications to change the
value of X from 1 to 0 and to set the value of Y to "USER2" and the
value of Z to 42. The application in this case is using X as a lock
and is depending on the atomic nature of ModifyRequests to provide
mutual exclusion for lock access.
In the single-server case, the two operations would have occurred
sequentially. Since a ModifyRequest is atomic, the entire first
operation would succeed. The second ModifyRequest would fail, since
the value of X would be 0 when it was attempted, and the modification
changing X from 1 to 0 would thus fail. The atomicity rule would cause
all other modifications in the ModifyRequest to fail as well.
In the multi-master case, it is inevitable that at least some of the
changes will be reversed despite the use of the lock. Assuming the
changes from A have priority per the conflict resolution algorithm, the
value of X should be 0 and the value of Y should be "USER1" The
interesting question is the value of Z at the end of the replication
Huber, et al Expires August 2001 [Page 9]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
cycle. If it is 42, the atomicity constraint on the change from B has
been violated. But for it to revert to its previous value, grouping
information must be retained and it is not clear when that information
can be safely discarded. Thus, requirement G6 in [ReplicaReq] may be
violated.
The utility of locking mechanisms cannot be guaranteed, and therefore
are likely to be misleading. Its use in multi-master environments is
therefore deprecated.
4.3.1.2 Partitioning
Suppose two servers, A and B, are members of Replica-ring X; and
servers B and C are members of replica-ring Y. A ModifyRDN operation
on server B moves an entry from ring X to ring Y. If the appropriate
delete is done on A, the operation may be considered successful even
though hidden circumstances on ring Y prevent the move. If on C another
change made the modifyRDN that worked on B impossible, because an RDN
of a different GUID exists there already, then the change on A is
inconsistent and will need to be reversed. Others examples of cases of
this class include group membership modification and access control
scoping.
4.4 General Principles
The examples above discuss some of the most difficult problems that can
arise in multi-master replication. While they can be dealt with,
dealing with them is difficult and can lead to situations that are
quite confusing to the application and to users.
The common characteristics of the examples are:
. Several directory users/applications are changing the same data
. They are changing the data at the same time
. They are using different directory servers to make these changes
. They are changing data that are parts of a distinguished name or they
are using ModifyRequest to both read and write a given attribute
value in a single atomic request
If any one of these conditions is reversed, the types of problems
described above will not occur. There are many useful applications of
multi-master directories where at least one of the above conditions
Huber, et al Expires August 2001 [Page 10]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
does not occur. For cases where all four do occur, application
designers should be aware of the possible consequences.
5 Failover Considerations
One of the major reasons to use directory replication is to improve
reliability of the directory system as a whole. Replication permits
hot- and warm-standby configurations to be built easily.
But there are some issues that must be considered during design. In
this situation, single-master systems actually raise more concerns than
multi-master. Both are addressed below.
5.1 Common Issues
In both the single- and multi-master cases, clients must be able to
find an alternate quickly when a server fails. Some possible ways to
do this are detailed in [FindingLDAP] and [LDAPinDNS]. If all else
fails, a list of possible servers can be built into client
applications. Designers should consider how clients are notified that
the server is again available.
When the failed server comes back up, it is brought back into
synchronization with the other servers and is ready to take requests.
Note that the process used to bring a failed server back into
replication can also be used to add a server to a set of replicating
servers. In this case, the new server might be initialized from a
backed-up copy of the directory or it may acquire the entire DIB via
replication. The former method is usually preferable when the
directory is large.
5.2 Single Master Issues
In a single-master system, the master is a single point of failure, as
all modification has to originate at the master server. When high
availability is a requirement, a quick, automated failover process for
converting a slave replica to a new master is desirable, as the
failover process time becomes a major factor in determining system
availability. The considerations in section 5.1 apply here; clients
must know how to find the new master or a new slave in case of failure.
To aid in promotion of a slave replica, the master could replicate
control information and meta-data (included replication credentials) so
that this information is available during failover promotion. This
data may either be replicated on a single "failover designate" slave
(which would become the master in during failover) or it could be
Huber, et al Expires August 2001 [Page 11]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
replicated to all slaves. The first possibility has the advantage of
minimizing the amount of extra replication while the second more
robustly handles multiple failovers (i.e. failover of the newly
promoted master to another slave before the original master has been
restored). If this method is followed, data privacy mechanisms SHOULD
be used to protect the replication session.
In addition, if data privacy mechanisms (e.g. encryption) were used to
protect the replication session, the new master MUST have the necessary
key information. Further this key information should be independent of
the master that is using it (i.e. not tied to the IP address of the
master server).
Restoration of the failed or broken master can be handled in one of two
ways:
- It could join the ring and function as a slave.
- It could join the ring and negotiate with the new master to
synchronize and then take over as master.
In either case, clients need a way to know that a new server is
available.
The slave replicas may also use the replication agreement to filter
which master is allowed to submit changes. Such a model allows the
slave servers to function correctly when the master server is "broken"
and sending out incorrect updates. However, then it is necessary to
update the replication agreement during the fail over process so that
the slaves will accept updates from the new master. This is the case
for both the original failure and the restoration of the restored
master if that is how the restored master rejoins the replica ring.
5.3 Multi-Master Issues
Typically, a multi-master configuration was used because high
availability is required for writes as well as reads in the directory.
Multi-master implies that there are multiple active servers prepared to
take write requests, so there is no "switchover" time in this case.
But clients still need to be able to find an alternate server, so the
considerations of Section 5.1 apply here.
6 Impact of Non-LDAP Changes/Constraints
6.1 Changes Outside of LDAP
Huber, et al Expires August 2001 [Page 12]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
LDAP directories are typically built on top of some database or file
system. Thus there are ways to change the data that do not go through
the normal LDAP change mechanisms (e.g. ModifyRequest). If the data is
modified outside of LDAP, the changes will not be checked for schema
conformance nor will access controls be checked as the changes are
made. Since both integrity and security checks are omitted, security
can be adversely affected.
Also, many systems use the normal LDAP modification mechanisms to
trigger replication. Changes made using non-LDAP mechanisms may not be
replicated at all, leading to inconsistencies between replica copies.
6.2 Application Triggers
Directory servers commonly integrate one or more specific applications.
To achieve this integration the directory server may intercept updates
and run application-specific "trigger" code. Such triggers enforce
directory invariants that cannot be expressed by the LDAP schema.
A simple trigger example is password policy enforcement. A directory
server might interpret a request to replace the current value of the
userPassword attribute with some new value as a request to first check
that the new value conforms to the server's password policy (e.g. the
value is sufficiently long and complex) before storing the new value.
Using this trigger the directory server voids the security risk
associated with passwords that are easy to attack.
A more complex trigger example is password hashing. A directory server
might interpret a request to replace the current value of the
userPassword attribute with some new value as a request to compute one
or more secure hashes of the new value and store these hashes in one or
more attributes, storing no value in the userPassword attribute. Using
this trigger the directory server avoids the security exposure of
storing the plaintext password.
Replication between directory servers with different application
triggers will compromise directory integrity.
7 Security Considerations
The document discusses issues that arise in replication. Some of these
issues are security related (e.g. replication of access control
information) and the security implications are discussed in the
relevant sections.
8 Acknowledgements
Huber, et al Expires August 2001 [Page 13]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
The authors would like to thank Mark Brown for identifying the issues
discussed in section 6.
9 References
[ACModel] E. Stokes, R. Blakeley, D. Rinkevich, R. Byrne, "Access
Control Model for LDAPv3", Internet Draft, draft-ietf-ldapext-acl-
model-06.txt, July 2000.
[CIMNames] Desktop Management Task Force, "Guidelines for CIM-to-LDAP
Directory Mappings", DMTF Specification DSP0100, May 2000 (available
online at http://www.dmtf.org/spec/DEN/DSP0100.htm).
[FindingLDAP] R. Moats, R. Hedberg, "A Taxonomy of Methods for LDAP
Clients Finding Servers", Internet Draft, draft-ietf-ldapext-ldap-
taxonomy-04.txt, October 2000.
[LDAPinDNS] M. Armijo, L. Esibov, P. Leach, R. L. Morgan, "Discovering
LDAP Services with DNS", Internet Draft, draft-ietf-ldapext-locate-
04.txt, August 2000.
[ReplicaReq] E. Stokes, R. Weiser, R. Moats, R. Huber, "LDAPv3
Replication Requirements", Internet Draft, draft-ietf-ldup-replica-req-
06.txt, January 2001.
[RFC1255] The North American Directory Forum, "A Naming Scheme for
c=US", RFC 1255, September 1991.
[RFC2119] S. Bradner, "Key Words for Use in RFCs to Indicate
Requirement Levels", RFC 2119, March 1997.
[RFC2251] M. Wahl, T. Howes, S. Kille, "Lightweight Directory Access
Protocol", RFC 2251, December 1997.
[RFC2377] A. Grimstad, R. Huber, S. Sataluri, M. Wahl, "Naming Plan
for Internet Directory-Enabled Applications", RFC 2377, September 1998.
Authors' Addresses
Richard V. Huber
Room C3-3B30
AT&T Laboratories
200 Laurel Avenue South
Middletown, NJ 07748
USA
E-Mail: rvh@att.com
Telephone: +1 732 420 2632
Fax: +1 732 368 1690
Huber, et al Expires August 2001 [Page 14]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
Gerald F. Maziarski
Room C3-3Z01
AT&T Laboratories
200 Laurel Avenue South
Middletown, NJ 07748
USA
E-Mail: gfm@att.com
Telephone: +1 732 420 2162
Fax: +1 732 368 1690
Ryan D. Moats
Coreon, Inc.
15621 Drexel Circle
Omaha, NE 68135
USA
E-Mail: rmoats@coreon.net
Telephone: +1 402 894 9456
Full Copyright Statement
Copyright (C) The Internet Society (2000). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it or
assist in its implementation may be prepared, copied, published and
distributed, in whole or in part, without restriction of any kind,
provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing the
copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of developing
Internet standards in which case the procedures for copyrights defined
in the Internet Standards process must be followed, or as required to
translate it into languages other than English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL
NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Huber, et al Expires August 2001 [Page 15]
Internet-Draft Usage Profiles for LDAPv3 Replication February 2001
Funding for the RFC Editor function is currently provided by the
Internet Society.
Huber, et al Expires August 2001 [Page 16]
