Internet Research Task Force J. Levine
Internet-Draft Taughannock Networks
Intended status: Experimental December 21, 2010
Expires: June 24, 2011
An efficient method to publish ranges of IP addresses in the DNS
draft-levine-iprangepub-00
Abstract
The DNS has long been used to publish lists of IPv4 address ranges in
blacklists and whitelists. The size of the IPv6 address space makes
the entry-per-IP approach used for IPv4 lists impractical. A new
technique for publishing IPv6 address ranges is described. It is
intended to permit efficient publishing and querying, and to have
good DNS cache behavior.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on June 24, 2011.
Copyright Notice
Copyright (c) 2010 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
Levine Expires June 24, 2011 [Page 1]
Internet-Draft Efficent IP range publishing December 2010
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Assumptions and Goals . . . . . . . . . . . . . . . . . . . . 4
3. DNS record format . . . . . . . . . . . . . . . . . . . . . . 4
4. Lookup algorithm . . . . . . . . . . . . . . . . . . . . . . . 5
5. Details of blob representation . . . . . . . . . . . . . . . . 5
6. Building and updating DNSBLs . . . . . . . . . . . . . . . . . 6
6.1. Building static DNSxLs . . . . . . . . . . . . . . . . . . 6
6.2. Building and updating dynamic DNSxLs . . . . . . . . . . . 7
7. Estimated performance . . . . . . . . . . . . . . . . . . . . 8
8. Security considerations . . . . . . . . . . . . . . . . . . . 8
9. Topics for further consideration . . . . . . . . . . . . . . . 9
10. IANA considerations . . . . . . . . . . . . . . . . . . . . . 9
11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 10
11.1. References - Normative . . . . . . . . . . . . . . . . . . 10
11.2. References - Informative . . . . . . . . . . . . . . . . . 10
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 10
Levine Expires June 24, 2011 [Page 2]
Internet-Draft Efficent IP range publishing December 2010
1. Introduction
For many years, the Domain Name System[RFC1034] [RFC1035] has been
the Internet's de facto distributed database. Blacklists and
whitelists of IPv4 addresses have been published in the DNS using a
simple system adapted from rDNS[RFC5782]. A DNSxL (a DNSBL or DNSWL)
is a DNS sub-tree, typically also a DNS zone, with each listed IP
having an A and/or TXT record at a name corresponding to the IP
address. While this publication method has worked well for IPv4
addresses, the size of the IPv6 address space makes an analogous
approach unworkable.
In an IPv4 Internet, each network is typically limited to a few
thousand or at most a few million addresses. A single host typically
has a single address, or at most a few hundred addresses. The
limited size of each network forces a host to use its assigned
address or addresses. In IPv6 networks, hosts typically use
Stateless Address Autoconfiguration [RFC4862] to select an IP
address, with the low 64 bits of the address being almost entirely
arbitrary. A hostile host sending mail can easily switch to a new
address for every message it sends, never reusing an address, due to
the vast size of the IPv6 address space.
An IPv6 DNS blacklist would use wildcards or a specialized server
such as rbldnsd [RBLDNSD] to list entire /64 or larger ranges, but
that does not help DNS performance. Since wildcards are expanded by
the DNS server, every query for a unique IP address causes a unique
query to the DNSBL's server. Moreover, every unique query will take
up a cache entry in the client's local DNS cache, either a regular
entry if the DNSBL lists the query's address, or a negative
entry[RFC2308] if it doesn't. In the event that hostile mailers
(which we will call "spammers" for short) use a unique address per
message, the normal DNSBL query traffic will both flood the DNSBL's
server and fill local caches with useless single-use entries, forcing
out other cached data and causing excess traffic to all the other
servers the caches query as well.
For blacklists, an obvious approach would be to limit the granularity
of DNSBLs, so that, say, each /64 had a separate listing, and the
queries only used the high 64 bits of each address. While this might
limit the damage from DNSBL queries, it is not helpful for DNS
whitelists, which by their nature list individual IP addresses, and
are likely to be far more popular with IPv6 mail than they have been
with IPv4 mail.
The problem of looking up an address in a sorted list of IP addresses
stored on a remote DNS server is not unlike the problem of searching
for a key in a sorted list of keys stored on a disk, a problem that
Levine Expires June 24, 2011 [Page 3]
Internet-Draft Efficent IP range publishing December 2010
is common in database applications. An approach called _B-trees_ has
been widely used in databases since the 1970s, and is described in
standard references such as [KNUTHV3]. The technique in this
document stores ordered sets of IP address ranges in DNS records,
using a straightforward adaptation of the way that B-trees store
ordered sets of key strings in disk blocks.
2. Assumptions and Goals
This design is intended to meet this set of design goals:
1. Handle arbitrary mixtures of CIDR ranges and individual IPs.
2. Work well with existing DNS servers and caches. The number of
different queries should be limited, both to limit cache->server
traffic and the number of cache entries it uses. I assume that
client->cache queries are cheap, while cache->server queries are
much more expensive.
3. Don't assume senders will be well behaved. In particular, bad
guys may use a new IP for every message, hopping around within
each /64 (or whatever the network size is) in which a zombie
lives, and there will be many zombies active at the same time.
4. Don't assume MTAs remember query results from one lookup to the
next, so the necessary info should be available in the DNS cache.
(If they do, it doesn't hurt.)
5. Be compatible with DNSSEC, but don't depend on DNSSEC.
6. Don't attempt to be compatible with existing DNSBLs. I expect
these lists will mostly be served from something like rbldnsd,
although it would also be possible to have an external program
create the zone files and serve them from a conventional DNS
server such as BIND.
3. DNS record format
Each DNS record is a blob of bytes containing multiple self-
describing logical entries, each of which is a CIDR range, which
might be a single IP if the CIDR is a /128. In each blob, the CIDRs
are in address order from lowest to highest. The name of each record
is the address of the entry that implicitly points to it, which is
the entry immediately preceding it. In this context, there is no
advantage to doing rDNS-style nibble reversed naming, so the name is
just 32 ASCII characters, the address as a hex number. One blob is
Levine Expires June 24, 2011 [Page 4]
Internet-Draft Efficent IP range publishing December 2010
the tree root, which is named with all zeros, such as
00000000000000000000000000000000.dnsxl.example.
The simplest way to represent a blob is as a TXT record, with all the
strings concatenated. The more entries that fit in a blob, the
better this will work, so although it would be possible for each
logical entry to be a separate string in the TXT record, it doesn't
to do that.
Each blob contains a set of the CIDRs that are in the DNSxL. For
blobs that are not leaves (identified by a per-blob flag, described
below), the the entries in the blob also partition the address space,
with the CIDRs between the ones in the current blob in sub-blobs,
each named by the entries in the current blob.
To minimize edge cases, the root blob always contains the lowest and
highest entries.
4. Lookup algorithm
A client looks up an IP address in the DNSxL as follows:
1. Make the root blob the current blob and fetch it.
2. Search through the current blob for a CIDR entry that contains
the target IP address. If you find it, stop, the DNSxL contains
the IP.
3. If the IP address is lower than the first entry in the current
blob, or higher than the last entry, stop, the DNSxL does not
contain the IP.
4. If this is a leaf blob, stop, the DNSxL does not contain the IP.
5. Find the entry in the current blob that is just below the IP.
Use the address in that entry as the name of new blob, fetch that
blob, and make it the current one.
6. Go to Paragraph 2.
It should be evident that this is analogous to a binary tree search,
except that each node has a lot more than two descendants.
5. Details of blob representation
The first byte of each blob is a flag:
Levine Expires June 24, 2011 [Page 5]
Internet-Draft Efficent IP range publishing December 2010
L P P P P P P P
The L bit is set if this is a leaf. The PPPPPPP is the prefix size.
If all of the CIDR addresses in the blob have the same initial bits
as the name of the blob does, which is fairly likely due to the way
that IPv6 addresses are allocated, this is the number of common
initial bits. The prefix bits are not stored in the blob's entries,
but are logically prefixed to each CIDR address in the blob.
After that is some number of CIDRs:
X S S S S S S S
Address
X is reserved for now. SSSSSSS is 0-127, the CIDR size minus one.
The Address is 128-P-S bits long, rounded up to a byte. That is, it
omits the prefix bits which are obtained from the name of the blob,
and the suffix bits beyond the CIDR size.
For example, say an entry is 2001:1234:5678:9ABC::/64, and the prefix
size is 16. Then the entry would be (in hex)
3f 12 34 56 78 9A BC
The 3f is the CIDR size (64-1), the 2001 is omitted since that's the
prefix, and the rest is the address.
Each blob is as big as will fit in a DNS answer. If you don't
believe in EDNS0, the limit is about 450 bytes. If you do believe in
EDNS0, it's whatever size you think clients ask for, probably in the
2K to 4K range.
6. Building and updating DNSBLs
[[ Note: This section is somewhat speculative. I have experience
with disk-based B-trees, but haven't implemented any of this yet. ]]
DNSxLs are typically compiled as a list of IP addresses or CIDR
ranges. They must be turned into a tree of named blobs before being
served in the DNS. The technique varies a little depending on
whether the tree will be updated incrementally, or rebuilt from
scratch if it changes.
6.1. Building static DNSxLs
A static DNSxL should have the minimum number of blobs, each of which
should be as full as possible. The technique to build the tree is a
Levine Expires June 24, 2011 [Page 6]
Internet-Draft Efficent IP range publishing December 2010
direct adaptation of the B-tree building technique in [WPBTREE].
Start with a sorted list of addresses or CIDRs. Save one entry for
the next pass, then take as many entries as possible and make a blob
out of them. Repeat saving and entry and creating a blob until all
entries are used. These will be the leaf blobs. Now, take the list
of saved addresses and repeat to create the blobs at the next level
up. Keep repeating to create each level of the tree until the
process creates only one blob. Insert the one saved entry into that
blob which will be the root. That might make the blob overflow, in
which case split it in half and move the first, middle, and last
entries into a new root blob.
When the list changes, rebuild it from scratch.
6.2. Building and updating dynamic DNSxLs
One of the reasons that B-trees are so widely used is that it is
possible to update them efficiently without rebuilding them. The
same should apply here.
The general approach to updates is to add or delete an entry, then if
that makes a blob too big or makes it empty, rebalance the tree to
fix the problem. If a blob is too big, move entries into an adjacent
blob if possible, otherwise split the blob. This will require
updating the blob above, which in turn might overflow if the update
involves adding rather than replacing an entry. (It might overflow
even with a replacement if it makes the compressible prefix shorter.)
In that case, repeat the process, potentially all the way to the
root. When deleting an entry, if the blob becomes empty, move its
pointer entry from the blob above up into one of the adjacent blobs,
then adjust the upper blob as needed. Again, this might cause
overflow in which case move entries between blobs or split the full
one.
A tree in which each blob is completely full is quite expensive to
update. The first insertion will cause a leaf to overflow, with
overflows rippling all way up the tree to the root. It would
probably be a good idea when building a list that is intended to be
updated to leave some slack space in each blob, to limit the ripple
effect from changes.
A significant difference between this design and a conventional
B-tree is the version skew due to DNS caching. In a normal B-tree,
the pages (blobs) are locked while being changed, and the changes are
immediately visible to all the clients. In this case, the clients
cache each blob for the DNS TTL. If updates change the entries in
non-leaf blobs, they will break the links between blobs since they
Levine Expires June 24, 2011 [Page 7]
Internet-Draft Efficent IP range publishing December 2010
use the entries as pointers. A possible band-aid is to add temporary
CNAME records at the former names pointing to the closest new name,
so most (admittedly not all) of the entries can still be located.
Once the TTL on the old blobs has expired, the CNAMEs can be deleted.
7. Estimated performance
The size of entries varies depending on the CIDR size and the amount
of prefix compression. An uncompressed /64 would take 9 bytes, so
I'll use 10 bytes as an estimate of average entry size. With EDNS0
and 4K records, that would allow 400 entries per blob. A two-level
tree could hold 160,000 entries, a three level tree 64 million
entries, which would need 160,000 blobs. Large v4 DNSBLs like the
CBL have about seven million entries now, so this should be adequate.
If blobs have to fit in 512 byte responses, that would be about 40
entries per blob, A five-level tree could hold 100 million entries in
about 2.5 million blobs, still adequate.
The number of queries for any particular lookup is the number of
levels, which is unlikely to be more than five. The cache behavior
obviously depends on both the layout of the entries and the query
pattern, but this design avoids some obvious worst cases. If a /64
is either entirely listed, not listed at all, or just has a single
/128, all queries for addresses in that /64 will refetch the same
four records. If a large range of addresses is either listed in one
CIDR, or not listed at all, all queries will refetch the same set of
blobs, which would be likely to be cached.
The total number of DNS records used is always less than the number
of records for a traditional entry-per-IP DNSxL. Since all the DNS
queries are made by following the tree of entries, clients shouldn't
make queries that fail, so there will be no negative cache entries.
(This isn't quite true due to version skew in updated DNSxLs, but
it's hard to imagine a plausible scenario in which there would be a
large number of failing queries.) This suggests that the overall
cache behavior will be no worse than, and quite possibly much better
than the behavior of IPv4 DNSxLs.
8. Security considerations
Semantically, there is little difference between a DNSxL published
using this scheme and one published using the traditional entry per
IP approach, since both publish the operators opinion about some
subset of the IP address space.
One significant practical difference is that it is much easier for
Levine Expires June 24, 2011 [Page 8]
Internet-Draft Efficent IP range publishing December 2010
clients to obtain copies of all or part of the database. For a
traditional DNSxL, the only way to determine its contents is to query
the entire IPv4 address space (or at least the active part of it),
which would require several billion queries, and is deterred by rate
limiting the queries. In this scheme, the names of all of the DNS
records are easy for clients to determine, so they can efficiently
walk the tree. While rate limiting is possible, it is less effective
since clients fetch more data with each query. It is also easy for a
client to fetch all the entries for a particular IP range, such as
the range of a network the client controls to see what parts of it
are blacklisted.
9. Topics for further consideration
o Conventional IPv4 DNSBLs generally can return an A record or a TXT
record. The A record often has information coded in the low byte
or two that identifies the type of listing, or which of several
sub-lists the entry is on. If it's important to return a byte or
two of result, it would be straightforward to add the byte or two
to each entry, at a modest increase in entry size and hence some
loss in performance. The TXT records are typically either all the
same, or there's one per A value, perhaps with the IP address
interpolated into the string to provide a URL to look up. Those
strings could be constructed in client libraries, with templates
stored in the DNS, e.g. the string for code 42 might be in a TXT
record at 42._strings.dnsxl.example.
o There might be better ways to do prefix compression, e.g., a per-
entry field that says how many bits are the same as the previous
entry. Entries in blobs could be bit rather than byte aligned,
although I expect that would be a lot more work for minimal extra
compression. There may be clever tricks to allocate entries into
blobs to maximize the size of the prefix in each blob. If a blob
consists entirely of /128's it might be worth a special case,
leaving out the length byte on each entry.
10. IANA considerations
This document makes no requests to IANA. All data are stored and
queried using existing DNS record types and operations.
11. References
Levine Expires June 24, 2011 [Page 9]
Internet-Draft Efficent IP range publishing December 2010
11.1. References - Normative
[RFC1034] Mockapetris, P., "Domain names - concepts and facilities",
STD 13, RFC 1034, November 1987.
[RFC1035] Mockapetris, P., "Domain names - implementation and
specification", STD 13, RFC 1035, November 1987.
11.2. References - Informative
[KNUTHV3] Knuth, D., "The Art of Computer Programming: Volume 3,
Sorting and Searching", 1998.
[RBLDNSD] Tokarev, M., "rbldnsd: Small Daemon for DNSBLs".
[RFC2308] Andrews, M., "Negative Caching of DNS Queries (DNS
NCACHE)", RFC 2308, March 1998.
[RFC4862] Thomson, S., Narten, T., and T. Jinmei, "IPv6 Stateless
Address Autoconfiguration", RFC 4862, September 2007.
[RFC5782] Levine, J., "DNS Blacklists and Whitelists", RFC 5782,
February 2010.
[WPBTREE] Wikipedia, "B-tree", December 2010.
Author's Address
John Levine
Taughannock Networks
PO Box 727
Trumansburg, NY 14886
Phone: +1 831 480 2300
Email: standards@taugh.com
URI: http://jl.ly
Levine Expires June 24, 2011 [Page 10]