Minutes IETF99: maprg
Measurement and Analysis for Protocols
||Minutes IETF99: maprg
Measurement and Analysis for Protocols Research Group (maprg) - IETF-99 (Prague)
Intro & Overview, Project "Advertisements" - Mirja Kuhlewind & Dave
Agenda is combination of:
-Reporting on nascent measurement results
-Call to arms for new projects spinning up
Chairs keen to hear feedback on how the agenda has been structured.
Lars Eggert: We had ANRW workshop on Saturday for second time, next summer will
be in Montreal. There will be a CFP etc. This year we didn't realise conflict
with IMC deadline - will try to avoid next year. Please keep in mind if you
have something relevant to publish. If you are a North American academic and
would like to be considered for TPC chair, let us know.
Dave Plonka: Where to send comments and feedback on ANRW?
Lars: irtf-discuss list.
IPv6 Reluctanct Devices and Applications - Mikael Abrahamsson
Abstract: There are numerous reports of consumer products and devices that
either (a) don't support IPv6 yet or (b) ostensibly support IPv6 in that they
configure an IPv6 address, but subsequently seem not to use it, perhaps due to
their Happy Eyeballs implementation. We request that a measurement effort be
undertaken to identify and report on this behavior, so that we can mitigate
this impediment to IPv6[-only] operation.
Dave: This is something we're interested in - can we proactively do something
with measurement requests from IETF?
Fred Baker: Root operator looking at IPv6 deployment suggested that we go for
interop test where we can prove that devices do IPv6-only. So people would get
the message in a positive manner. Is this useful?
Mikael: Would this be for apps? Devices?
Fred: Any kind of device. Cisco is working with customers on IPv6-only
deployment. Several networks too. So you're extending this to the reach of
Mikael: Apple did NAT64 testing - IPv6-only WiFi - one way of testing. At least
then you know that device and application can speak IPv6 - spreading that
approach into more places would be good.
Tim Winters (UNH-IOL): Involved the program that monitor device IPv6 stacks. No
smart TVs though - coverage not complete. We can configure device to turn on
IPV6, but not necessarily on by default - could make that a requirement in
future. Testing we do shows more application testing in the last year in
IPv6-only environments - starting to see movement in that direction from
Mikael: Would love to see IPv6-default-on be a requirement.
Tim Winters (UNH-IOL): People don't always want it on all the time - I'll take
an action to press on this.
Mikael: When people say 'market doesn't require IPv6' I say market requires
INternet access - today that is both stacks with default to IPv6-on.
Fred: Have preso later today on challenges of enabling IPv6
Dave: Interested in ideas from the group about how to conduct this measurement
study. More discussion on the list, and if we have something to share by the
next meeting that would be good.
Fingerprint-based detection of DNS hijacks using RIPE Atlas - Pawel Foremski
Abstract: The DNS protocol does not protect from hijacking resolver traffic. In
fact, some network operators and government agencies transparently redirect DNS
queries made from end-user devices to popular recursive resolvers to their own
servers. This effectively allows the hijacking party to easily monitor, block,
and manipulate the content published to the World Wide Web, and in general to
control its Internet users. In this paper, we present a novel fingerprinting
algorithm that is able to detect hijacking of recursive DNS resolver traffic.
By sending specific DNS queries to the resolver and analyzing the replies using
machine learning algorithms, we are able to classify the resolver as legitimate
or rogue. We evaluate our technique through the RIPE Atlas platform, collecting
DNS measurements on Google Public DNS and OpenDNS servers from 9,500 probes
distributed around the world.
Aaron Falk: Encourage you to publish this so I can share a link to your work.
Surprised to see that RTT was on there given use of anycast for DNS. Was
anycast a factor in your measurements? Did you look at the contents of the
response - just interception or data manipulation?
Pawel: Regarding anycast, as far as I know both Google and OpenDNS are anycast
operators so couldn't compare. Regarding replies we haven't had time yet to
analyse - that is future work.
Colin Perkins: Did you do anything to infer the reason for the hijacking?
Pawel: Yes, interesting - that is future work. Hard to think of source of
information to answer this question - motivation often hard to infer.
Jana Iyengar: This is great work, please publish. Have you considered calling
some of these ISPs and just talking to them?
Pawel: Yes, we are considering that - but it would be future work.
Jana: We've done some of that and found that in some cases they are no clueful
about the implications of what they are doing and willing to learn.
Giovane Moura: Root DDoS attacks in 2015 - can download data and noticed that
hijacked probes had very short RTT - if you just use public measurements with
standardised queries - difference between your method and that would be
interesting. Does your method show more hijacks?
Pawel: That's interesting. We would need to compare the measurements. Data is
open. Would be easy to run more meaurements.
??: How did you split your data set into training and testing? Did you perform
any cross validation?
Pawel: 30 times randomly, so splitted equally. No cross validation yet but on
Aaron Falk channeling jabber (Simon Ferlin): Would be interesting to see user
experience for hijacked probes?
Pawel: Definitely but how would you collect this data? May crowd-sourcing?
Rate-limiting of IPv6 traceroutes is widespread: measurements and mitigations -
Abstract: With IPv6, high-frequency traceroutes show many more missing hops
than IPv4. This appears to be due to the fact that RFC 4443 states IPv6 routers
MUST rate-limit the ICMPv6 error packets that traceroute and similar programs
depend on to determine hops on a route. We measured the characteristics of this
rate-limiting from many vantage points to thousands of targets across
continents. We find that about 2/3 of all routers exhibit rate-limiting at
frequencies < 100Hz. The distribution of the data suggest most of these
rate-limits might be factory defaults. We discuss strategies to overcome this
limitation, including altering the order of packets across single or multiple
traceroutes, and merging information from traceroutes to different targets.
Mirja: Are there any defaults in the RFC?
Jen Linkova: Problem here is that there is default value that you could change.
But hardware cannot be changed. Don't know how many ICMP messages router is
sending to other hosts. Could also be affected by neighbour discovery - I'm not
surprised to see IPv6 is worse here. Changing defaults may not be possible due
Mikael Abrahamsson: Did you try changing source addresses.
Pablo: Did quick and dirty test - looks like limits are per router, not per
Mikael: I don't think you can make that leap - could be per line card, there
might be several of these. Did you differentiate between getting between 0, 1,
2 or 3 responses for a router?
Pablo: Only testing each hop once.
Mikael: Would be interesting to see what happened if you did send more.
Andrew McGregor: Having worked on a few IPv6 router implementations it's likely
there are higher limits out there. You're likely to find that despite CPU being
fast, there isn't enough PPS available between control and forwarding planes to
get limits higher - you find limits in exotic places inside router chassis. Yes
it's usually linecard CPU doing this.
Pablo: There are additional rate limits beyond token bucket?
Aandrew: possibly, or limit is not token bucket, but some PPS limit inside
router implementation - e.g. for all control plane packets.
Pablo: The data I got are consistent in most cases with a token bucket.
John Brzozowski: It would be interesting to run this internally within a large
provider network. Addressing methodology could prevent getting a proper
response - maybe links aren't using globally routable addresses.
Pablo: IPv4 traces often include RFC1918 addresses. Didn't filter out
non-globally routable addresses. Can look at data more for this.
Mirja: Did you just say you want to conduct some measurements and make the
John: I could do that.
Andrew ??: limit may be a hardware limit, e.g. if slides are used. You will hit
limits, ICMP are usually software imposed limits, talking to vendors could help
find a way to conduct this testing without difficulties like this. It's really
old hardware that you would have a concern with.
Olivier Bonaventure: Is the data that you collect publicly available?
Pablo: Not sure - talk to me offline.
Olivier: Public data projects (CAIDA etc.) help to incentivise future
researchers to make data public, and use existing traceroute tools instead of
writing their own tools to get private data.
kIP: a Measured Approach to IPv6 Address Anonymization - David Plonka
Abstract: Related pre-print, "kIP: a Measured Approach to IPv6 Address
Anonymization" (Plonka & Berger)Privacy-minded Internet service operators
anonymize IPv6 addresses by truncating them to a fixed length, perhaps due to
long-standing use of this technique with IPv4 and a belief that it's "good
enough." We claim that simple anonymization by truncation is suspect since it
does not entail privacy guarantees nor does it take into account some common
address assignment practices observed today. To investigate, with standard
activity logs as input, we develop a counting method to determine a lower bound
on the number of active IPv6 addresses that are simultaneously assigned, such
as those of clients that access World-Wide Web services. In many instances, we
find that these empirical measurements offer no evidence that truncating IPv6
addresses to a fixed number of bits, e.g., 48 in common practice, protects
individuals' privacy. To remedy this problem, we propose kIP anonymization, an
aggregation method that ensures a certain level of address privacy. Our method
adaptively determines variable truncation lengths using parameter k, the
desired number of active (rather than merely potential) addresses, e.g., 32 or
256, that can not be distinguished from each other once anonymized. We describe
our implementation and present first results of its application to millions of
real IPv6 client addresses active over a week's time, demonstrating both
feasibility at large scale and ability to automatically adapt to each network's
address assignment practice and synthesize a set of anonymous aggregates
(prefixes), each of which is guaranteed to cover (contain) at least k of the
active addresses. Each address is anonymized by truncating it to the length of
its longest matching prefix in that set.
Tim Chown: Don't think you can tell identifiers apart based on randomisation.
Sometimes you get new IID, sometimes you don't depending on address class.
Dave: 3rd column is stability of address. Can use sliding time window and test
whether I've ever seen that value before. I'm saying this works if
temp/slaac/privacy address mechanism - we can reject persistent pseudorandoms.
Alex ???: We can apply to other areas, e.g. rate-limiting for services. Would
be great if in some way we could take this work into IETF to make a protocol
where I could identify suitable prefix lengths for anonymization. European data
legislation is also relevant now.
Dave: agree. are results portable - if place you're porting them from has
better visibility than where you are.
Alex: Can we publishing results and aggregating them? To know where to cut the
Dave: Integrate observations in some private way. There are methods that allow
you to share addresses in an opaque way.
Mirja: Guidance document that we should take on as a group.
Dave: Don't know where in the IETF community to talk about such a thing. Could
be candidate for a draft in this group initially.
Pablo Alvarez: In terms of re-use, do you have some idea of temporal stability.
Do ISP properties change over time? How often would you need to re-new and
Dave: Don't know, but you're right to be concerned. It works well for offline
analysis. We need to do more work here.
Mikael Abrahamsson: Anti-abuse handling in IETF would be interestd if you can
identify prefix lengths allocated by ISPs. Where is the household level of
aggregation. RIRs have a way of letting ISPs publish this, but nobody does.
Measurement would help.
Dave: We can discover, or they can tell us - would love to compare the two.
Measuring Latency Variation in the Internet - Toke Hoiland-Jorgensen
Abstract: Related paper, "Measuring Latency Variation in the Internet" (Toke
Hoiland-Jorgensen et al., CoNEXT '16)We analyse two complementary datasets to
quantify the latency variation experienced by internet end-users: (i) a
large-scale active measurement dataset (from the Measurement Lab Network
Diagnostic Tool) which shed light on long-term trends and regional differences;
and (ii) passive measurement data from an access aggregation link which is used
to analyse the edge links closest to the user. The analysis shows that
variation in latency is both common and of significant magnitude, with two
thirds of samples exceeding 100 ms of variation. The variation is seen within
single connections as well as between connections to the same client. The
distribution of experienced latency variation is heavy-tailed, with the most
affected clients seeing an order of magnitude larger variation than the least
affected. In addition, there are large differences between regions, both within
and between continents. Despite consistent improvements in throughput, most
regions show no reduction in latency variation over time, and in one region it
even increases. We examine load-induced queueing latency as a possible cause
for the variation in latency and find that both datasets readily exhibit
symptoms of queueing latency correlated with network load. Additionally, when
this queueing latency does occur, it is of significant magnitude, more than 200
ms in the median. This indicates that load-induced queueing contributes
significantly to the overall latency variation.
Mikael Abrahamsson: Is same TCP congestion avoidance algorithm used throughout
Toke: Good questions. Don't know.
Andrew McGregor: MLabs server side TCP hasn't changed over this period. Client
side may have.
Mikael: TCP Window scaling might also have changed over this time span.
Mirja: Did you try to detecet presence of AQM?
Toke: No, not directly.
Mirja: Data is from 2015? Most interesting data would be 2015-2017.
Toke: Think it would be straightforward to re-run this on a new dataset.
Mirja: Please share dataset and code with list.
Vaibhav Bajpai: Caveat - measurements towards MLabs - may not be 'normal'.
Toke: Not speedtest.net.
Vaibhav: That's not normal traffic either - traffic to facebook, google, etc.
Qiabong Xie (Netflix): how big was total data set?
Toke: 200M - we erred on the side of minimising false positives, hence ended up
with 5M flows.
Qiabong Xie (Netflix): if you see 80% of flow that see congestion it might not
be sender induced.
Toke: some flows do not get a single congestion event at all over 10 seconds,
but latency increases quite a lot.
Qiabong Xie (Netflix):: percentage means that rest of the flows don't have this
pattern - likely cross-traffic induced, not sender induced.
Toke: could be - i wouldn't say that none of the other flows didn't have
sender-induced congestion events, but certainly for a lot of these flows that
Garret Tyson:: How did MLAb evolve during your measurement period? for example
africa has more measurement servers now than previously.
Toke: that's true.
Measuring YouTube Content Delivery over IPv6 - Vaibhav Bajpai
Abstract: To appear in the SIGCOMM Computer Communication Review (CCR), July
issue. We measure the performance of YouTube over IPv6 using ~100 SamKnows
probes connected to dual-stacked networks representing 66 different origin
ASes. Using a 29-months long (Aug 2014 - Jan 2017) dataset, we show that
success rates of streaming a stall-free version of a video over IPv6 have
improved over time. We show that a Happy Eyeballs (HE) race during initial TCP
connection establishment leads to a strong (more than 97%) preference over
IPv6. However, even though clients prefer streaming videos over IPv6, we
observe worse performance over IPv6 than over IPv4. We witness consistently
higher TCP connection establishment times and startup delays (~100 ms or more)
over IPv6. We also observe consistently lower achieved throughput both for
audio and video over IPv6. We observe less than 1% stall rates over both
address families. Due to lower stall rates, bitrates that can be reliably
streamed over both address families are comparable. However, in situations,
where a stall does occur, 80% of the samples experience higher stall durations
that are at least 1s longer over IPv6 and have not reduced over time. The worse
performance over IPv6 is due to the disparity in the availability of Google
Global Caches (GGC) over IPv6. The measurements performed in this work using
the youtube test and the entire dataset is made available to the measurement
Mo Boucadair: There is no recommendation about 300ms in the RFC. It's up to the
implementors to choose timers. Formally there's no update to the RFC required.
Dave Wilson: Interesting that stall rates, bit rates the same - would user
really see a difference? Not sure that latency is that important for video.
Vaibhav: Startup delay is also worth looking at.
Dave Wilson: A lot of people think measuring latency is a guide to customer
experience - but your results suggest otherwise.
Mikael Abrahamsson: Would be interesting to get the diagrams for the last 12
months of data to see what the current state is vs. what was happening in 2013.
Don't want historical problems to impact conclusion about what to do now.
Jana Iyengar: regarding dip on slide 8 - is client running TCP?
Vaibhav: Yes that is TCP.
Jana: one of the graphs in my presentation coming up next is going to coincide
exactly with that dip and i don't know why.
Ian Swett: Did you compare IPv4-only vs. Happy Eyeballs?
Vaibhav: We run a test over IPv6 then over IPv4. Almost at same time.
Ian: GGC nodes that don't speak IPv6 are just ignored from sample?
Vaibhav: Yes, we have to.
Ian: Results are surprising to me, but I will make some enquiries.
Mikael Abrahamsson: If closest GCC node is IPv4 only, then IPv6 tests will be
over a different path.
Vaibhav: Would be interested in expansion of dual-stacked GGC nodes.
Giovane Moura: are earlier measurements comparable with this?
Vaibhav: 2015 work was latency towards websites in general, this work is
specific to YouTube.
Jen Linova: Would be interested to see difference between when nearest GGC node
is dual-stack vs. when nearest node is IPv4-only.
Vaibhav: Yes we are working on this now.
Jen: Onus is on operators to provide IPv6 connectivity - it's not up to Google
to provision IPv6 on GGC nodes - they'll be dualstack when on dualstack
Vaibhav: Understood, my mistake.
The QUIC Transport Protocol: Design and Internet-Scale Deployment - Jana Iyengar
Abstract: QUIC is an encrypted, multiplexed, and low-latency transport protocol
designed from the ground up to improve transport performance for HTTPS traffic
and to enable rapid deployment and continued evolution of transport mechanisms.
QUIC has been globally deployed at Google on thousands of servers and is used
to serve traffic to a range of clients including a widely-used web browser
(Chrome) and a popular mobile video streaming app (YouTube). We estimate that
7% of Internet traffic is now QUIC. This talk will cover the Internet-scale
process that we used to perform iterative experiments on QUIC, performance
improvements seen by our various services, and our experience deploying QUIC
Mirja: Which connection control does this data use?
Jana: Cubic with two connection emulation because TCP uses to connection.
Andrew Doganow: Now QUIC is by default enabled in Chrome. I live in Singapore
where you're not helping; you making it slower because slow connections is e.g.
1 Gig or 200MB/s. What we're seeing is some of the infrastructure is blocking
QUIC for various 'security' reasons. Are you looking at making it more
Jana: First thing is that it shouldn't hurt. If operators are blocking it,
that's fine - Chrome will use TCP.
Andrew D.: Fallback delay is what I'm talking about - it still works, but
addtional delay is noticeable.
Jana: Expect that QUIC will be more feature-rich in future, e.g. encryption
integrated. Would like to better understand what delays are that you are
seeing. We are working on issues where middleboxes will drop QUIC traffic after
Colin Perkins: Are you able to break down performance results based on TCP
version running at the receiver. Do you win out against all versions fo TCP, or
do you not have the data to tell?
Jana: Don't have that data. There's some other micro-benchmarking data. Chrome
user base is mostly Windows, so you are asking which version of Windows. Chrome
distribution data would help - might be able to get that.
Mikael Abrahamsson: Are you doning packet-level path MTU discovery?
Jana: Did a bunch of experiments and fixed at 1350 bytes for all of our clients
Martin Gunnarsson: Were mobile clients connected over wifi or cellular?
Jana: Don't have that data right now.
Jana: Please read the paper. There's a fun story about middlebox ossification;
believe it or not QUIC already ossifying.
Mirja: Thanks to all our speakers. If you have data, send it to the mailing
list. If you would like to present at our next meeting the earlier you can let
the chairs know the better.