Minutes IETF101: maprg
minutes-101-maprg-00

Measurement and Analysis for Protocols Research Group (maprg) Agenda @ IETF-101
(London)
========================================================================================

Date: Tuesday, March 20, 9:30-12:00 - Tuesday Morning session I
Room: Sandringham

Scribed by: Mat Ford

Abstracts of all talks are included at the foot of these notes.

## Intro & Overview
Dave Plonka (DP) and Mirja Kühlewind (MK)

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-intro-overview-05

## Heads-up talk: Challenges in measuring 1 Gbps access speeds
Jason Livingood

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-measurement-challenges-in-the-gigabit-era-01

## Heads-up talk: Zesplot in five minutes - An attempt to visualise IPv6
address space Luuk Hendriks

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-zesplot-an-attempt-to-visualise-ipv6-address-space-00

DP: Interesting transitioning from tool to useful measurement (anycast)

## Update on previous presentation: Measuring the quality of DNSSEC deployment
Roland van Rijswijk-Deij

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-measuring-the-quality-of-dnssec-deployment-00

## Update on previous presentation: Update on IPv6 Performance Data
Tommy Pauly (TP)

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-update-on-ipv6-performance-data-tommy-pauly-01

Tim Chown: Where are UK cellular measurements coming from?

TP: This is just sampling from Apple devices, iPhones etc., other devices may
have been configured differently. This is the view that we have of UK carriers.

TC: I believe some UK operator has about a million handsets on IPv6, but maybe
not Apple devices.

MK: Do you have absolute numbers?

TP: For this specific measurement, it's order of 1000 connections, so very
little.

Lorenzo Colliti: Based on your charts I get the impression that across the
board you end up using IPv6 half the time of the total connections.

TP: Based on happy-eyeballs data that would be mainly from hosts that are not
dual stack.

LC: Say you had a large head start for IPv6, how would the numbers change?

TP: Looking at our happy-eyeballs data, when IPv6 is available in the network
and you have a dual-stack host, we use IPv6 95% of the time.

LC: Maybe I'm just misreading graph. Why is WiFi used only 14% of time if there
is 39% IPv6 on US WiFi?

TP: I imagine it's because a lot of our connections are to things that don't
have dual-stacked service.

LC: So the 15% difference is the content factor.

TP: Yes.

Michael Tuexen: What is the RTT of a TCP connection?

TP: This is smoothed average. At the end of the connection we look at smoothed
average of what TCP saw during lifetime of connection.

Geoff Huston: You're not measuring IPv6 and IPv4 to the same endpoint. You're
measuring IPv6 when happy eyeballs says use IPv6, IPv4 when happy eyeballs says
use IPv4 and IPv4 when there is no IPv6 to use, right?

TP: Yes, lots of biases in here.

GH: I get a very different answer when I look at one endpoint and IPv6 and IPv4
to the same endpoint. If you changed happy eyeballs timers would you change
this profile?

TP: We're including all IPv4 connections, some of which not eligible for happy
eyeballs. It's just trying to get overall picture.

GH: You're comparing oranges and mandarins!

TP: Yes.

## Update on client adoption for both TLS SNI and IPv6
Erik Nygen

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-update-on-tls-sni-and-ipv6-client-adoption-01

## On the use of TCP's Initial Congestion Window in IPv4 and by Content
Delivery Networks Jan Rüth (JR)

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-on-the-use-of-tcps-initial-congestion-window-in-ipv4-and-by-content-delivery-networks-00

Bob Briscoe: There was a presentation at a recent IETF that measured ECN but
also incidnetally measured IW10 - got similar results - didn't go into as much
detail - that study showed some measurement results for mobile networks - I can
post URL on the mailing list.

JR: Thanks.

Stephen Strowes: I'm looking forward to seeing your IPv6 results! CDNs do
pretty aggressive MSS clamping - interested to see how that affects your
results.

JR: We were wondering how CDNs were delivering their initial windows - some of
them seem to pace their data, not all.

## A First Look at QUIC in the Wild
Jan Rüth (JR)

https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-a-first-look-at-quic-in-the-wild-00

DP: This is an upcoming publication next week at PAM so you're some of the
first to see these results.

Dmitri Tikhonov: LiteSpeed is spelled L I T E.

MK: There is a question in jabber about the availability of data? Your own
measurements might be available or shareable?

JR: Some measurement data is available - IXP and ISP traces are available -
MAWI data is available from their website, we can't publish TLD data because
domain lists are under NDA, https://quic.netray.io - IPv4 weekly scans run on
Friday, data available on Saturdays.

## Adoption, Human Perception, and Performance of HTTP/2 Server Push
Torsten Zimmermann (TZ)

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-adoption-human-perception-and-performance-of-http2-server-push-00

DP: How do you translate from an Alexa1M domain name to a CDN to determine who
is hosting content?

TZ: Started with IPs, mapped to AS numbers, also doing regular DNS
measurements, and analyse DNS chain CNAMES and identifiers - this could be an
incomplete list.

DP: Hint - using passive DNS can tell you the collection of FQDNs below a name
- there are many Alexa services that use many CDNs simultaneously.

Erik Nygren: Especiallly as more sites move to SNI the zmap IPv4 scan is going
to become decreasingly useful because many servers will behave differently
depending on which SNI they see. IPv6 deployment growth is also a
consideration. Alexa very much focuses on www style sites but increasing amount
of content is on separate domains for images, videos etc. that don't show up as
well on that list. When looking at push performance behaviour - you show that
there are different providers providing push - is that common across the board
or did some providers have behaviours that were generally negative while others
were more generally positive?

TZ: We couldn't attritbute that to the provider itself, it's more like the
configuration of the website so the user of the website. Push requires manual
configuration, but we also see some sites that use plugins, like WordPress
plugins, scan the file tree, observe static files to do that configuration

EN: There are starting to be product features from various vendors that analyse
site behaviour and then modify push behaviour. You may see cases where that
kind of closed-loop analysis of what to push has very different beaviours than
more static configurations.

Ian Swett: Thanks for publishing this work - I've long been looking for data on
the benefits or otherwise of push - it's hard to get good data and this is the
best I've seen - it makes clear the challenges of making push work in the wild.
Have you considered evaluating hash digest and whether that makes things much
better?

TZ: We will look into that. There will be a talk at httpbis meeting today on
hash digest - there's still a lot that can wrong with server push - it's a cool
feature but maybe standardised too early - without hash digest it shouldn't be
used.

## Inferring BGP Blackholing Activity in the Internet
Georgios Smaragdakis (GS)

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-inferring-bgp-blackholing-in-the-internet-00

Alexander Azimov: Unfortunately blackholing is a service - it's not only DDOS
mitigation - it's also a service for censorship - especially in Russia it is
used to block some resources and it is quite popular.

GS: I didn't want to say that, you said that, fine. Some of the long-lived ones
are candidates for censorship - we have enough indications for that but we
don't have ground truth - if you see days of blackholing and also we use other
analysis we can find that these are for websites of political content.

AA: It looks like blackholing plus hijacks.

## Deploying MDA Traceroute on RIPE Atlas Probes
Kevin Vermeulen (KV)

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-deploying-mda-traceroute-on-ripe-atlas-probes-kevin-vermeulen-00

DP: Interesting to contrast what you've found with this breadth of paths versus
what we usually call the diameter of Internet. We usually see a maximum length.
Is this terminology with diamonds something new or is there some existing
literature about this?

KV: There is a paper from Transactions on Networking that conducted a survey 7
years ago, it defined length and width - we have added symmetry and meshing for
our heuristics these metrics are important.

Kyle Rose: Could these techniques be used by an attacker to discover weak
points in a network? A wide node could mean something for example. Could you
identify a specific load-balanced node and try to take it down because you've
identified it as a bottle-neck for example.

KV: I'm not into security, but maybe, i suppose so - the goal here was to make
a nice map so by merging all the traceroutes that we get indeed we see the core
of the network which may be useful to an attacker.

Tim Chown: I regularly use a package called perfsonar to measure loss, latency
and throuput between a large number of sites. It's trace task has no idea when
something changes where the change occurred. It's nice to have an algorithm
that helps distinguish between the endpoints, endpoint networks and the network
in-between.

KV: The heuristics don't take into account where the load-balancing is taking
place but they could - it's a question that we've considered.

TC: Will follow up by email.

## An endhost-centric approach to detect network performance problems
Olivier Tilmans (OT)

Slides:
https://datatracker.ietf.org/meeting/101/materials/slides-101-maprg-an-endhost-centric-approach-to-detect-network-performance-problems-00

Tim Chown: Is code available?

OT: If you want it, I can give it to you, but it's not yet published - I am
open to deploy elsewhere and compare results. Sharing dataset has privacy
implications for the students.

Lorenzo Colliti: Are you using cgroup eBPF filters?

OT: I am putting a tracing mechanism in the kernel attached to

LC: So you see socket calls not packets?

OT: I'm intercepting kprobes -

LC: What does this give you for quic? It shouldn't give you anything.

OT: You can do this in user space as well.

LC: You have to do something like either ld preload or whatever to attach

OT: Not even actually - we can set up the kernel such that any userspace or
kernel space call matching some symbol will have

LC: So your implementation only works for a given version of the quic binary?

OT: Yes that is a challenge we have. We know that students for example have
Chrmomium - if we want to move to another implementation we have to make it
compatible - need to define a better way/generic way

LC: It could break between chromium version 65 and 66.

OT: yes but we own machine so we own the version - we control everything

LC: Alright, thankyou.

Neal Cardwell: Wanted to offer a conjecture on some of the mysterious delay
spikes you're seeing. Some of the TCP RTTs you're seeing are suspiciously
similar to delayed ack values on common operating systems (200ms and 40ms). So
something to look at.

Theresa Enghardt: Where can i find your code?

OT: I can share privately but it's not published yet. If you want to
collaborate drop me an email.

## Closing remarks

DP: You may have noticed a difference between the last two talks and the
remainder of the agenda. This was deliberate to carve out space to have talks
about tooling and measurement strategies. In maprg we want to bring insight to
engineering and operation of protocols. We suggest that you can only come and
talk about a tool if you have a novel measurement for us.

MK: Thanks all for being on time and please give us feedback if there were
talks you liked, didn't like, or things you'd like to see in future.

Abstracts
=========

Zesplot in five minutes - An attempt to visualise IPv6 address space (Luuk
Hendriks) Visualising IPv6 address space is a challenging exercise. While
approaches based on Hilbert curves have proven to be useful in the IPv4 space,
they end up producing uselessly large visualisations when applied to the IPv6
space. Inspired by the IPv6 Hackathon organized by the RIPE NCC in November
2017, our experimental tool Zesplot is an attempt to apply the idea of
so-called squarified treemaps [0] on IPv6 prefixes and addresses. Zesplot
produces plots based on two inputs: a list of prefixes, and a list of
addresses. The list of IPv6 prefixes is used to display squares, where the size
of the square reflects the size of the prefix. Then, the list of IPv6 addresses
is used to determine the colour of said squares: the more addresses are within
a certain prefix, the brighter that square is coloured. Thus, one can easily
spot outliers in an input set: a small but bright square for example, means
many 'hits' from a small prefix. Example use cases are visualisation of access
logs of e.g. webservers, origin of spam mail, or gaining insights in
measurement results for anything related to IPv6. Another possible use case is
education or address planning, where one can directly see the impact of
splitting up a prefix in different ways. Currently, Zesplot outputs to SVG with
an HTML/JS wrapper, allowing for zooming in/out on the plot, and providing
additional info (think ASN, number of addresses per prefix) while hovering the
squares. We are eager to learn what use cases are most useful for people, both
operators and researchers, to determine the direction for Zesplot. A first
version of the tool should be available under a permissive open source license
soon. [0] https://www.win.tue.nl/~vanwijk/stm.pdf

Measuring the quality DNSSEC deployment (Roland van Rijswijk-Deji)
In 2017 we performed two extensive studies of the DNSSEC ecosystem using
longitudinal data collected by the OpenINTEL active DNS measurement system
(https://openintel.nl/). Both studies focused on the quality of DNSSEC
deployments. In other words: if organisations bother to deploy DNSSEC, do they
deploy it in a secure way? We find that in generic TLDs, DNSSEC deployment is
low (1%). Fortunately, that 1% does mostly get it right; "real" errors in
DNSSEC deployment are rare. When we zoom in on two ccTLDs that have
incentivized DNSSEC deployment (.nl and .se), the picture is a bit more grim.
While errors are rare, deployments seldom follow best practices, leading to
potentially insecure DNSSEC deployment.

Update on client adoption for both TLS SNI and IPv6 (Erik Nygen)
With the exhaustion of IPv4, the multi-tenancy enabled by TLS SNI is critical
to supporting the rapid adoption of HTTPS.  Over the past few years, TLS SNI
has gone from having insufficient adoption for being generally useful to being
viable in a majority of cases.  IPv6 can also help here by not being
address-limited and has also seen solid growth in many countries.  Akamai has
been closely tracking global adoption of both IPv6 and TLS SNI (and taking
steps to influence both) over the past few years. This talk will provide an
update on where the world is with end-user and client adoption for both TLS SNI
and IPv6, based on traffic statistics being collected from Akamai traffic
delivery. We will highlight both leaders and laggards, looking at areas that
can have the most leverage for increasing global adoption of both.

On the use of TCP's Initial Congestion Window in IPv4 and by Content Delivery
Networks (Jan Rüth) Paper “Large-Scale Scanning of TCP’s Initial Window”:
https://conferences.sigcomm.org/imc/2017/papers/imc17-final43.pdf Improving web
performance is fueling the debate over sizing TCP's initial congestion window
(IW). This debate yielded several RFC updates to recommended IW sizes, e.g., an
increase to IW10 in 2010. The current adoption of IW recommendations is,
however, unknown. First, we conduct large-scale measurements covering the
entire IPv4 space inferring the IW distribution size by probing HTTP and HTTPS.
We find that many relevant systems have followed the recommendation of IW10,
yet a large body of legacy systems is still holding on to past standards.
Second, to understand if standardization and research perspective still meet
Internet reality, we further study the IW configurations of major Content
Delivery Networks (CDNs) as known adaptors of performance optimizations. Our
study makes use of a globally distributed infrastructure of VPNs giving access
to residential access links that enable to shed light on network dependent
configurations. We observe that most CDNs are well aware of the IW's impact and
find a high amount of customization that is beyond current Internet standards.
Further, we find CDNs that utilize different IWs for different customers and
content while others resort to fixed values. We find various initial window
configurations, most below 50 segments yet with exceptions of up to 100
segments — the tenfold of current standards. Our study highlights that Internet
reality drifted away from recommended practices and thus updates are required.

A First Look at QUIC in the Wild (Jan Rüth)
Paper (author's version): https://arxiv.org/abs/1801.05168 For the first time
since the establishment of TCP and UDP, the Internet transport layer is subject
to a major change by the introduction of QUIC. Initiated by Google in 2012,
QUIC provides a reliable, connection-oriented low-latency and fully encrypted
transport. We provide the first broad assessment of QUIC usage in the wild. We
are monitoring the entire IPv4 address space since August 2016 and about 46% of
the DNS namespace to detected QUIC-capable infrastructures. As of October 2017
our measurements show that the number of QUIC-capable IPs has more than tripled
since then to over 617.59K. We find around 161K domains hosted on QUIC-enabled
infrastructure, but only 15K of them present valid certificates over QUIC. We
publish up to date data through: https://quic.comsys.rwth-aachen.de. Second, we
analyze over one year of traffic traces provided by MAWI, one day of a major
European tier-1 ISP and from a large IXP to understand the dominance of QUIC in
the Internet traffic mix. We find QUIC to account for 2.6% to 9.1% of the
current Internet traffic, depending on the vantage point. This share is
dominated by Google pushing up to 42.1% of its traffic via QUIC.

Adoption, Human Perception, and Performance of HTTP/2 Server Push (Torsten
Zimmerman) The web is current subject to a major protocol shift with the
transition to HTTP/2, that overcomes limitations of HTTP/1. For instance, it
now is a binary protocol that enables request-response multiplexing and
introduces Server Push as a new request model. While Push is regarded as key
feature to speed-up the web by saving unnecessary round-trips, the IETF
standard does not define its usage, i.e., what to push when. The goal of our
work is to inform standardization with an up-to-date picture on i) its current
usage, ii) its influence on user perception, and iii) optimization potential.
Our Push usage assessment is based on large-scale measurements [1] covering the
IPv4 and the complete set of .com/.net/.org domains. We regularly report our
results at https://push.comsys.rwth-aachen.de. We find both the HTTP/2 and the
Push adoption to steadily increase, yet Push usage is orders of magnitudes
lower than HTTP/2, highlighting its complexity to use (e.g., 220K domains on
the Alexa 1M support HTTP/2 and only 932 Push). Second, our performance
evaluation of Push enabled sites shows that Push can both speed-up and
slow-down the web [1][2]. These detrimental effects cannot be simply attribute
to simple factors like type, size, or fraction of pushed objects, again
highlighting the complexity to use push correctly. We assessed if these effects
are user perceivable in a user study [2], i.e., to assess if current
engineering and standardization efforts are indeed sufficient to optimize the
Web. Server Push can yield human-perceivable improvements, but also lead to
impairments. Notably, these effects are highly website specific and indicate
that finding a generic strategy is challenging. Our ongoing work studies how to
better use push. We thus thoroughly analyze Push performance impacts in a
controlled and isolated testbed. Based on these results and the previous
contributions, we investigate a novel approach to realize Server Push,
incorporating website specific knowledge and client-side aspects, that can lead
to improvements for some websites. We believe that our work can help to
understand how standardized features are applied in the wild and what are the
resulting consequences.

[1]
https://www.comsys.rwth-aachen.de/fileadmin/papers/2017/2017-zimmermann-networking-push.pdf
[2]
https://www.comsys.rwth-aachen.de/fileadmin/papers/2017/2017-zimmermann-internetqoe-push.pdf

Inferring BGP Blackholing Activity in the Internet (Georgios Smaragdakis)
The Border Gateway Protocol (BGP) has been used for decades as the de facto
protocol to exchange reachability information among networks in the Internet.
However, little is known about how this protocol is used to restrict
reachability to selected destinations, e.g., that are under attack. While such
a feature, BGP blackholing, has been available for some time, we lack a
systematic study of its Internet-wide adoption, practices, and network
efficacy, as well as the profile of blackholed destinations. In this
presentation we describe how we develop and evaluate a methodology to
automatically detect BGP blackholing activity in the wild. We apply our method
to both public and private BGP datasets. We find that hundreds of networks,
including large transit providers, as well as about 50 Internet exchange points
(IXPs) offer blackholing service to their customers, peers, and members.
Between 2014-2017, the number of blackholed prefixes increased by a factor of
6, peaking at 5K concurrently blackholed prefixes by up to 400 Autonomous
Systems. We assess the effect of blackholing on the data plane using both
targeted active measurements as well as passive datasets, finding that
blackholing is indeed highly effective in dropping traffic before it reaches
its destination, though it also discards legitimate traffic. We augment our
findings with an analysis of the target IP addresses of blackholing. We also
show that BGP blackholing correlates with periods of high activity of DDoS
attacks. Our tools and insights are relevant for operators considering offering
or using BGP blackholing services as well as for researchers studying DDoS
mitigation in the Internet.

Deploying MDA Traceroute on RIPE Atlas Probes (Kevin Vermeulen)
Traceroute is widely used by network operators, to troubleshoot, and by
scientists, to understand the topology of the internet. In the presence of load
balancing, classic traceroute can lead to errors and misinterpretations, and
these have been corrected by the widely used Paris Traceroute, which we have
developed and maintain. Paris Traceroute's Multipath Detection Algorithm (MDA)
allows a user to discover the load balanced paths between a source and a
destination, with configurable statistical guarantees on the completeness of
the results. The more complex these topologies are, the more packets the MDA
requires in order to provide the guarantees. For a single route trace, the
numbers can run into the thousands of packets, and even tens of thousands. In a
resource constrained environment, such as a RIPE Atlas probe, this is too
costly. We have made an empirical study of the patterns in which load balancers
tend to reveal themselves in route traces, and we are using the results to
implement and deploy a new MDA, which promises to significantly reduce the
number of probe packets required to discover complete multipath routes. We
describe ongoing work that has been presented last week at CAIDA's AIMS
workshop. In order to reduce the number of probe packets required to discover
complete multipath routes, we have done a survey on load balancers, consisting
in probing 350000 ip destinations from 35 planet lab node as sources, and
extract some metrics. This talk will mainly focus on the results we have found
in the survey, and the metrics we have extracted to classify the load-balancers
we have found..

An endhost-centric approach to detect network performance problems (Olivier
Tilmans and Olivier Bonaventure) As enterprises increasingly rely on cloud
services, their networks become a vital part of their daily operations. Many
enterprise networks use passive measurements techniques and tools, such as
Netflow. However, these do not allow to estimate Key Performance Indicators
(KPIs) of connections, for example losses or delays. Although monitoring
functions on routers or middleboxes can be convenient from a deployment
viewpoint, they miss a lot of information about performance problems as they
need to infer the state of each connection and they will become less and less
useful as encrypted protocols are getting deployed (e.g., QUIC encrypts
transport headers). It is time to revisit the classical approaches to network
monitoring and exploit the information available on the end hosts. In this
talk, we propose a new monitoring framework where monitoring daemons directly
instrument end-hosts and export KPIs about the different transport protocols
towards an IPFIX collector. More specifically, our monitoring daemons insert at
runtime lightweight probes in the native transport stacks (e.g., the Linux
kernel TCP stack, libc’s name resolution routines, QUIC implementations) to
extract general statistics from the state maintained for each connection. An
aggregation daemon analyzes these statistics to detect events (e.g., connection
established, RTOs, reordering) and exports KPIs towards an IPFIX collector. We
will present a prototype deployment of these monitoring daemons in a campus
network, and discuss early measurement results.