RASP Agenda 15:00-16:30 (IETF 119)

15:00 - 15:10 Chairs, welcome and agenda

15:10 - 15:30 Nick Sullivan, LLMs and RFCGPT: leveraging large language
model platforms to understand standards

Alexander Raeilean (Siemens): A good use of this would be to generate
text cases

Jay Daley: there are plans to turn the RFCs into a open vector database
and making that into an open resource for the community, would that be
helpful?
Nick: Yes, that would be helpful

Jay Daley: regarding fine tuning, are you looking for specific data?

Nick: with regard to fine tuning, it's not necessarily about questions
and answers as much as just training new data into the model. And
updating it with with new data without having to retrain the entire
model, which can be extremely expensive.

Marek Blachut: Regarding audience, who should be using this and who not?

Nick: This is a dangerous tool in the wrong hands, if you are not aware
of how the IETF works. This is for folks familiar with the IETF and need
a tool to explore the available data.

Jerome Francois: You mentioned RAG, I was wondering if the next step is
to measure qualities, what would be the difference from the RAG approach
to be contextualized in the model?

Nick: ChatGPT actually allows some of that, and there's lots you can do.

Carolina Caeiro: Was wondering if you can extend this to internet
drafts?
Nick: As a chair it would be useful to get a sense of the timelines of
drafts and get more context.

15:30 - 15:50 Carolina Caeiro, Internet Standards Tracker update

Have you planned for an API?
Carolina: Not really but we could do it.
Are you thinking of publishing a dictionary
Carolina: Woudl love to follow up on that

Jonathan Hoyland: How do you deal with mistakes in data?

Carolina: This is actually a good question. We don't do any manual work.
We scrape them and what we what the system sort of picks up, it shows on
the tracker and what it didn't or what it missed, it just doesn't. And
one of the the suggestions we received, in the ProgyBend was maybe
working with, the folks developing the data tracker. And, you know,
working with them in, sort of figuring out waste in which we can know
whether a given draft is being discussed at a meeting and sort of having
a more stable, way essentially often fairing that information straight
off the data tracker. Right now, because that information doesn't exist,
we sort of you know, how to, you know, we did this work around, I guess.
But it is indeed imperfect.

Jonathan: And is there a way for people to share corrections?

Carolina: Thank you for highlighitng this area for improvement.

15:50 - 16:10 Priyanka Sinha, Computationally understanding the IETF
consensus process

Jonathan Hoyland (Cloudflrare): Do you consider author as one of the key
factors?

Priyanka: Yes, we do experiments we we consider the author. For this
study we didn't distinguish.

Johathan: Group dynamic plays a factor?

Priyanka: Of course, there are internal factors and external factors
that might impact the results.

Susan: Did you consider the area of the working group? It would be
facinating to compare WG between different areas? Can you also share the
definition of consensus that you were using?

Priyanka: I totally agree.

16:10 - 16:30 Geoff Huston, Making of an RFC in today’s IETF

Lars: Have you filtered out the drafts that are with the IESG? the
context is that the drafts have presisting disagreements
Lars: None of the drafts that are outliers are IETF drafts

Richard Barnes: I noticed the decline in 2018 started with when github
started being used, so I wonder if the reason is that versioning is
happening there.

Geoff: It could well be

Susan Hares: I did a look at draft in details, IESG time matters.
Publication time varies, it varies for a variety of reasons. The IETF
process has a lot of ebb and flow. While the numbers are interesting,
you're getting a lot of false positives and false negatives.

Alistair Woodman: is this meant to be taken as a negative observation?

Geoff: Not at all, I had a negative experience, but I was looking at the
numbers and wanted to share.

Alistair: My observation is that it should get harder, because all the
low hanging fruit is gone.

Colin Perkins: Individual drafts that get adopted as working group
drafts as some parts of the conversion are not just drafts that
disappear. It gets adopted by a working group. Goes to the ISG name
changes a few times.

Alistair: It is great to show the average, it would be nice to show the
spread (poisson distribution) of the set.

Colin Perkins: Going back to the comment you made earlier, it would be
good to look at the number of discussions on the mailing list and
comments, and there is a clear trend of increasing complexity, there are
delays, but it's clear that they are increasing.

GeofF: I have no value judgement, I understand there are reasons, but
I'm not trying to bring them up in this talk.

Colin: it's a lot of work, and I think it's some unsurprising increase
as as the technology matures, as the network matures, backwards
compatibility, it it's harder to It's harder to produce an RFC than when
you're starting from scratch. We looked into this in a 2021 IMC paper.

Susan Hares: I looked at 10% of the RFCs, one of the reason that between
2002 and 2006 is that there is some weakness in the data, because of the
datatracker transition. I think the deep dives are useful, and it's
important to look at where the reasons are.

Andrew Campling: In six or 8 years we might want to question whether
things are obsolete, maybe we should give some KPIs to the

Jonathan Hoyland: Thank you for giving me hope that my RFCs might come
out some day.

Colin Perkins: To correct the record, it's not the IESG's fault, the
time is increasing, but it's not their fault. We think there are 3 to 4
years to release an IETF.