Jepsen Radix Report discussion

The Jepsen analysis of Radix was published today at:
https://jepsen.io/analyses/radix-dlt-1.0-beta.35.1

Some parts of the report are uncomfortable to read.

What are your thoughts on the report and RDX Works response?

3 Likes

I’m glad they published it as they didn’t have to, it shows a great degree of transparency. The wording of the report comes across very damning in parts, but the new Gateway service hopefully resolves most of the issues highlighted. It would be great for this to be verified by Jepsen in future.

I think Radix’s response was good at adding some context around the findings, but some more clarity would be useful, especially around for example the throughput tests which were found to be much less than the 50tps we’re expecting from Olympia.

As for the issue highlighted in the case of a single validator going down, that was all too evident in the past week, so I hope the proposed solution that the team have alluded to will be coming sooner rather than later.

1 Like

It’s indeed a bit uncomfortable to read. Seems almost as if there was a bit of friction between the team and Jepsen. The report has some completely unnecessary sarcastic undertone to it. I haven’t read other reports by them, so don’t know if that’s normal.

Anyway, are we really surprised on the findings? There nothing catastrophic in there, right?

The technical outcome of the report was that the old Archive architecture was not fit for purpose. Radix responded by prioritising its replacement and most of the concerns have been addressed by the 1.1.0 release. It’s great to see that the team are re-using the test harness that Jepsen used and have recreated tests to address the concerns in the report.

Unfortunately the report was overshadowed by tearing apart some of the marketing claims that are made by Radix. I don’t think the tone was necessarily sarcastic - just lots of exclamation marks expressing incredulity at claims made versus observed results. It’s natural to be defensive but it is really not a good look to be asking to remove unflattering sections of the report and I respect Jepsen hugely for not doing so. The statement that “blockchain/DLT readers would normally understand present-tense English statements like these to be referring to potential future behavior, rather than the present” deserves the mockery it received on Hacker News.

I think it took real courage by Radix to engage Jepsen to do a report. You don’t hire them for a soft, warm and fuzzy analysis for marketing purposes. Their analysis is the best in class and its findings are closely scrutinised. I genuinely believe that the Radix system will be all the better for having done so.

5 Likes

The request to remove content was specific to using the terms ‘Liveness Break’ and ‘Safety Break’ in the ‘Introductory Abstract’ section of the report until they had been presented later in the report with a comparison between the definition in a DLT world versus the academic definition. Not to remove any findings related to these points.

2 Likes

To Jepsen’s surprise, RDX Works asserted that phenomena such as aborted read, intermediate read, and lost writes do not constitute safety violations (in the DLT sense). RDX Works claims that to describe these errors as safety violations would not be understood by readers from a DLT background; this report is therefore “factually incorrect”. On these grounds, RDX Works requested that Jepsen delete any mention of our findings from the abstract of this report.

Jepsen respectfully declines to do so.

Easy to miss the word “abstract” I suppose. Still not a good look though.

1 Like

Not ideal, just badly worded I guess.

The tricky thing about these reports is that you can spin them two ways: “Radix has fixed the issues, the most important stuff worked”, or “See? So many issues… Told you Radix sucks!”

Something like the supposed double-spend issue doesn’t help. In the end, it was, again, an API issue (and no on-ledger double-spend) from what I’m reading, but it’s really easy to grab that info and create FUD, unfortunately.

Either way, I’m glad they got someone who tried to break what they built. I like that :slight_smile:

Someone also linked to these ‘tweets’ or whatever they are from Jepsen’s site:

6 Likes