Hetzner Terms of Service prohibits running any kind of crypto node

Using our products for any application related to mining, even remotely related, is not permitted. This includes Ethereum. It includes proof of stake and proof of work and related applications. It includes trading. It is true for all of our products, except colocation. Even if you just run one node, we consider it a violation of our ToS. We are aware that there are many Ethereum users currently at Hetzner, and we have been internally discussing how we can best address this issue.

You can find the full statement here and an article explaining more here.

Currently, 25.30% of total delegated stake on Radix is hosted on Hetzner nodes.

The situation for other cryptos isn’t that different, with Ethereum at a 8.8%:

Solana 35.8%:

Avalanche 12.2%:


My thoughts:
Hetzner doesn’t really know what’s the difference between staking/PoS and mining/PoW is, so they’re just making this blanket ban about anything related to cryptocurrency, just to be sure no one is abusing their resources (which now, thanks to the increasing energy prices, are even more expensive).

8.3 of their ToS:

The operation of applications for mining cryptocurrencies remains prohibited. These include, but are not limited to, mining, farming and plotting of cryptocurrencies. We are entitled to lock the Customer’s access to their Hetzner services or account in the event of non-compliance.

It is not clear what they mean by that. Is even a simple gateway node prohibited? It’s not much different than a database server. I imagine that soon Hetzner will clarify what they meant, since they’re now getting a lot of attention and not the good kind.

As a node runner, this concerns me but not by a lot, since I self host my own node from home using my FTTH connection (you can learn more about my infrastructure here: Introducing 🍕 MattiaNode).
I use Hetzner’s dedicated servers as backup nodes but I also have another self hosted node elsewhere, with independent power and connectivity from my primary.

I will consider moving away from Hetzner soon unless they give a proper statement about what they meant and what it means for PoS coins and nodes.

Self hosting comes with some disadvantages, which may translate in losing some proposals, but I’d rather lose a few proposals but improve decentralization and the safety of the whole Radix network. With the exception of some maintenance or some particular issue, my uptime is usually 100%.


Does that mean that 25% of Radix validators are soon illegal because they run on Hetzner?

No, running a validator isn’t illegal, but it may be against their ToS. Like I said though, it would be highly improbable for Hetzner to bring down all nodes at the same time. If they were to do so, not only would they lose lots of customers but they would also bring down some and slow down other cryptocurrencies that use their services.

In the highly improbable scenario where Hetzner would drop all Radix nodes at the same time from its network, what would be the impact on the Radix network?

If that were to happen, there would be a very noticeable impact on the whole Radix network: we’ve seen a similar but lighter scenario with the Viskosity incident, where a top 10 node started missing all proposals for some days. Epochs became much longer, from 40 minutes to 1 hour~ and because of that, the APY from staking dropped significantly.

Losing 25% of total stake will have a much worse impact on epoch length. As a comparison, it would be like losing almost all top 10 nodes instead of just one, like it happened with Viskosity, since the top 10 nodes hold 28.47% of the total stake:

image

Will there be an impact on the Radix network if node runners decide to move away from Hetzner?

The impact will likely be minimal, if any. Node runners are used to switching between backup and main nodes without losing proposal, especially if not in the top 10 (more time between every proposal for the failover).

From @Faraz:

Stakers don’t need to do anything if a validator moves their node. We moved out of Hetzner when it was first discovered that 25% of Radix validators were hosted there. It can be done without even missing a single proposal (depending on validator stake weight) and even in a worst case scenario, a validator simply unregisters for an epoch (a small price to pay for stakers to miss one epoch of rewards in the interest of decentralisation and security).

How can I find out where each validator is located geographically and what provider it’s using?

You can use the great tool from the Stakesafe team: https://validators.stakesafe.net/


What do you guys think about this? I’d like to hear from both node runners and delegators.
Would you consider moving your stake to a decentralized/self-hosted node after this incident? Or do you prefer trying to maximize your uptime delegating to nodes hosted on major providers like AWS/Hetzner?

And to my fellow node runners, if you’re using Hetzner, what are you planning to do? Will you move your nodes elsewhere or wait and see what happens?


Sources:

4 Likes

Great post Mattia, very comprehensive. My hope is that this encourages a few more node runners to move their nodes and improve decentralisation, as the situation wasn’t great even before this announcement.

Of course that too comes with the risk that it increases centralisation to other cloud providers such as AWS, but thankfully the dialogue between node runners is normally very good and it would be nice to see this become a coordinated effort to improve decentralisation of the network.

In reality, I think Hetzner’s TOS are a bit too broad, and they will probably have to clarify them in the coming weeks before they could enforce anything.

1 Like

I think they over react without understanding properly what POS does of the fear of backlash from the Media to position them in a bad light as a host. And the media is full of sh*t to be honest, thinking it knows better what’s best or just wants clicks.

Either way is a good way to decentralize.
We aim also to move from Amazon as they are crazy expensive for what they do.

Wonder if other hosts have such TOS.

1 Like

Hey, completely agree with the decentralisation. Spread the nodes over many providers to limit the chance of whats currently happening now with the likes of ETH, AVAX, SOL having a huge % vendor tied!

If your looking for a supportive web3 hosting provider, take a look at servers.com, we believe in the web3 future and the decentralised hosting model. We are global bare metal infrastructure provider operating in a cloud like fashion.

Reach out to me if you would like to have any discussions
nathan@servers.com

Hi Nathan, welcome to RadixTalk!
I gave a quick look to your website and I’m impressed by the stats! (Up to) 40 Gbps per server and low latency are a big plus for PoS nodes.

Traffic seems to be limited to 20002 GB which could become an issue in the future. Also, just to be clear, your ToS completely allows running cryptocurrency nodes? If yes, does that include PoW mining? How do you deal with the energy costs in those cases?

Thanks and welcome again.


A week after this post, the total stake delegated on Hetzner nodes dropped by 3% to 22.30%, thanks to @Radnode.io switching his node to OVH. It’s still a worryingly high number though.

I also moved my nodes backup servers away from Hetzner.

3 Likes

40 Gbps is phenomenal - I notice that you provide private racks, so am also curious; please give us further info about your infra and outlook :call_me_hand:

1 Like

Hi Mattia, thanks for the reply. Yeah, we feel that our set-up very much suits this industry!

So the 20002GB traffic is what’s included with each server for free. We then charge 0.03$ per GB after that, however, we can also agree on a fixed price for anything over the 20002GB should you feel you will exceed.

Yes our ToS allows cryptocurrency nodes, please see our most recent blog on our web3 support:

(apologies for the link, please feel free to go to servers.com website, navigate to news and then blog to find this post if you don’t want to follow the link)

We accommodate mining, including PoW. All costs for our customers are included in our offering price, which may be why we are slightly more expensive than others. So with mining, power consumption is not something our customers need to worry about.

If you would like to have further discussions around our offerings or how we could assist, let me know :slight_smile:

Thanks Mattia,

1 Like

Thanks for checking out our website. We do private racks, what would you like to know exactly?

Would you be open for a call to have a quick chat so we can give you all the info you need?

Just checked the location of my validators, and I am good to go. :grinning:

The easy way is to use the Radixdltdashboard.com along with the link provided by the OP to verify the actual identity and location of the hosting provider.

Hi Loid, do you mean https://www.radixdashboard.com/? Did they also add location to their list?

No just used the validator name and compared it to the listed site above. Yes the link you gave is the correct one.

1 Like

They actually went through with it and blocked 1000 Solana validators!
https://www.theblock.co/post/182283/1000-solana-validators-go-offline-as-hetzner-blocks-server-access

Radix validators hosted on Hetzner seem to be fine. It looks like we’re under the radar as of now.

IMHO, Hetzner with their 25% + of the XRD stake is a major issue that actually needs to be resolved. When Hetzner takes down or blocks the hosted Radix validator nodes, what impact do you think it will have on the Radix Public Network (RPN). We saw what happened when Allnode validator, which was #1 in staked XRD, and that was only about 120M tokens.

Hetzner has 25% of the 3.218 billion XRD tokens staked. This is IMO a significant security issue that needs to be addressed by the Radix Community.

As of Jan. 2, 2023 stakesafe.net reports that Hetzner now controls 13.5% of the total Radix stake. That is still too large but if most of the validators using Hetzner have standby nodes with other hosts them the existential threat to the network is minimal. Still I have made sure that the 10 validators that I support don’t use Hetzner and I hope that other Radix stakers also check the hosts used by their validators using the stakesafe.net website.

Amazon is now at the top with 20.34%. It looks like the problem is still there, it just changed its name :laughing:

While Amazon does not accept cryptocurrency payments, they have hired cryptocurrency programmers at times suggesting that they have looked into it. They restrict mining apps on the Amazon App store but AWS has never shown any indication that they will restrict mining or validation on their platform.

Every major crypto seems to use AWS to some degree. A 20% deployment rate might be on the low side compared with other L1s.

Worth keeping an eye on, but this is not a current threat to the Radix mainnet. If AWS utilization we’re to spike up past 25%, then I think it would be appropriate to raise some alarm bells about it.

1 Like