Dear GetBlock customers,
For the first time in its history, an unknown attacker of GetBlock, a leading blockchain nodes provider, managed to seriously affect its operations.
As we’re committed to maintaining an open dialogue with our community, the time has come to share with you what happened with GetBlock - and what is next for us.
TL;DR Malicious actor exploited the third-party tool vulnerability and accessed a number of servers our nodes were using. They were privately demanding us to pay a ransom in cryptocurrency to withdraw the threat to delete all our data.
To protect users, we immediately switched all of our service to ‘rescue’ mode: it took about an hour. At the same time, as we employ one of the largest server networks in the blockchain API providers segment (hundreds of hosts in various regions of the world) in the industry, restoring the status of operations takes time.
No ransom was paid. No sensitive data was leaked.
We’re deeply humbled by the support we received in handling this event and would like to sincerely apologize for any inconvenience caused by GetBlock’s downtime.
What happened?
On November 13, 2022 at around 01:30 a.m. (hereinafter - UTC timezone) our service was attacked. Hackers managed to exploit the vulnerability of the third party tool.
We received a message from the attackers via our contact form. They confirmed that they gained access to all our servers and were ready to delete the data from them.
Our DevOps and Customer Support teams are working 24/7 from various locations across the globe. That’s why we stayed in touch with our customers. GetBlock’s DevOps team managed to provide security for all our customers in an hour.
We immediately activated a ‘kill switch’ mechanism to protect our customers: all servers were put in ‘rescue mode’.
To guarantee the sustainability and security of our infrastructure, we needed to eliminate any possibility for effective backdoors. Simply put, we needed to reinstall the OS on all our hosts without losing data. As our infrastructure includes hundreds of hosts all over the globe, this procedure caused a downtime for our users. We started this process immediately and were delivering recovered nodes to our clients step by step.
Attack timeline
To show the whole picture, let’s track the timeline of this unfortunate event we’re getting through together:
Nov. 13, 2022:
- 01:30-02:00 a.m. - GetBlock team notices that the node services are down and receives the message from the attackers;
- 02:00-03:00 a.m. - the system is in rescue mode, GetBlock team is researching the attack vector. As a result, the safety hazards are eliminated.
- 03:00-11:00 a.m. - the analysis of attack details is finished: team estimates the effect it has on GetBlock’s infrastructure. Engineers started updating the security protocols.
- 11:00 a.m. - 04:00 p.m. - Team is developing a methodology of recovery of all our hosts with minimal downtime in focus. As we support 50+ blockchains and the accessibility of every blockchain is our #1 priority, this took time and resources.
Nov. 13-15:
- entire re-configuration of the infrastructure with new data credentials;
- implementation of a new architecture of service designed to remove the threat of safety and stability in future;
- implementation of updated security protocols is in progress; the urgent measures are already taken.
Nov.16:
restoring the operability of the service. As our infrastructure is massive, this process requires time.
Since the very first days of GetBlock, we adhered to the security policies that eliminated storage and processing of sensitive data.
As such, no personal information of our users is in danger. The downtime is the only negative effect of the recent attack for end users.
Rising again, stronger than ever
First of all, we are ready to introduce new security practices, in particular - for the operations with any third-party tooling. Not only will they protect our customers from malefactors, but also set new standards in security and privacy for the nascent BaaS segment.
Also, next week, we’re going to implement a series of updates to ensure even higher availability for all our users and stability of shared nodes. The updates are designed to allow GetBlock’s tech architecture to handle the larger workload.
What does this mean for our customers?
First of all, it’s about attack-resistance. With upgraded tech infrastructure, GetBlock will be protected from the most sophisticated attack vectors. As the markets are approaching the next bull run, this feature will be of paramount importance for dApps progress.
Also, November 2022 updates will advance sustainability and availability of our services.
Closing thoughts
While GetBlock has been through two dark days, this is a learning experience for all of us. Attacks happen, and there’s never a dull day in crypto! Attacks on third-party services are among the most dangerous hacks designs: typically ‘targets’ have nothing to do with the flaws that made this or that attack possible.
Recently, the team of 3 Commas trading bot mitigated the attack that leveraged Binance and FTX API instruments: someone managed to use them in a fake copy of trading bot app. The Solana blockchain was also attacked by hackers who exploited a third-party services vulnerability.
For infrastructure services, the situation is even more dangerous as thousands of applications are relying on them: let’s recall recent Cloudflare and AWS outages.
GetBlock is going to maintain the highest possible standards in security and cooperation with its customers. Right now, our infosec engineers are working heads down on implementing a brand-new, more robust security policies to remove the possibility of further issues. All layers of our ecosystem will be stress-tested and double-checked.
Also, we initiate a third-party audit of our tech design from a Tier-1 white-hat hackers team: stay tuned for the details.
P.S. GetBlock’s compensation initiative: Who is eligible?
We’re feeling for everyone affected by our outage. We appreciate your patience. We would like to say thank you to each and every one of those who helped us throughout our investigation.
For all of our customers, we’re launching a downtime compensation initiative. The eligibility tier depends on previous experience this or that team had with GetBlock:
- All paid tariff users will get free requests (package similar to the one that was purchased last in 90 days prior to the attack)
- All unlimited packages will get 10 free days;
- All dedicated nodes’ users will have the full downtime period compensated + $200 voucher (exp. Dec.31, 2022).
Please accept our sincere apologies for the downtime and any inconvenience for you, your businesses and customers. Together with your game-changing products, we will bounce back harder than ever and will continue building amazing Web3 stuff.
Sincerely yours,
GetBlock Team
