Both my seed and witness nodes are now running the Steem Blockchain v0.19.5. Not without a little bit of panic. It should have taken me less time to get back up and running but the update instructions didn't work for me straight away. The regular restart didn't seem to work and the witness stalled for too long so I had to do a replay.
What is a replay?
When setting up a new witness server, you need to somehow download a copy of the blockchain onto the server and index its content from day 1 into the memory. You have two ways to achieve this:
- either by performing a sync, where your witness node will fetch from the public seed nodes all the block from the beginning (2 years ago), one by one and reconstruct the blockchain into a file named
block_log - or download an existing copy of the
block_logfile and replaying it locally
The replay is much faster as it is reading from a local copy of the blockchain on disk and does not perform slow network requests. By the time you have downloaded the block log file, it is already out-of-date, so when the block_log has been replayed and indexed into memory, a sync will be performed to download the remaining blocks until it reaches the head block.
With the 32 GB of RAM I have on the server, it takes less than 24 hours to complete a replay. With 64 GB of RAM it should take only few hours. The reason is that currently, the blockchain data in memory takes over 40 GB of space. So the excess data has to be stored onto disk on what we call a SWAP. Disk access being slower than memory, this is why it takes more time when you have less memory.
Issues encountered
I initially thought my copy of the blockchain was corrupted because the process was stuck for a very long time on the witness node while the seed node was up and running very quickly. After some digging around I realised that I created my two swap files in the wrong order, the primary swap was set to the one on the slowest disk.
I also noticed that when restarting the process, I sometimes see the following error: Lock timeout, moving to lock 1. This is usually fixed by deleting /dev/shm/shared_memory.meta file and restarting the process. Although it did remove the error message for me, the process was still hanging for over 15 minutes.
So I decided to reshuffle the swap around. But you can't do this when you have data in memory and in the swap. So here is the process I did (I'm using @someguy123 Steem-in-a-box which uses Docker)
./run.sh stoppbzip2 -c < /dev/shm/shared_memory.bin > shared_memory.bin.bz2rm /dev/shm/shared_memory*- use swapoff and swapon to turn off/on the swap files in order to reorder them. I set the swap on the fastest disk to be the primary swap by turning it on first
pbzip2 -dc shared_memory.bin.bz2 > /dev/shm/shared_memory.bin./run.sh start
Here are some explanations:
- I first stop the steemd process so that it stops adding more data into memory and disk
- pbzip2 is a modified bzip2 that does parallel file compression much faster than the regular bzip2. I used it to compress the shared memory file from memory back to disk. It's faster than copying the file directly. And this can also be used to set up a new node by copying the compressed file from one server to another, that will allow you to start the witness server even faster than a replay (you will also need to copy the corresponding block_log file).
- once the compression has done (10-20 minutes if memory serves), I then deleted the shared_memory file and its metadata file from /dev/shm
- I then turned off all the swap files and turn them back on in order of priority
- I then used pbzip2 again to decompress the shared memory file back to memory
- and then restarted steemd
It seems to have worked, the process is starting up much faster:
And pretty fast started to receive blocks from other witnesses:
I have now re-enabled my witness and it is awaiting for a block to sign.
Summary of some of my other contributions
| Project | Description | Link |
|---|---|---|
| Steemed Phish | Chrome browser extension to make external links more visible and alert you on Phishing/Scamming links | Steemed Phish |
| Dtube Community Support | This project by @evildido allows users to run their own IPFS node and pin videos in order to cache them for longer. Dtube has the tendency of deleting non-popular video after few days only. I helped Dockerizing the project for an easier deployment on almost any platform. I've also contributed by running two nodes, one in Australia and one in Canada to cache #teamaustralia and #teamnz videos. | Dockerizing Dtube Community Support |
| DiscoSteem Bot | This is a Discord/Steem bot project by @planetenamek. When I started Steemulant Bot for the Vietnamese community, I didn't know the existence of DiscoSteem Bot and wanted something written in NodeJS. I've ended up continuing my own project as it contains code and logic for personal use and for the Vietnamese community. But I've decided to share some useful features that can benefit the users of DiscoSteem Bot and will continue to add more functionalities as I go. | Dockerizing DiscoSteem Bot |
| A Dollar A Day | This is a project run by @pennsif and will donate 15 SBD every day to a charity | "A dollar a day" fundraiser - Karaoke Singing: Le chanteur malheureux |
| Team Vietnam | I started a new Vietnamese community on Discord and developed @teamvn, a community bot to help grow the Vietnamese community and help them create higher quality content. |
Credits
- The image at the top has been generated with the Canva app using my own photo.
Related content
- Witness Update 2018-07-03 - The day the Steem blockchain froze
- Witness update 2018-06-17 - New witness server and a seed node
- Witness Update 2018-06-02 - A little witness success
- Witness Update 2018-05-25: @teamvn the Vietnamese community bot
- New Steem Witness announcement! Greetings from @quochuy
- My first votes for Witnesses
I don't follow for follow, I don't upvote for upvote. If you make quality posts that I like/enjoy then I will upvote and/or follow you