Note: this witness update will be technical.
In order to effectively maintain a server you need a monitoring system to constantly check its health by looking at various stats such as resource consumption (disk, memory, CPU, network, etc...) and system logs.
Steemd, the blockchain software, is outputting its logs to the standard output so it is required to redirect it to a file. In my setup, it is redirecting it to /home/quochuy/steemd.log and I'm rotating and compressing that log file on a daily basis, keeping them only for the last 7 days using logrotated.
This is how a regular steemd.log looks like during normal operation:
Sumo Logic
In order to easily check and analyse the log output of steemd and setup some automated alerts, I've decided to setup a data collector that sends it to Sumo Logic, a log analytics service I'm already using for other personal projects and also at work, it has a free tier which I think should be enough for processing witness logs at my current rank. The collector is a little script that will stream the logs to Sumo Logic which in turn will parse and store it in their database. I can then access their web interface to do queries, analysis and other alerts.
After an initial data ingestion here is what the logs look like in Sumo Logic:
I can then apply a logreduce filter to it and Sumo Logic will try to categorise each of the log lines and group them by similarities and display the number of time they have occurred. This is very useful to reduce the noise and easily find an issue. In the screenshot below, you can see that Sumo Logic has managed to group block production logs by witness and removed slight variations from each line (number of transaction in a block, block offset etc...):
Now the interesting part is ability to spot errors from that logreduced result. From the screenshot below, noticed the unique types of errors and how many times they occurred (first number in blue).
Those are non-critical errors and exceptions (not sure why they are exception btw). But let suppose one of them is a critical error and I've noticed it freezes the app or makes me miss a block, then I could easily set an alert for when the same type of error is happening again.
I can also run a query that checks for popular keywords that are usually used in case of an error as seen below:
On a more fun side, I can also check how many times my witness has generated blocks too. The current data is less than 24 hours old so I have only two block shown in Sumo Logic:
Of course, I can also live tail the log file straight from Sumo Logic without the need of doing an SSH connection to my server. And I can also setup some field extraction rules, I can extract fields and their values from the log messages and then use that in a search or dashboard. Here for example, I'm extracting the number of transactions, the block number and the name of the witness who produced the block:
These extracted fields will allow me to generate this graph of transactions signed by each witness:
After creating multiple searches and graphs, I've came up with this simple dashboard that gives me visibility on my own number of generated blocks in the last 7 days, the number of transactions per witness in the last 24 hours and the number of transactions per block in the last 15 minutes:
I can obviously add more panels to this widget for monitoring errors and warnings.
Log rotation
For those interested in how I rotate my log file and compress them to save space like here:
All I do is create a file in /etc/logrotate.d/ called steemd with the following content:
/home/quochuy/steemd.log {
rotate 7
daily
compress
missingok
}
Replace /home/quochuy/steemd.log with the path to your own log file and also make sure you have logrotate installed and running. Also note, this won't work with Steem-in-a-box (aka Steem Docker by @someguy123), it's only for if you are running steemd natively. Update: They do have a collector for Docker so you can monitor the logs of a container, this could be configured to use with Steem Docker: https://github.com/SumoLogic/sumologic-collector-docker
Related content
- Witness Earnings Report - 2018-10-08
- Witness Earnings Report - 2018-10-01
- Witness Earnings Report - 2018-09-24
- Witness Update 2018-09-24 - Ready for HF20 (again)
Summary of some of my other contributions
| Project | Description | Link |
|---|---|---|
| Steemed Phish | Chrome browser extension to make external links more visible and alert you on Phishing/Scamming links | Steemed Phish |
| Dtube Community Support | This project by @evildido allows users to run their own IPFS node and pin videos in order to cache them for longer. Dtube has the tendency of deleting non-popular video after few days only. I helped Dockerizing the project for an easier deployment on almost any platform. I've also contributed by running two nodes, one in Australia and one in Canada to cache #teamaustralia and #teamnz videos. | Dockerizing Dtube Community Support |
| DiscoSteem Bot | This is a Discord/Steem bot project by @planetenamek. When I started Steemulant Bot for the Vietnamese community, I didn't know the existence of DiscoSteem Bot and wanted something written in NodeJS. I've ended up continuing my own project as it contains code and logic for personal use and for the Vietnamese community. But I've decided to share some useful features that can benefit the users of DiscoSteem Bot and will continue to add more functionalities as I go. | Dockerizing DiscoSteem Bot |
| A Dollar A Day | This is a project run by @pennsif and will donate 15 SBD every day to a charity | "A dollar a day" fundraiser - Karaoke Singing: Le chanteur malheureux |
| Team Vietnam | I started a new Vietnamese community on Discord and developed @teamvn, a community bot to help grow the Vietnamese community and help them create higher quality content. | |
| POCKET-JS | This is my Javascript (NodeJS) port of the POCKET confirmer by @biophil. POCKET is an experimental crypto token on top of STEEM. | POCKET-JS Confirmer |
| Skippy Bot | I've written this Discord bot for @teamaustralia, it has three main features: a level bot (like Tatsumaki but with Australian animal images), a chatter bot and a footer banner generator for both @teamaustralia, @teamnz and @steemph | Witness Update 2018-07-30 - Team Australia's Skippy Bot |
Credits
- The image at the top has been generated with the Canva app using my own photo.