Last night we had a sync issue of about 30 minutes. Our monitoring system alerted us so we could take quick action. Luckily our automated scripts took all of the work out of our hands we just had to wait for the scripts to kick in and do their work.
Although downtime was only 30 minutes it seems that there is something wrong with the telegram bot from ADApools.org which showed a much larger downtime which of course is not correct.
Even now, after more than 8 hours, ADApools bot still have not notified us about the restored relays. To avoid this misleading info we contacted ADApools. Hopefully they can give us some more feedback on why messages are delayed and/or not being send thus giving the misleading information of malfunctioning nodes.
Normally we do not post a lot about slot assignments but since our latest block was from 2 epochs ago we wanted to assure you that pool is still running as it supposed to. We simply just had the bad luck that the lottery did not assigned us any slots. Luckily we have a new slot for this epoch in a couple of days!
Today we are performing some controlled disaster recovery tests. For this some relays will be temporarily unavailable (no block minting will be at risk). At the same time we are performing the DR tests we are also evaluating if our alerting system is functioning as expected.
Added industrial-graded firewalls. This will allow greater protection and better control over the network flows. For security reasons we will not disclose which one we selected but lets say it is recognized in the Gartner Magic Quadrant for Network Firewalls.