Three posts today… a new record.
The VM that runs the project was choking when I woke up this morning. In fact it appears it’s been choking for a few weeks on the 4AM run. There was about a week’s worth of 4AM runs running and the box was really bogged down.
I should have seen this coming, because up until a few weeks back, the hourly runs were published usually no more than a couple of minutes after the hour and they had slipped to ten or twelve past the hour.
I stumbled on this while checking the status of the proxy judges with a few ad hoc MySQL queries that took way too long to run.
What I found out during that exercise was that a handful of my proxy judges had gone dark.
This is not good and cause for a certain percentage of false negatives.
So I dug up and verified a whole new batch of judges (on of these days I’m going to publish My Dirty Little Secret about proxy judges – you’re going to love it), rebooted the VM, fixed the issues with the 4AM run, and started resurrecting dead proxies.
Hopefully things will pick up next week.
0 Responses to “Choked Up”
Leave a Reply