syslog-ng eating up CPU time for no particular reason
I encountered a really odd phenomenon with syslog-ng yesterday. A mail- and web-server with considerable traffic was disconnected from the net for some hours during a router failure. After the router came back online and the server was reachable again, I noticed an unusual high system CPU load. Strangely, the server itself wasn't changed in any way (it wasn't even shut down) and suddenly the CPU utilization raised to around 80%, whereas it normally stays under 20%. Strange thing was, that most CPU time seemed to be eaten up by the syslog-ng daemon.
After an hour of searching for the problem and trying almost every syslog-ng tuning hint I found on the net, I was almost about to switch to another syslogger daemon, when I finally found out that the problem lies in the massive amount of messages being logged to the VGA console.
As said, there was nothing changed on the server itself and the amount of logged messages couldn't be a problem for the raid storage device. But syslog-ng was configured to log all messages to tty12, which ate up the CPU time (probably the scrolling text on the console). That hasn't been a problem before, since the server runs headless and the console is always switched to tty1 after a boot. But after the network outage, the repair-team must have plugged in a display and switched the console to tty12 to see if mails are getting accepted again.
So a note to myself: Always remove the default configuration statement that logs all messages to a tty, or you'll get some funny surprises later.