Shedding load is the responsibility of each Logger backend at the moment. I would recommend any logger that needs to go over the network to batch those requests in the handler and then send to a separate process that will do the “upload”. The separate process can do the shedding if necessary since it is extremely important that Logger handlers do not block.

That’s a fair point. Moving work to the caller also sounds sensible.

Are there recommended patterns for building backends that log over network or can otherwise block for long time? I’ve had my whole system come down because of Logger getting choked to death from too much logging, which timeouted pretty much every process in the whole system. That was obviously my fault, but it sounds like a 10s latency spike in some http-based backend could lead do the same result. Or perhaps are better approaches to the whole logging problem in a cluster environment?

Another potentially very useful feature for Logger would be ability to shed load under massive load (be it from user’s own mistake, some backend stalling, or some cascade of errors). That would result in lost log messages, but at least it would not bring down a lot of processes with it.

The Logger is one of the few use cases where a GenEvent seems like a reasonable solution. Because different clients are sending iodata to a different process, you don’t want to copy that iodata to different processes over and over again as in the example above.

We are discussing some solutions where we move most of the work to the client, which would be better to exploit concurrency, but until then we don’t have plans to drop :gen_event from Logger.