Post by Roberto De IorisPost by Jean-Baptiste QuenotPost by Roberto De IorisIf i remember correctly you run with full logging disabled (so reporting
it in the logs would be useless). Maybe the master can print the 'percent'
variation ? Something like
*** listen queue is now 10 % full
*** listen queue is now 20 % full
*** listen queue is now 10 % full
*** listen queue is now 30 % full
or you want to manage it in the code accessing something like
env['uwsgi.listen_queue'] ?
Yes I'd like to manage it with something like
uwsgi.stats['listen_queue']. A 'stats' dictionary in the uwsgi module
sounds better than WSGI variables.
Sadly i cannot go (for now) with dictionaries as in threading mode they get
clobbered (and i do not want to add another lock in the stack ;) )
I will export a uwsgi.listen_queue() function that will return the value
taken from a shared area (no locking needed as the value is updated only
by master)
Hi Roberto,
I'm thinking again about monitoring the listen queue. There are
various problems:
1) listen_queue() will never give numbers above the maximum queue
size, so obviously it can only be useful to know that the queue is
full. So maybe providing listen_queue_full() boolean result instead
would make more sense.
2) as a matter of facts the error message that is printed in the logs
does not need to mention tcpi_unacked, as it will never exceed the
queue size + 1. For example, with only one worker, only one slot in
the listen queue and many concurrent requests, here is what the
messages look like:
Tue May 31 10:32:06 2011 - *** uWSGI listen queue of socket 4 full !!! (2/1) ***
Tue May 31 10:32:07 2011 - *** uWSGI listen queue of socket 4 full !!! (2/1) ***
Tue May 31 10:32:08 2011 - *** uWSGI listen queue of socket 4 full !!! (2/1) ***
Tue May 31 10:32:09 2011 - *** uWSGI listen queue of socket 4 full !!! (2/1) ***
Tue May 31 10:32:10 2011 - *** uWSGI listen queue of socket 4 full !!! (2/1) ***
Tue May 31 10:32:11 2011 - *** uWSGI listen queue of socket 4 full !!! (2/1) ***
Tue May 31 10:32:12 2011 - *** uWSGI listen queue of socket 4 full !!! (2/1) ***
Tue May 31 10:32:13 2011 - *** uWSGI listen queue of socket 4 full !!! (2/1) ***
Tue May 31 10:32:14 2011 - *** uWSGI listen queue of socket 4 full !!! (2/1) ***
Tue May 31 10:32:15 2011 - *** uWSGI listen queue of socket 4 full !!! (1/1) ***
3) I tried tweaking the error message to mention the tcpi_lost field,
but it is always zero.
So to sum up, when the queue is getting full, the sysadmin cannot
determine the severity of the downtime just by looking at the size of
the listen queue, because it remains constant at the maximum size (eg
at 128), and it is not reliable to check an instantaneous value
periodically. Knowing the number of dropped requests would be great,
but this information is not available on Linux (may be available on
FreeBSD). So I believe the only option is to add a new counter in the
master, that will be incremented everytime the listen queue has been
checked full.
I would monitor this counter and use a DERIVE data source in RRD to
graph the severity of my UWSGI app's down times.
Does it make sense?
And last but not least, querying the uwsgi python module involves
doing an HTTP request and getting a worker to serve this internal
request just for knowing the value of a counter. Or do you think of
another way to access the number? (sending a signal to the master,
like Java does for getting a stack trace, providing a unix socket
connection and fetch the data with socat, like haproxy does, or
providing a monitoring port that talks HTTP?)
Thanks in advance,
--
Jean-Baptiste Quenot