[uWSGI] reload-on-exception and threads

[Matt Phipps]

Hi uWSGI,

We've got a Flask (Python) app using dogpile.cache, and are having a
strange production issue where the cache doesn't seem to be refreshing. My
suspicion is that a request locked a cache key, started getting a fresh
value from the DB, then was terminated before it could unlock the cache key.

So here's my question: if we have reload-on-exception on and multiple
threads per worker (as we do), 2 threads are handling requests at the same
time, and one of them encounters an exception (e.g. IOError from SIGPIPE)
that causes uWSGI to reload the worker, will uWSGI wait for the second
thread to finish its response, or will it just kill the worker immediately?

Thanks,
Matt

[Roberto De Ioris]

Post by Matt Phipps
Hi uWSGI,
We've got a Flask (Python) app using dogpile.cache, and are having a
strange production issue where the cache doesn't seem to be refreshing. My
suspicion is that a request locked a cache key, started getting a fresh
value from the DB, then was terminated before it could unlock the cache key.
So here's my question: if we have reload-on-exception on and multiple
threads per worker (as we do), 2 threads are handling requests at the same
time, and one of them encounters an exception (e.g. IOError from SIGPIPE)
that causes uWSGI to reload the worker, will uWSGI wait for the second
thread to finish its response, or will it just kill the worker
immediately?

It will be killed immediately, i am not sure that trapping IOError is a
good thing, --reload-on-exception is for non-recoverable exception (like
inconsistent db connections).

[Matt Phipps]

Post by Roberto De Ioris
It will be killed immediately, i am not sure that trapping IOError is a
good thing, --reload-on-exception is for non-recoverable exception (like
inconsistent db connections).

I agree, sorry, that was a bad example. The real reason we have
reload-on-exception on is exactly what you mentioned: sometimes
SQLAlchemy raises while trying to tear down its session at the end of
a request, and then gets into a weird state. Looks like if we want to
keep using threads and this locking design, we'll need to turn off
reload-on-exception and figure out how to ensure the session teardown
always works.

I saw in a couple places on the list that --lazy-apps is recommended
when using SQLAlchemy. Why is that? If I understand fork() correctly
(which is dubious TBH), as long as you don't open any DB connections
during app initialization (which we don't), then preforking is ok.

Actually, just to make sure I understand preforking, is the master
basically a worker (i.e. it has the app loaded in memory) that never
serves requests and does all the master-y stuff like creating and
reloading workers, but otherwise has the same memory layout? And it
just fork()s when it creates or reloads a worker? Sorry if that was
incoherent, I know some C but haven't worked on anything as complex as
uWSGI. Your project is great, by the way!

Thanks,
Matt

[Roberto De Ioris]

Post by Matt Phipps

Post by Roberto De Ioris
It will be killed immediately, i am not sure that trapping IOError is a
good thing, --reload-on-exception is for non-recoverable exception (like
inconsistent db connections).

I agree, sorry, that was a bad example. The real reason we have
reload-on-exception on is exactly what you mentioned: sometimes
SQLAlchemy raises while trying to tear down its session at the end of
a request, and then gets into a weird state. Looks like if we want to
keep using threads and this locking design, we'll need to turn off
reload-on-exception and figure out how to ensure the session teardown
always works.

Check if uwsgi locking framework could be useful for you

Post by Matt Phipps
I saw in a couple places on the list that --lazy-apps is recommended
when using SQLAlchemy. Why is that? If I understand fork() correctly
(which is dubious TBH), as long as you don't open any DB connections
during app initialization (which we don't), then preforking is ok.

yes it is ok if you do not open file descriptors on it.

Post by Matt Phipps
Actually, just to make sure I understand preforking, is the master
basically a worker (i.e. it has the app loaded in memory) that never
serves requests and does all the master-y stuff like creating and
reloading workers, but otherwise has the same memory layout? And it
just fork()s when it creates or reloads a worker? Sorry if that was
incoherent, I know some C but haven't worked on anything as complex as
uWSGI. Your project is great, by the way!

yes, this is exactly how it works