Global variables

[s-good]

Hi @BillMills, I've been thinking a little bit about what you told me about global module variables being a problem when running with multi processors. This is awkward as some of the EN checks (particularly EN std level check) relies on this. I wonder if a possible work around is to remove the parallel running from within the Python code and set it up so that we can tell it to process a particular set of profiles in serial. E.g. we would run it like so:

python AutoQC.py test 0 1000

Where the 0 and 1000 tells it to process profiles 0 through 1000. Then to get the parallel processing we could have a shell script or another Python program that automatically spawns lots of the AutoQC processes, each processing a different set of profiles.

I think this will avoid the global variables issue as each AutoQC process has its own set of global variables and will be processing the profiles one at a time, but we will still get the speed up from running in parallel. What do you think?

[katieannemills]

I like this idea and wish it would work, but if I've understood correctly (possibly not), Python's global interpreter lock might get in the way of this - a whole bunch of python programs running simultaneously actually take turns in a single queue.

Luckily though, I have almost finished solving the problem for the EN tests on branch no-global. Basically, I'm just:

• checking the database for precomputed QC results instead of having them hang out on a global variable
• parking the other globals on the parameters dictionary that gets handed around between all the tests. This works essentially like a global data store, but avoids confusing scope issues that the old data.ds datastore might have had, being imported separately in a bunch of threads. I think this will work, though the current implementation on no-global still needs to be 'namespaced' by uid, so concurrent profiles don't overwrite each other's 'globals' - but I think I can get this done tonight this weekend.

All told, I'm hoping no-global will be complete, validated and mergable by EOB Monday.

[s-good]

I think the GIL only applies if using threads (see https://docs.python.org/2/library/multiprocessing.html which says that it avoids the GIL by using sub-processes rather than threads). This would be completely separate Python processes so shouldn't be affected (although I should caveat this by saying that I don't really know what I'm talking about here!)

Have you found a way to store the data saved in the EN background check, which is then used in the EN std level check (and also I think in the EN background available check)? My concern with going down the route of accessing results from the database and/or saving EN background check data in memory is the potential to slow everything down through disk access (either by have to access the database file or by filling up physical memory so that we have to start using virtual memory).

Anyway, don't want to stop you if you can see a way to get it all to work in the no-global implementation as this would be the nicer solution overall, but wanted to raise the idea as it could be a simple solution to get to v1.0 of AutoQC and a more elegant method could be introduced later.

[katieannemills]

I'm also pretty shaky on exactly how GIL works, so if you think it's okay, then it probably is. If I can't get this sorted by the time you come back to work next week, then we'll just put no-global off to v2 - but I think I can get this done by then, barring any spectacular new regressions.

The EN_background variables that need to be passed along were what I was referring to in appending them to the parameters dictionary; I'm hoping this doesn't blow up execution time, since there will only be a number of sets of these parameters in memory at once equal to the number of parallel processes - we can drop these from parameters after each profile completes.

[katieannemills]

Alright - now I understand where this gets particularly nasty; carrying results from a given profile from one test to the next is no big deal, but when we need results from buddy profiles, things get messier. no-global almost reproduces master correctly, but with small discrepancies that likely won't be sorted out this weekend; I'll implement and validate your suggestion on Sunday, and leave this issue and a re-think about out parallelization scheme for v2.

[katieannemills]

Update: single-queue implements your suggestion and produces the same results on my batch of testing profiles, but I think I can speed it up a bit with a hybrid approach; running in different python processes means (at a minimum) that each process will run all of EN_track for all profiles, since the different processes check their own in-memory records rather than the db. Checking the db for precomputed qc results didn't present any problems in no-global, so pulling that in should be easy and may speed things up; as it stands, running 10 processes instead of 1 resulted in a speed-up of only a factor of 3, which is order of magnitude consistent if you imagine EN_track originally taking up about a quarter of the running time and running effectively not in parallel, while the other 3/4 of the time is sped up by 10x.

I'll try and combine these approaches first thing Monday, see if we get any more speed, and then it's time to sign off for v1.

[katieannemills]

Alright, that's that (for now) - #191, assuming it passes testing, implements @s-good's suggestion for process level parallelization. Unfortunately a hybrid db-reading strategy as described above did not improve performance as I had hoped, so I've opted for the smaller and simpler changeset for now. My best guess as to what's working against our parallelization speed gains is that queues with equal numbers of profiles will not necessarily execute in equal amounts of time; one queue could have many profiles with a large number of levels, for example, causing that queue to run much longer than the others. Anyway - that's something to think about for v2, but given the volume of data we need to wade through before October, it's time to move on.