Deadlock due to race condition in download thread when using Bigquery Storage API

[kien-truong]

There is a race condition in the download thread and the main thread when using Bigquery Storage API to fetch data.

python-bigquery/google/cloud/bigquery/_pandas_helpers.py

Lines 797 to 801 in a76af35

	for page in rowstream.pages:
	if download_state.done:
	return
	item = page_to_item(page)
	worker_queue.put(item)

python-bigquery/google/cloud/bigquery/_pandas_helpers.py

Lines 934 to 942 in a76af35

	finally:
	# No need for a lock because reading/replacing a variable is
	# defined to be an atomic operation in the Python language
	# definition (enforced by the global interpreter lock).
	download_state.done = True
	# Shutdown all background threads, now that they should know to
	# exit early.
	pool.shutdown(wait=True)

When the download thread is blocked on worker_queue.put(item), if the main thread exit, causing the pool to shut down, the download thread will be stuck. This behavior prevents the program from exiting.

[kien-truong]

@chalmerlowe can you take a look at this issue and my corresponding PR #2034? thanks

[tswast]

FWIW: We think this was affecting the BigQuery DataFrames benchmark tests. Basically, if one starts iterating through the dataframe pages but then stops iterating through, the workers would stay producing messages and never go totally out of scope / shutdown.