cloudiot_mqtt_example_test: test_gateway_send_data_for_device failed

[flaky-bot]

This test failed!

To configure my behavior, see the Build Cop Bot documentation.

If I'm commenting on this issue too often, add the buildcop: quiet label and
I will stop commenting.

---

commit: ce56213
buildURL: Build Status, Sponge
status: failed

Test output

test_topic = name: "projects/python-docs-samples-tests/topics/test-device-events-b6b57f0d-7211-44b3-aa36-e653f489bf74"
message_stor..._persistence_regions: "us-west2"
  allowed_persistence_regions: "us-west3"
  allowed_persistence_regions: "us-west4"
}
test_registry_id = 'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865'

device_and_gateways = ('test-device-noauthbind', 'test-device-RS256', 'test-device-RS256-err')

capsys = <_pytest.capture.CaptureFixture object at 0x7fd654734470>
@pytest.mark.flaky(max_runs=5, min_passes=1)
def test_gateway_send_data_for_device(
        test_topic, test_registry_id, device_and_gateways, capsys): # noqa
    (device_id, gateway_id, _) = device_and_gateways

    # Setup for listening for config messages
    num_messages = 5
    jwt_exp_time = 60
    listen_time = 20

    # Connect the gateway
    cloudiot_mqtt_example.send_data_from_bound_device(
        service_account_json, project_id, cloud_region, test_registry_id,
        device_id, gateway_id, num_messages, rsa_private_path, 'RS256',
        ca_cert_path, mqtt_bridge_hostname, mqtt_bridge_port, jwt_exp_time,
        listen_time)

    out, _ = capsys.readouterr()
    assert 'Publishing message 5/5' in out
    assert 'Received message' in out
    # We know we sometimes get 'Out of memory' in the output.
    # We'd like to know when this occurs with verbose log output.


  assert 'Out of memory' not in out  # Indicates could not connect


E       assert 'Out of memory' not in ('Device client_id is '\n "'projects/python-docs-samples-tests/locations/us-central1/registries/test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/devices/test-device-RS256'\n"\n 'Creating JWT using RS256 from private key file resources/rsa_private.pem\n'\n 'Subscribing to /devices/test-device-RS256/commands/#\n'\n 'Waiting for device to attach.\n'\n 'on_disconnect 1: Out of memory.\n'\n 'on_connect Connection Accepted.\n'\n 'on_disconnect 1: Out of memory.\n'\n 'on_connect Connection Accepted.\n'\n 'Starting gateway at: 1590749092.683442\n'\n 'on_connect Connection Accepted.\n'\n 'Publishing message 1/5: '\n "'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-1' "\n 'to /devices/test-device-noauthbind/state\n'\n 'on_disconnect 1: Out of memory.\n'\n "Received message '' on topic '/devices/test-device-RS256/config' with Qos 1\n"\n 'on_publish\n'\n 'on_disconnect 1: Out of memory.\n'\n 'on_connect Connection Accepted.\n'\n 'Publishing message 2/5: '\n "'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-2' "\n 'to /devices/test-device-noauthbind/state\n'\n 'on_connect Connection Accepted.\n'\n 'on_connect Connection Accepted.\n'\n 'Publishing message 3/5: '\n "'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-3' "\n 'to /devices/test-device-noauthbind/state\n'\n 'on_disconnect 1: Out of memory.\n'\n 'on_publish\n'\n 'on_connect Connection Accepted.\n'\n 'on_disconnect 1: Out of memory.\n'\n 'Publishing message 4/5: '\n "'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-4' "\n 'to /devices/test-device-noauthbind/state\n'\n 'on_connect Connection Accepted.\n'\n 'on_connect Connection Accepted.\n'\n 'Publishing message 5/5: '\n "'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-5' "\n 'to /devices/test-device-noauthbind/state\n'\n 'on_disconnect 1: Out of memory.\n'\n 'on_disconnect 1: Out of memory.\n'\n 'on_connect Connection Accepted.\n'\n 'on_connect Connection Accepted.\n'\n 'Detaching: /devices/test-device-noauthbind/detach\n'\n 'Finished.\n')

E         'Out of memory' is contained here:

E           Device client_id is 'projects/python-docs-samples-tests/locations/us-central1/registries/test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/devices/test-device-RS256'

E           Creating JWT using RS256 from private key file resources/rsa_private.pem

E           Subscribing to /devices/test-device-RS256/commands/#

E           Waiting for device to attach.

E           on_disconnect 1: Out of memory.

E           on_connect Connection Accepted.

E           on_disconnect 1: Out of memory.

E           on_connect Connection Accepted.

E           Starting gateway at: 1590749092.683442

E           on_connect Connection Accepted.

E           Publishing message 1/5: 'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-1' to /devices/test-device-noauthbind/state

E           on_disconnect 1: Out of memory.

E           Received message '' on topic '/devices/test-device-RS256/config' with Qos 1

E           on_publish

E           on_disconnect 1: Out of memory.

E           on_connect Connection Accepted.

E           Publishing message 2/5: 'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-2' to /devices/test-device-noauthbind/state

E           on_connect Connection Accepted.

E           on_connect Connection Accepted.

E           Publishing message 3/5: 'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-3' to /devices/test-device-noauthbind/state

E           on_disconnect 1: Out of memory.

E           on_publish

E           on_connect Connection Accepted.

E           on_disconnect 1: Out of memory.

E           Publishing message 4/5: 'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-4' to /devices/test-device-noauthbind/state

E           on_connect Connection Accepted.

E           on_connect Connection Accepted.

E           Publishing message 5/5: 'test-registry-93ec3d25-1f90-48dd-af13-9c210a777fa2-1590748865/test-device-RS256-test-device-noauthbind-payload-5' to /devices/test-device-noauthbind/state

E           on_disconnect 1: Out of memory.

E           on_disconnect 1: Out of memory.

E           on_connect Connection Accepted.

E           on_connect Connection Accepted.

E           Detaching: /devices/test-device-noauthbind/detach

E           Finished.
cloudiot_mqtt_example_test.py:190: AssertionError

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (ce56213), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[tmatsuo]

Another failure

From the log, it seems like the device is getting the message.
Maybe I can remove the assertion about Out of memory.