[BUG] multiple-pause-resume.sh sof-test fails to FW_GEN_MSG failed with err 3018

[kv2019i]

Describe the bug
The multiple-pause-resume.sh sof-test started to fail with latest Zephyr on Intel cAVS2.5 based platforms (TGL, ADL). First seen on
#8764

To Reproduce
Build SOF with recent Zephyr, run sof-test.

Reproduction Rate
50+%

Expected behavior
Chain-dma host dma stop fails

Impact
SOF CI PR tests are failing with high rate

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
   See [DNM] Zephyr smp rework #8764
2. Name of the topology file
   • Topology: any Intel cavss2.5 tplg with HDMI (HDMI uses chain-dma) , see [DNM] Zephyr smp rework #8764 test results
3. Name of the platform(s) on which the bug is observed.
   • Platform: tgl, adl, rpl

Screenshots or console output
See https://sof-ci.01.org/sofpr/PR8764/build2195/devicetest/index.html?model=TGLU_UP_HDA-ipc4&testcase=multiple-pause-resume-50

[  825.139933] <inf> ipc: ipc_cmd: rx	: 0xe050000|0x0
[  825.140968] <inf> chain_dma: chain_link_stop: comp:128 0x80 chain_link_stop(): dma_stop() link chan_index = 0
[  825.142083] <inf> ll_schedule: zephyr_ll_task_done: task complete 0xbe0bf810 0x204b0U
[  825.142095] <inf> ll_schedule: zephyr_ll_task_done: num_tasks 3 total_num_tasks 3
[  825.142133] <err> ipc: ipc_cmd: ipc4: FW_GEN_MSG failed with err 3018
[  825.142873] <inf> ipc: ipc_cmd: rx	: 0xe040000|0x0
[  825.142901] <inf> dma: dma_put: dma_put(), dma = 0x9e09f650, sref = 1
[  825.142915] <inf> dma: dma_put: dma_put(), dma = 0x9e09f6f0, sref = 1

[kv2019i]

zephyrproject-rtos/zephyr#68101 didn't work. 5000us is a REALLY long time to wait already and we still had fails, so something else goes wrong. On other platforms using chain-dma, this is not seeing, making this quite curious. Seems to be specific to cAVS2.5.

[kv2019i]

I now suspect the DMA stop problem the Zephyr update uncovers, is actually a problem in our host DMA stop sequence. With current kernel and SOF, the order of stop seems wrong:

• CHAIN_DMA IPC is sent -> this clears DGCS_GEN and DGCS_FIFORDY bits to stop the DMA (in Zephyr driver)
• BE trigger is run in kernel and this clears the RUN bit on host side

As far as I can tell, this is against the recommended programming flow. With lcoal stress tests thesofproject/linux#4798 seems to fix the issue (GBUSY no longer stuck), but more stress testing is needed.

FYI @jsarha @ujfalusi @plbossart @ranj063 @RanderWang