Simplify threading: remove the use of inner_pool.

[szabadka]

We only used inner_pool in the prefill FFW function, and there we can achieve sufficient parallelism on the rows of the matrix-vector multiplications.

Benchmark results on a 1600-token summarization task:

               Prefill speed
Num threads    BEFORE         AFTER
4               9.24 t/s       9.76 t/s
18             31.41 t/s      31.16 t/s
32             31.41 t/s      45.13 t/s
64             31.03 t/s      57.85 t/s

[jan-wassenberg]

Nice simplification, thank you @szabadka !
Was talking today with Austin about using matmul in Prefill, that will also help speed this up.

Unrelatedly, what do you think of moving some of the function call arguments into a struct, for example AuxOut(layers_output) and/or Sampling(temp/gen)?

[szabadka]

Unrelatedly, what do you think of moving some of the function call arguments into a struct, for example AuxOut(layers_output) and/or Sampling(temp/gen)?

Yes, that could be useful, these functions have quite a few arguments.

[jan-wassenberg]

Will do :)