all: asynchronous Eigen threadpool runtime support

[dzarukin]

This is the final PR that covers all the current needs for asynchronous Eigen threadpool runtime support.

Important notes:

• Benchdnn treats async runtime same way as GPU when it comes for perf measurements. It also got an abstraction to ensure that for correctness validation submission happens before any thread starts execution.
• Auto-gen based implementation becomes disabled for any threadpool implementation as it's impossible to determine the threadpool flag at creation time.
• Verbose functionality doesn't properly return times as the call can't be blocking, otherwise, it'll lead to a deadlock. It's an area for a future improvement.
• The support itself (if not considering the amount of preliminary work promoted to main) is limited to two major changes:
  • Parallel calls change their capture from the capture-by-reference to the capture-by-copy. This ensures that all stack variables and objects and also an execution context will be used in delayed parallel tasks with their actual values or addresses on the heap instead of referencing them on stack where they can be already destroyed by the time of execution.
  • Sequential calls to perform the last reduction (or any kind of post parallel work) are wrapped into a parallel call with a single task issued. This is necessary to ensure the dependencies are followed (guarantees of proper synchronization between consecutive parallel calls are provided by the runtime itself). Paired with a change on the main branch that forces a parallel call with a single task to be redirected into the threadpool runtime and not to shortcut an execution on-the-spot as for other runtimes, it works. The latter runtimes shouldn't be affected performance-wise (besides one extra call).

These two key principles now must be followed for updated CPU implementations (in the future, probably, every primitive and every implementation).

In a nutshell, a developer should treat async Eigen runtime in the following way:

• Submitter thread comes and extracts all or most of the pointers from the execution context, creates local variables, and reaches:
  • the parallel section - in this case it saves all the stack variable till that moment and provides this snapshot to parallel tasks (lambda functions). All the objects will be saved either by value (POD types) or by ref_count increase, ensuring those objects won't be destroyed till all the tasks finished.
  • nested primitive - in this case it prepares a new execution context and scratchpad grantor ensuring the lifetime of all newly created objects, and steps into the next execute call, either repeating this step, or getting into the previous one.
• and then leaving the execute function saving parallel tasks on the runtime side and destroying everything created along the way even before tasks started execution. In the reality it's not always the case, and threads can pick up tasks while submitter is still inside the execute, but for simplicity of thinking, that is the worst case scenario and the runtime should be treated as such when changes are considered.

Covered primitives:

• Reorder
• Matmul
• Convolution (all flavors)
• Lnorm
• Softmax
• Eltwise/Binary/Reduction
• Graph patterns based on those primitives + SDPA.

[dzarukin]

make test

[TaoLv]

Thank you Dima for taking care of the graph part. 👍

[TaoLv]

We may need below to avoid compiler or tidy complains:
MAYBE_UNUSED(g_stream);
MAYBE_UNUSED(scratchpad);