[POC] Adds TH_TENSOR_APPLY2_PARALLEL

[fmassa]

Following the discussion in #323, I've tried to add a macro TH_TENSOR_APPLY2_PARALLEL, which uses omp if both tensors are contiguous. For the moment, I haven't set a threshold to use omp or not.

As a proof of concept, I added it to the unary operations implemented by LAB_IMPLEMENT_BASIC_FUNCTION (like abs, tan, etc).

Any thoughts ?
cc @dominikgrewe

[fmassa]

Just a quick heads up on this PR.
A simple benchmark of the log function on a 2 cores/4 threads machine gave an speed up of 2.5 times compared to the non-threaded version, for a 2e8 elements contiguous double vector.
This could be specially useful in pointwise operations in nn, as for now the parallelism in some of the transfer functions has been removed torch/nn#403

[dominikgrewe]

Feels like there's a lot of code duplication here. Can't we just call THTensor_isContiguous? Same for counting the number of elements. We'd need the tensor type as parameter to the macro (in addition to the value type), but that can be easily added.

[dominikgrewe]

Sorry for the delay in getting back. It looks good to me in principle, but we should try to reduce the amount of code duplication.

[fmassa]

Hi Dominik,
Thanks for the comments.
I agree that there are lots of code duplication. It actually comes from the original TH_TENSOR_APPLY2 macro.
As I wanted to make this apply function as similar as possible to the non-omp one, I copied/pasted the TH_TENSOR_APPLY2 macro and added few lines to make it work with omp, the main difference is here.
When I found the need of the is_Contiguous function, I realised it was already present in the original macro (but not explicitly set up, see here for example), so I just added the variable isContiguous in this part.

I'll replace those repetitive parts by their respective function calls.

[fmassa]

Hi @dominikgrewe ,

I was thinking again about this again, and I'm not sure we would like to have a different interface than the one in TH_TENSOR_APPLY2.
My initial goal with this PR was to modify TH_TENSOR_APPLY2, but then I realised that there are some use cases where this parallelisation woudn't work (because of reductions or breaks in the middle of the for loop). I thus decided to create a new macro (following your idea in #323 ), but keeping the interface the same to allow easily replacing the TH_TENSOR_APPLY2 by this new one, when applicable (including in nn).

I agree though that there are lots of code duplication, making the compilation time quite long.

What do you think ? @soumith @andresy @koraykv @dominikgrewe