Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Vector pow for x86_64 and tests. 
Here is implementation of vectorized pow containing SSE, AVX,
AVX2 and AVX512 versions according to Vector ABI
<https://groups.google.com/forum/#!topic/x86-64-abi/LmppCfN1rZ4>.

  * bits/libm-simd-decl-stubs.h: Added stubs for pow.
    * math/bits/mathcalls.h: Added pow declaration with __MATHCALL_VEC.
    * sysdeps/unix/sysv/linux/x86_64/libmvec.abilist: New versions added.
    * sysdeps/x86/fpu/bits/math-vector.h: Added SIMD declaration and asm
    redirections for pow.
    * sysdeps/x86_64/fpu/Makefile (libmvec-support): Added new files.
    * sysdeps/x86_64/fpu/Versions: New versions added.
    * sysdeps/x86_64/fpu/libm-test-ulps: Regenerated.
    * sysdeps/x86_64/fpu/multiarch/Makefile (libmvec-sysdep_routines): Added
    build of SSE, AVX2 and AVX512 IFUNC versions.
    * sysdeps/x86_64/fpu/svml_d_wrapper_impl.h: Added 2 argument wrappers.
    * sysdeps/x86_64/fpu/multiarch/svml_d_pow2_core.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_pow2_core_sse4.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_pow4_core.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_pow4_core_avx2.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_pow8_core.S: New file.
    * sysdeps/x86_64/fpu/multiarch/svml_d_pow8_core_avx512.S: New file.
    * sysdeps/x86_64/fpu/svml_d_pow2_core.S: New file.
    * sysdeps/x86_64/fpu/svml_d_pow4_core.S: New file.
    * sysdeps/x86_64/fpu/svml_d_pow4_core_avx.S: New file.
    * sysdeps/x86_64/fpu/svml_d_pow8_core.S: New file.
    * sysdeps/x86_64/fpu/svml_d_pow_data.S: New file.
    * sysdeps/x86_64/fpu/svml_d_pow_data.h: New file.
    * sysdeps/x86_64/fpu/test-double-vlen2-wrappers.c: Added vector pow test.
    * sysdeps/x86_64/fpu/test-double-vlen2.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen4-avx2-wrappers.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen4-avx2.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen4-wrappers.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen4.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen8-wrappers.c: Likewise.
    * sysdeps/x86_64/fpu/test-double-vlen8.c: Likewise.
    * NEWS: Mention addition of x86_64 vector pow.
  • Loading branch information
Andrew Senkevich committed Jun 17, 2015
1 parent 1663be0 commit c10b9b1
Show file tree
Hide file tree
Showing 31 changed files with 6,934 additions and 4 deletions.
33 changes: 33 additions & 0 deletions ChangeLog
View file
@@ -1,5 +1,38 @@
2015-06-17 Andrew Senkevich <andrew.senkevich@intel.com>

* bits/libm-simd-decl-stubs.h: Added stubs for pow.
* math/bits/mathcalls.h: Added pow declaration with __MATHCALL_VEC.
* sysdeps/unix/sysv/linux/x86_64/libmvec.abilist: New versions added.
* sysdeps/x86/fpu/bits/math-vector.h: Added SIMD declaration and asm
redirections for pow.
* sysdeps/x86_64/fpu/Makefile (libmvec-support): Added new files.
* sysdeps/x86_64/fpu/Versions: New versions added.
* sysdeps/x86_64/fpu/libm-test-ulps: Regenerated.
* sysdeps/x86_64/fpu/multiarch/Makefile (libmvec-sysdep_routines): Added
build of SSE, AVX2 and AVX512 IFUNC versions.
* sysdeps/x86_64/fpu/svml_d_wrapper_impl.h: Added 2 argument wrappers.
* sysdeps/x86_64/fpu/multiarch/svml_d_pow2_core.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_pow2_core_sse4.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_pow4_core.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_pow4_core_avx2.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_pow8_core.S: New file.
* sysdeps/x86_64/fpu/multiarch/svml_d_pow8_core_avx512.S: New file.
* sysdeps/x86_64/fpu/svml_d_pow2_core.S: New file.
* sysdeps/x86_64/fpu/svml_d_pow4_core.S: New file.
* sysdeps/x86_64/fpu/svml_d_pow4_core_avx.S: New file.
* sysdeps/x86_64/fpu/svml_d_pow8_core.S: New file.
* sysdeps/x86_64/fpu/svml_d_pow_data.S: New file.
* sysdeps/x86_64/fpu/svml_d_pow_data.h: New file.
* sysdeps/x86_64/fpu/test-double-vlen2-wrappers.c: Added vector pow test.
* sysdeps/x86_64/fpu/test-double-vlen2.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen4-avx2-wrappers.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen4-avx2.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen4-wrappers.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen4.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen8-wrappers.c: Likewise.
* sysdeps/x86_64/fpu/test-double-vlen8.c: Likewise.
* NEWS: Mention addition of x86_64 vector pow.

* sysdeps/unix/sysv/linux/x86_64/libmvec.abilist: New symbols added.
* sysdeps/x86/fpu/bits/math-vector.h: Added SIMD declaration and asm
redirections for expf.
Expand Down
2 changes: 1 addition & 1 deletion NEWS
View file
Expand Up @@ -53,7 +53,7 @@ Version 2.22
condition in some applications.

* Added vector math library named libmvec with the following vectorized x86_64
implementations: cos, cosf, sin, sinf, log, logf, exp, expf.
implementations: cos, cosf, sin, sinf, log, logf, exp, expf, pow.
The library can be disabled with --disable-mathvec. Use of the functions is
enabled with -fopenmp -ffast-math starting from -O1 for GCC version >= 4.9.0.
The library is linked in as needed when using -lm (no need to specify -lmvec
Expand Down
3 changes: 3 additions & 0 deletions bits/libm-simd-decl-stubs.h
View file
Expand Up @@ -49,4 +49,7 @@
#define __DECL_SIMD_expf
#define __DECL_SIMD_expl

#define __DECL_SIMD_pow
#define __DECL_SIMD_powf
#define __DECL_SIMD_powl
#endif
2 changes: 1 addition & 1 deletion math/bits/mathcalls.h
View file
Expand Up @@ -150,7 +150,7 @@ __END_NAMESPACE_C99

_Mdouble_BEGIN_NAMESPACE
/* Return X to the Y power. */
__MATHCALL (pow,, (_Mdouble_ __x, _Mdouble_ __y));
__MATHCALL_VEC (pow,, (_Mdouble_ __x, _Mdouble_ __y));

/* Return the square root of X. */
__MATHCALL (sqrt,, (_Mdouble_ __x));
Expand Down
4 changes: 4 additions & 0 deletions sysdeps/unix/sysv/linux/x86_64/libmvec.abilist
View file
Expand Up @@ -4,6 +4,7 @@ GLIBC_2.22
_ZGVbN2v_exp F
_ZGVbN2v_log F
_ZGVbN2v_sin F
_ZGVbN2vv_pow F
_ZGVbN4v_cosf F
_ZGVbN4v_expf F
_ZGVbN4v_logf F
Expand All @@ -12,6 +13,7 @@ GLIBC_2.22
_ZGVcN4v_exp F
_ZGVcN4v_log F
_ZGVcN4v_sin F
_ZGVcN4vv_pow F
_ZGVcN8v_cosf F
_ZGVcN8v_expf F
_ZGVcN8v_logf F
Expand All @@ -20,6 +22,7 @@ GLIBC_2.22
_ZGVdN4v_exp F
_ZGVdN4v_log F
_ZGVdN4v_sin F
_ZGVdN4vv_pow F
_ZGVdN8v_cosf F
_ZGVdN8v_expf F
_ZGVdN8v_logf F
Expand All @@ -32,3 +35,4 @@ GLIBC_2.22
_ZGVeN8v_exp F
_ZGVeN8v_log F
_ZGVeN8v_sin F
_ZGVeN8vv_pow F
6 changes: 6 additions & 0 deletions sysdeps/x86/fpu/bits/math-vector.h
View file
Expand Up @@ -44,6 +44,8 @@
# define __DECL_SIMD_exp __DECL_SIMD_x86_64
# undef __DECL_SIMD_expf
# define __DECL_SIMD_expf __DECL_SIMD_x86_64
# undef __DECL_SIMD_pow
# define __DECL_SIMD_pow __DECL_SIMD_x86_64

/* Workaround to exclude unnecessary symbol aliases in libmvec
while GCC creates the vector names based on scalar asm name.
Expand All @@ -65,6 +67,10 @@ __asm__ ("_ZGVbN4v___expf_finite = _ZGVbN4v_expf");
__asm__ ("_ZGVcN8v___expf_finite = _ZGVcN8v_expf");
__asm__ ("_ZGVdN8v___expf_finite = _ZGVdN8v_expf");
__asm__ ("_ZGVeN16v___expf_finite = _ZGVeN16v_expf");
__asm__ ("_ZGVbN2vv___pow_finite = _ZGVbN2vv_pow");
__asm__ ("_ZGVcN4vv___pow_finite = _ZGVcN4vv_pow");
__asm__ ("_ZGVdN4vv___pow_finite = _ZGVdN4vv_pow");
__asm__ ("_ZGVeN8vv___pow_finite = _ZGVeN8vv_pow");

# endif
#endif
4 changes: 3 additions & 1 deletion sysdeps/x86_64/fpu/Makefile
View file
Expand Up @@ -13,7 +13,9 @@ libmvec-support += svml_d_cos2_core svml_d_cos4_core_avx \
svml_s_logf_data svml_d_exp2_core svml_d_exp4_core_avx \
svml_d_exp4_core svml_d_exp8_core svml_d_exp_data \
svml_s_expf4_core svml_s_expf8_core_avx svml_s_expf8_core \
svml_s_expf16_core svml_s_expf_data \
svml_s_expf16_core svml_s_expf_data svml_d_pow2_core \
svml_d_pow4_core_avx svml_d_pow4_core svml_d_pow8_core \
svml_d_pow_data \
init-arch
endif

Expand Down
1 change: 1 addition & 0 deletions sysdeps/x86_64/fpu/Versions
View file
Expand Up @@ -4,6 +4,7 @@ libmvec {
_ZGVbN2v_sin; _ZGVcN4v_sin; _ZGVdN4v_sin; _ZGVeN8v_sin;
_ZGVbN2v_log; _ZGVcN4v_log; _ZGVdN4v_log; _ZGVeN8v_log;
_ZGVbN2v_exp; _ZGVcN4v_exp; _ZGVdN4v_exp; _ZGVeN8v_exp;
_ZGVbN2vv_pow; _ZGVcN4vv_pow; _ZGVdN4vv_pow; _ZGVeN8vv_pow;
_ZGVbN4v_cosf; _ZGVcN8v_cosf; _ZGVdN8v_cosf; _ZGVeN16v_cosf;
_ZGVbN4v_sinf; _ZGVcN8v_sinf; _ZGVdN8v_sinf; _ZGVeN16v_sinf;
_ZGVbN4v_logf; _ZGVcN8v_logf; _ZGVdN8v_logf; _ZGVeN16v_logf;
Expand Down
12 changes: 12 additions & 0 deletions sysdeps/x86_64/fpu/libm-test-ulps
View file
Expand Up @@ -1947,6 +1947,18 @@ ifloat: 4
ildouble: 2
ldouble: 2

Function: "pow_vlen2":
double: 1

Function: "pow_vlen4":
double: 1

Function: "pow_vlen4_avx2":
double: 1

Function: "pow_vlen8":
double: 1

Function: "sin":
ildouble: 1
ldouble: 1
Expand Down
3 changes: 2 additions & 1 deletion sysdeps/x86_64/fpu/multiarch/Makefile
View file
Expand Up @@ -65,5 +65,6 @@ libmvec-sysdep_routines += svml_d_cos2_core_sse4 svml_d_cos4_core_avx2 \
svml_s_logf16_core_avx512 svml_d_exp2_core_sse4 \
svml_d_exp4_core_avx2 svml_d_exp8_core_avx512 \
svml_s_expf4_core_sse4 svml_s_expf8_core_avx2 \
svml_s_expf16_core_avx512
svml_s_expf16_core_avx512 svml_d_pow2_core_sse4 \
svml_d_pow4_core_avx2 svml_d_pow8_core_avx512
endif
38 changes: 38 additions & 0 deletions sysdeps/x86_64/fpu/multiarch/svml_d_pow2_core.S
View file
@@ -0,0 +1,38 @@
/* Multiple versions of vectorized pow.
Copyright (C) 2014-2015 Free Software Foundation, Inc.
This file is part of the GNU C Library.
The GNU C Library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
The GNU C Library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with the GNU C Library; if not, see
<http://www.gnu.org/licenses/>. */

#include <sysdep.h>
#include <init-arch.h>

.text
ENTRY (_ZGVbN2vv_pow)
.type _ZGVbN2vv_pow, @gnu_indirect_function
cmpl $0, KIND_OFFSET+__cpu_features(%rip)
jne 1f
call __init_cpu_features
1: leaq _ZGVbN2vv_pow_sse4(%rip), %rax
testl $bit_SSE4_1, __cpu_features+CPUID_OFFSET+index_SSE4_1(%rip)
jz 2f
ret
2: leaq _ZGVbN2vv_pow_sse2(%rip), %rax
ret
END (_ZGVbN2vv_pow)
libmvec_hidden_def (_ZGVbN2vv_pow)

#define _ZGVbN2vv_pow _ZGVbN2vv_pow_sse2
#include "../svml_d_pow2_core.S"

0 comments on commit c10b9b1

Please sign in to comment.