BLD: Check Intel compiler when checking complex types in build by lysnikolaou · Pull Request #25044 · numpy/numpy

lysnikolaou · 2023-10-31T16:43:09Z

Fixes numpy#25034. use clearer syntax Co-authored-by: Sebastian Berg <sebastian@sipsolutions.net>

seberg · 2023-11-06T13:44:42Z

Rebased to fix the accidental submodule merge (and squash commits). Since it's a one line change and nobody commented on it, will assume this is a step in the right direction.

Thanks @lysnikolaou.

lysnikolaou · 2023-11-06T13:49:19Z

I'm not sure about this, but I think I can recall clang-cl using the MSVC types. Should I look into it?

EDIT: Yes, clang-cl CI fails due to this.

seberg · 2023-11-06T13:52:39Z

Argg, sorry... Interesting, but that makes me think that more that switching to C11 requires also changing this here in gh-25072.

also add 'std=' flags and avoid a '-utf-8' flag by adding '-source-charset'

mattip · 2023-11-07T06:38:21Z

I added a few more changes, especially the one needed for fast math from this comment in the issue and adding a few more std= flags. That together with the workaround for -utf-8 gets me to a pretty clean build with one warning that repeats whenever compiling with cpp and one link-time error:

d:\Intel\oneAPI\compiler\latest\windows\bin-llvm\..\compiler\include\complex.h(37,14): warning: "The /Qstd=c99 compilation option is required to enable C99 support for C programs" [-W#warnings]
            #warning "The /Qstd=c99 compilation option is required to enable C99 support for C programs"
             ^
1 warning generated.
[421/496] Linking target numpy/_core/_multiarray_umath.cp310-win_amd64.pyd
FAILED: numpy/_core/_multiarray_umath.cp310-win_amd64.pyd numpy/_core/_multiarray_umath.cp310-win_amd64.pdb
"xilink.exe" @numpy/_core/_multiarray_umath.cp310-win_amd64.pyd.rsp
libmmd.lib(libmmd.dll) : error LNK2005: ldexpf already defined in meson-generated_arraytypes.c.obj
   Creating library numpy\_core\_multiarray_umath.cp310-win_amd64.lib and object numpy\_core\_multiarray_umath.cp310-win_amd64.exp
numpy\_core\_multiarray_umath.cp310-win_amd64.pyd : fatal error LNK1169: one or more multiply defined symbols found

mattip · 2023-11-07T08:32:52Z

Hmm. it seems clang-cl needs the msvc complex definitions, but the intel clang-based compilers need the other ones.

lysnikolaou · 2023-11-07T11:43:40Z

Hmm. it seems clang-cl needs the msvc complex definitions, but the intel clang-based compilers need the other ones.

Yup, that's right. I think if cc.get_argument_syntax() == 'msvc' and not cc.get_id().startswith('intel') should work in the meson.build file, right?

mattip · 2023-11-07T12:16:17Z

This is what we want, right?

`get_id()`	result
`msvc`	True
`clang-cl`	True
`intel-llvm-cl`	False

So `cc.get_id() == 'msvc' or cc.get_id() == 'clang-cl'

mattip · 2023-11-10T10:25:21Z

CI is passing

lysnikolaou · 2023-11-10T10:34:24Z

I don't understand the Intel/fastmath related change well enough to comment on it, but the rest looks good!

rgommers

This looks pretty good, thanks Matti and Lysandros. One question in a comment inline. Second question: there are quite a few more __INTEL_COMPILER usages in the code base, did you audit those, or only changed the 3 in this PR in response to build/test failures?

rgommers · 2023-11-13T19:16:46Z

+  _intel_c99_flag = cc.get_supported_arguments('/Qstd=c99')
+  add_project_arguments(_intel_c99_flag, language: 'c')
+  _intel_c17 = cpp.get_supported_arguments('/Qstd=c++17')
+  add_project_arguments(_intel_c17, language: 'cpp')


This whole if block looks incorrect, because it hardcodes flags that should be set already by the default options higher up. And that may be a problem soon because I already have a PR open to upgrade from C99 to C11. Did you add this to work around a problem with the default flags Meson is adding? If so, let's add a comment and link the upstream issue for it.

This was to solve the same problem as in #25072: hundreds of warnings about needed language standards. I am happy to wait for that PR to land first.

Oh yes, you added the /Qstd=c99 warning higher up. That should already be added, because we have c_std=c99' in default_options at the top of meson.build. If it's indeed not added, can you open a Meson issue for it?

To check if it's added:

cd build python ../vendored-meson/meson/meson.py introspect --targets -i | grep c99

On main of course the build fails when checking for complex, which what started this whole exercise in the first place. But if I run the command on the broken build, I do not see any c99. Maybe I need to create a minimal reproducer for a proper meson issue.

mattip · 2023-11-15T08:54:34Z

I only added __INTEL_LLVM_COMPILER next to __INTEL_COMPILER where it was required for compilation.

This PR is still incomplete: linking fails. To recap (from a comment in the issue), to test this PR one needs to

install the Intel oneAPI windows compiler from the intel download site

Set some environment variables:

<install path>\setvars.bat intel64
set CC=icx.exe
set CXX=icx.exe

install packages for the build (standard stuff)

pip install scipy-openblas64 -r build_requirements.txt -r test_requirements.txt
REM use scipy-openblas wheels to provide OpenBLAS
pip install scipy-openblas64
set PKG_CONFIG_PATH=<path>\.openblas

try to build NumPy with pip install . --no-build-isolation

If we decide this is important, there is an example github repo with a workflow that downloads and caches the compiler.

rgommers · 2023-11-15T09:32:46Z

If we decide this is important, there is an example github repo with a workflow that downloads and caches the compiler.

I tried this before in scipy/scipy#16957. It's a real pain to work with. I'm hoping the Intel compilers before available in conda-forge (we had some discussions with Intel engineers around that), then it'll be much easier. I wouldn't bother now.

charris · 2024-01-04T20:04:29Z

Needs rebase.

charris · 2024-02-05T00:44:29Z

Be good to finish this.

lysnikolaou · 2024-03-04T09:39:02Z

@rgommers @mattip I rebased this PR, so that it's easier to be reviewed/merged, in case we want to do so before 2.0.

mattip · 2024-03-04T10:40:29Z

There is so much already in 2.0 I didn't want to add more. Also

This PR is still incomplete: linking fails.

and I have not gotten it to work yet

lysnikolaou · 2024-03-04T10:43:56Z

Got it. I only wanted to remind you all of it, just in case it was missed.

rgommers · 2024-11-25T08:43:13Z

@mattip perhaps you could retry with latest Intel Fortran? We have had working CI jobs with those compilers in SciPy for a while now, and gh-25034 notes that an issue with 2023.1.0 is fixed in 2024.2.1

ashish-2022 · 2024-11-25T10:22:40Z

With new version of visual studio professional following is the status:
(I used tar.gz source files from PyPI for build)
Numpy=1.26.4
Visual Studio version 17.10.7
intelOneAPI 2024.2.1

Build command: python -m pip install . -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Csetup-args=-Dblas-order=mkl -Csetup-args=-Dlapack-order=mkl -Csetup-args=-Dblas=mkl -Csetup-args=-Dlapack=mkl

Selecting Microsoft compilers with MKL PASSES:
set CC=cl
set CXX=cl

set CC=clang-cl
set CXX=clang-cl

Setting intel icx fails with error npy_clongdouble definition is not compatible with C99 complex definition !
set CC=icx
set CXX=icx

By default numpy selected cl compiler + MKL during build with new intelOneAPI. icx is still a problem. same error.
So the Microsoft visual studio LLVM compilers are fixed, and also the default Microsoft visual studio cl compiler works.

For numpy 2.1.3 meson fails to detect icx:

File "D:\Python_Packages\numpy-2.1.3\vendored-meson\meson\mesonbuild\interpreter\interpreter.py", line 1522, in add_languages_for
          comp = compilers.detect_compiler_for(self.environment, lang, for_machine, skip_sanity_check, self.subproject)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "D:\Python_Packages\numpy-2.1.3\vendored-meson\meson\mesonbuild\compilers\detect.py", line 107, in detect_compiler_for
          env.coredata.process_compiler_options(lang, comp, env, subproject)
        File "D:\Python_Packages\numpy-2.1.3\vendored-meson\meson\mesonbuild\coredata.py", line 740, in process_compiler_options
          self.add_compiler_options(comp.get_options(), lang, comp.for_machine, env, subproject)
                                    ^^^^^^^^^^^^^^^^^^
        File "D:\Python_Packages\numpy-2.1.3\vendored-meson\meson\mesonbuild\compilers\c.py", line 547, in get_options
          raise RuntimeError('This is a transitory issue that should not happen. Please report with full backtrace.')
      RuntimeError: This is a transitory issue that should not happen. Please report with full backtrace.
      The Meson build system
      Version: 1.5.2
      Source dir: D:\Python_Packages\numpy-2.1.3
      Build dir: D:\Python_Packages\numpy-2.1.3\build
      Build type: native build
      Project name: NumPy
      Project version: 2.1.3

s-fleuren · 2026-04-16T14:37:21Z

Would it be possible if the changes to numpy/_core/include/numpy/npy_common.h from commit c9de672 were merged back?

For context, we are not building numpy from source, but do link it to our c application which is build with icx. This currently gives compilation errors because _Dcomplex is unknown, which is fixed by the npy_common.h patch.

rgommers · 2026-04-16T15:24:28Z

@s-fleuren thanks for the ping, this just fell through the cracks it looks like. Just to confirm, with only commit c9de672, does your application work fine?

s-fleuren · 2026-04-16T15:53:30Z

As far as I can see yes. We're still in the process of updating some other dependencies, so I can't run our full test suite yet. But the change to npy_common.h was enough to get the application to compile again, and basic Numpy operations work.

With these changes, downstream users wanting to use the Intel compilers can do so when linking against NumPy headers. Also include a build change for NumPy itself that is clearly correct: Intel compilers do not use `_Fcomplex` & co for complex types, but the regular C99 complex.h types. Taken over from PR 25044 Closes gh-25044 Closes gh-31337 (reports the same `_Fcomplex` issue) Co-authored-by: Matti Picus <matti.picus@gmail.com>

With these changes, downstream users wanting to use the Intel compilers can do so when linking against NumPy headers. Also include a build change for NumPy itself that is clearly correct: Intel compilers do not use `_Fcomplex` & co for complex types, but the regular C99 complex.h types. Taken over from PR 25044 Closes numpygh-25044 Closes numpygh-31337 (reports the same `_Fcomplex` issue) Co-authored-by: Matti Picus <matti.picus@gmail.com>

github-actions Bot added the 36 - Build Build related PR label Oct 31, 2023

lysnikolaou mentioned this pull request Oct 31, 2023

npy_cdouble related error when compiling 1.26.1 with Intel compilers on Windows #25034

Closed

seberg reviewed Nov 1, 2023

View reviewed changes

Comment thread numpy/_core/meson.build Outdated

BLD: Check Intel compiler when checking complex types in build

0ff966e

Fixes numpy#25034. use clearer syntax Co-authored-by: Sebastian Berg <sebastian@sipsolutions.net>

seberg force-pushed the check-intel-meson branch from 320cea0 to 0ff966e Compare November 6, 2023 13:43

seberg mentioned this pull request Nov 6, 2023

MAINT: default to C11 rather than C99, fix most build warnings with Clang 14 #25072

Merged

mattip added 2 commits November 7, 2023 08:21

BLD: add needed __INTEL_LLVM_COMPILER check

c9de672

BLD: avoid fast-math on intel compilers (taken from scipy)

33849a4

also add 'std=' flags and avoid a '-utf-8' flag by adding '-source-charset'

BLD: fix for clang-cl

5f00b1a

charris added the 09 - Backport-Candidate PRs tagged should be backported label Nov 10, 2023

rgommers reviewed Nov 13, 2023

View reviewed changes

charris removed the 09 - Backport-Candidate PRs tagged should be backported label Feb 5, 2024

Merge branch 'main' into check-intel-meson

dc29b3f

mattip mentioned this pull request Apr 3, 2024

BUG: numpy.any returns True given a boolean array of all False with the intel compiler #26197

Closed

mattip mentioned this pull request Apr 10, 2024

BUG: Cannot import numpy built with Intel oneAPI compiler icx and icpx #26257

Open

rgommers mentioned this pull request May 5, 2026

BLD/MAINT: improve support for Intel LLVM compilers #31389

Merged

seberg closed this in #31389 May 5, 2026

charris mentioned this pull request May 5, 2026

BLD/MAINT: improve support for Intel LLVM compilers #31391

Merged

Uh oh!

Uh oh!

Conversation

lysnikolaou commented Oct 31, 2023

Uh oh!

Uh oh!

seberg commented Nov 6, 2023

Uh oh!

lysnikolaou commented Nov 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg commented Nov 6, 2023

Uh oh!

mattip commented Nov 7, 2023

Uh oh!

mattip commented Nov 7, 2023

Uh oh!

lysnikolaou commented Nov 7, 2023

Uh oh!

mattip commented Nov 7, 2023

Uh oh!

mattip commented Nov 10, 2023

Uh oh!

lysnikolaou commented Nov 10, 2023

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

rgommers Nov 13, 2023

Choose a reason for hiding this comment

Uh oh!

mattip Nov 15, 2023

Choose a reason for hiding this comment

Uh oh!

rgommers Nov 15, 2023

Choose a reason for hiding this comment

Uh oh!

mattip Nov 15, 2023

Choose a reason for hiding this comment

Uh oh!

mattip commented Nov 15, 2023

Uh oh!

rgommers commented Nov 15, 2023

Uh oh!

charris commented Jan 4, 2024

Uh oh!

charris commented Feb 5, 2024

Uh oh!

lysnikolaou commented Mar 4, 2024

Uh oh!

mattip commented Mar 4, 2024

Uh oh!

lysnikolaou commented Mar 4, 2024

Uh oh!

rgommers commented Nov 25, 2024

Uh oh!

ashish-2022 commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s-fleuren commented Apr 16, 2026

Uh oh!

rgommers commented Apr 16, 2026

Uh oh!

s-fleuren commented Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

lysnikolaou commented Nov 6, 2023 •

edited

Loading

ashish-2022 commented Nov 25, 2024 •

edited

Loading