bump to latest hdf5 version and windows fortran support w/flang

[Krande]

I thought I might give fortran support for windows a try here as well (seeing as we've had some success in adding fortran support on windows using flang in mumps, mgis, mfront).

However, it failed locally in my initial attempts on windows with Fortran compiler requires either intrinsic functions SIZEOF or STORAGE_SIZE

If that is not a quick fix, I can take the fortran support out and put it in a different PR.

Checklist

• Used a personal fork of the feedstock to propose changes
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

[Krande]

@conda-forge-admin, please rerender

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[Krande]

@conda-forge-admin, please rerender

[Krande]

Hey @h-vetinari, I tried flang-new here in hopes of compiling HDF5 with fortran enabled on windows.

Unfortunately it fails during the configuration :( Any ideas where I ought to start looking? (I have posted a question about this on the flang-compiler slack channel also).

-- The Fortran compiler identification is LLVMFlang 18.1.3
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: D:/bld/hdf5_1713270629406/_build_env/Library/bin/flang-new.exe - skipped
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/C Compiler Compatibility
-- Verifying Fortran/C Compiler Compatibility - Success
-- Performing Test H5_FORTRAN_HAVE_SIZEOF
-- Performing Test H5_FORTRAN_HAVE_SIZEOF - Failed
-- Performing Test H5_FORTRAN_HAVE_C_SIZEOF
-- Performing Test H5_FORTRAN_HAVE_C_SIZEOF - Failed
-- Performing Test H5_FORTRAN_HAVE_STORAGE_SIZE
-- Performing Test H5_FORTRAN_HAVE_STORAGE_SIZE - Failed
-- Performing Test H5_FORTRAN_HAVE_ISO_C_BINDING
-- Performing Test H5_FORTRAN_HAVE_ISO_C_BINDING - Failed
-- Performing Test FORTRAN_HAVE_C_LONG_DOUBLE
-- Performing Test FORTRAN_HAVE_C_LONG_DOUBLE - Failed
-- Performing Test FORTRAN_C_LONG_DOUBLE_IS_UNIQUE
-- Performing Test FORTRAN_C_LONG_DOUBLE_IS_UNIQUE - Failed
-- Performing Test FORTRAN_C_BOOL_IS_UNIQUE
-- Performing Test FORTRAN_C_BOOL_IS_UNIQUE - Failed
-- Performing Test HAVE_ISO_FORTRAN_ENV
-- Performing Test HAVE_ISO_FORTRAN_ENV - Failed
CMake Error at config/cmake/HDF5UseFortran.cmake:128 (message):
  Fortran compiler requires either intrinsic functions SIZEOF or STORAGE_SIZE
Call Stack (most recent call first):
  CMakeLists.txt:1076 (include)

link to build logs

Based on

• Build HPC commonly-used Tools using LLVM Flang  llvm/llvm-project#56348
• https://forum.hdfgroup.org/t/cmake-error-with-aocc-flang-compiler/10208

I had hopes that this could work given that it seems LLVM flang compiles HDF5 on linux.

[Krande]

@conda-forge-admin, please rerender

[Krande]

Okay, here's another update.

Regarding the flang windows error. It seems the previous reported error was a simple CMake config forcing invalid flang comands. I added a patch for that which seems to work.

The error we're seeing now is

-- The Fortran compiler identification is LLVMFlang 18.1.3
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: C:/Work/miniforge3/envs/msvc-flang/Library/bin/flang-new.exe - skipped
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/C Compiler Compatibility
-- Verifying Fortran/C Compiler Compatibility - Success
-- Performing Test H5_FORTRAN_HAVE_SIZEOF
-- Performing Test H5_FORTRAN_HAVE_SIZEOF - Success
-- Performing Test H5_FORTRAN_HAVE_C_SIZEOF
-- Performing Test H5_FORTRAN_HAVE_C_SIZEOF - Success
-- Performing Test H5_FORTRAN_HAVE_STORAGE_SIZE
-- Performing Test H5_FORTRAN_HAVE_STORAGE_SIZE - Success
-- Performing Test H5_FORTRAN_HAVE_ISO_C_BINDING
-- Performing Test H5_FORTRAN_HAVE_ISO_C_BINDING - Success
-- Performing Test FORTRAN_HAVE_C_LONG_DOUBLE
-- Performing Test FORTRAN_HAVE_C_LONG_DOUBLE - Success
-- Performing Test FORTRAN_C_LONG_DOUBLE_IS_UNIQUE
-- Performing Test FORTRAN_C_LONG_DOUBLE_IS_UNIQUE - Success
-- Performing Test FORTRAN_C_BOOL_IS_UNIQUE
-- Performing Test FORTRAN_C_BOOL_IS_UNIQUE - Success
-- Performing Test HAVE_ISO_FORTRAN_ENV
-- Performing Test HAVE_ISO_FORTRAN_ENV - Success
-- Performing Test FORTRAN_CHAR_ALLOC
-- Performing Test FORTRAN_CHAR_ALLOC - Success
CMake Error at config/cmake/HDF5UseFortran.cmake:171 (list):
  list index: 2 out of range (-2, 1)
Call Stack (most recent call first):
  CMakeLists.txt:1076 (include)


CMake Error at config/cmake/HDF5UseFortran.cmake:178 (message):
  Failed to find available REAL KINDs for Fortran
Call Stack (most recent call first):
  CMakeLists.txt:1076 (include)


-- Configuring incomplete, errors occurred!

And by closer inspection of the CMakeConfigureLog.yaml

LINK: command "C:\\PROGRA~2\\MICROS~3\\2022\\BUILDT~1\\VC\\Tools\\MSVC\\1438~1.331\\bin\\Hostx64\\x64\\link.exe /nologo CMakeFiles\\cmTC_6dccc.dir\\testFortranCompiler1.f90.obj /out:cmTC_6dccc.exe /implib:cmTC_6dccc.lib /pdb:cmTC_6dccc.exe.dbg /version:0.0 /machine:x64 -stack:10000000 /INCREMENTAL:NO /subsystem:console ws2_32.lib wsock32.lib shlwapi.lib -libpath:C:/Work/miniforge3/envs/msvc-flang/Library/lib -libpath:C:/Work/miniforge3/envs/msvc-flang/Library/lib/clang/18/lib/windows /MANIFEST:EMBED,ID=1" failed (exit code 1120) with the following output:
        testFortranCompiler1.f90.obj : error LNK2019: unresolved external symbol _QMiso_fortran_envECinteger_kinds referenced in function _QQmain
        testFortranCompiler1.f90.obj : error LNK2019: unresolved external symbol _QMiso_fortran_envECreal_kinds referenced in function _QQmain\x0d
        testFortranCompiler1.f90.obj : error LNK2019: unresolved external symbol _QMiso_fortran_envEClogical_kinds referenced in function _QQmain\x0d
        cmTC_6dccc.exe : fatal error LNK1120: 3 unresolved externals\x0d
        ninja: build stopped: subcommand faile

it seems this might be related to llvm/llvm-project#77282.

[Krande]

Hey, here's a small update.

Once llvm/llvm-project#89403 is fixed it might be possible to compile HDF5 with fortran enabled on Windows using LLVM FLang. Related to HDFGroup/hdf5#4419

I'll just convert this PR to a draft for now (pending fix for LLVM flang).

[h-vetinari]

If I try to compile using a simple batch script it fails with the missing clang_rt.builtins.lib.

It sounds like the try_compile isn't using LDFLAGS, because otherwise there would be no problem.

[h-vetinari]

... which is because the FORTRAN_RUN CMake macro in HDF5 resets the flags as it wants. This is wrong and needs to be patched (or better: fixed upstream).

[Krande]

Hey @h-vetinari,

I was going to make a patch, so I have been trying a few different approaches locally where I force the appropriate flags to the FORTRAN_RUN (and the CMAKE build logs confirm that the flags are passed in). But it still seems like the could not open 'clang_rt.builtins.lib': no such file or directory is present no matter what I try..

Are you sure we aren't missing anything else?

I tried to simplify this by modifying the previous batch file example I made by doing this:

@echo off

setlocal enabledelayedexpansion

:: need to read clang version for path to compiler-rt
FOR /F "tokens=* USEBACKQ" %%F IN (`clang.exe -dumpversion`) DO (
    SET "CLANG_VER=%%F"
)

set CL_LIBDIR=%CONDA_PREFIX:/=/%/Library/lib/clang/!CLANG_VER:~0,2!/lib/windows

set "FFLAGS=-D_CRT_SECURE_NO_WARNINGS -D_MT -D_DLL --target=x86_64-pc-windows-msvc"
set "LDFLAGS=-fms-runtime-lib=dll -fuse-ld=lld"
set "LDFLAGS=%LDFLAGS% -Wl,-defaultlib:%CL_LIBDIR%/clang_rt.builtins-x86_64.lib"

set "CLI_ARGS=%FFLAGS% %LDFLAGS%"

flang-new %CLI_ARGS% read_and_integer_kinds.f90 -o read_and_integer_kinds.exe -v

call read_and_integer_kinds.exe

endlocal

That ends up with the following error output

flang-new version 19.1.0-rc2
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Work\Miniforge3\envs\flangdev\Library\bin
 "C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\bin\\flang-new" -fc1 -triple x86_64-pc-windows-msvc19.40.33811 -emit-obj -D _CRT_SECURE_NO_WARNINGS -D _MT -D _DLL -mrelocation-model pic -pic-level 2 -target-cpu x86-64 --dependent-lib=clang_rt.builtins-x86_64.lib -D_MT -D_DLL --dependent-lib=msvcrt --dependent-lib=FortranRuntime.dynamic.lib --dependent-lib=FortranDecimal.dynamic.lib -D_MSC_VER=1940 -D_MSC_FULL_VER=194033811 -D_WIN32 -D_M_X64=100 -resource-dir "C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\lib\\clang\\19" -mframe-pointer=none -o "C:\\Users\\KRISTO~1\\AppData\\Local\\Temp\\read_and_integer_kinds-87ae5c.o" -x f95-cpp-input read_and_integer_kinds.f90
 "C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\bin\\lld-link" -out:read_and_integer_kinds.exe "-libpath:C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Tools\\MSVC\\14.40.33807\\lib\\x64" "-libpath:C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Tools\\MSVC\\14.40.33807\\atlmfc\\lib\\x64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.26100.0\\ucrt\\x64" "-libpath:C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.26100.0\\um\\x64" "-libpath:C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\lib" /subsystem:console "-libpath:C:\\Work\\Miniforge3\\envs\\flangdev\\Library\\lib\\clang\\19\\lib\\windows" -nologo "-defaultlib:C:\\Work\\Miniforge3\\envs\\flangdev/Library/lib/clang/19/lib/windows/clang_rt.builtins-x86_64.lib" "C:\\Users\\KRISTO~1\\AppData\\Local\\Temp\\read_and_integer_kinds-87ae5c.o"
lld-link: error: could not open 'clang_rt.builtins.lib': no such file or directory
flang-new: error: linker command failed with exit code 1 (use -v to see invocation)
'read_and_integer_kinds.exe' is not recognized as an internal or external command,
operable program or batch file.

Is there something not working with the -defaultlib:<*.lib> flag? Or are we missing something else?

[h-vetinari]

The variables %FFLAGS% and %LDFLAGS% will already be populated correctly by the compiler activation. The error message

lld-link: error: could not open 'clang_rt.builtins.lib': no such file or directory

points to the "wrong" library (compare with LDFLAGS which points to [...]/clang_rt.builtins-x86_64.lib). Not sure how that happens.

But looking at some of the logs, it appears that lld-link is perhaps in windows-only mode, because otherwise it shouldn't complain about the following arguments:

        lld-link: warning: ignoring unknown argument '--target=x86_64-pc-windows-msvc'
        lld-link: warning: ignoring unknown argument '-fms-runtime-lib=dll'
        lld-link: warning: ignoring unknown argument '-fuse-ld=lld'
        lld-link: warning: ignoring unknown argument '-Wl,-defaultlib:D:\\bld\\hdf5_1723301788364\\_build_env/Library/lib/clang/19/lib/windows/clang_rt.builtins-x86_64.lib'

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.
I do have some suggestions for making it better though...

For recipe/meta.yaml:

• Jinja2 variable references are suggested to take a {{<one space><variable name><one space>}} form. See lines [64].

[Krande]

@conda-forge-admin, please rerender

[Krande]

@conda-forge-admin, please rerender

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found some lint.

Here's what I've got...

For recipe/meta.yaml:

• Recipe maintainer "hmaarrfk" does not exist
• Recipe maintainer "jakirkham" does not exist
• Recipe maintainer "gillins" does not exist
• Recipe maintainer "groutr" does not exist
• Recipe maintainer "ocefpaf" does not exist
• Recipe maintainer "astrofrog" does not exist
• Recipe maintainer "marcelotrevisani" does not exist
• Recipe maintainer "scopatz" does not exist
• Recipe maintainer "davidbrochart" does not exist
• Recipe maintainer "SylvainCorlay" does not exist
• Recipe maintainer "varlackc" does not exist
• Recipe maintainer "zklaus" does not exist

For recipe/meta.yaml:

• Jinja2 variable references are suggested to take a {{<one space><variable name><one space>}} form. See lines [64].

[Krande]

Hey @h-vetinari and @mjklemm. I thought I would revisit this.

Seeing as the HDF5 configuration still fails on the kinds check (see #217 (comment) for reference) on windows.

@mjklemm Did you manage to compile hdf5 on windows using a locally compiled version of flang? If so I might want to test compiling it for myself just to see what we're doing differently in our conda-forge build.

I thought I would simply skip this check by simply hardcoding the supported real and integers by flang. By doing something like this I managed to finish the configuration step, and start compiling!

if (CMAKE_Fortran_COMPILER MATCHES "flang")
    message(STATUS "PROG_OUTPUT=${PROG_OUTPUT}. Force override with \"1,4,8;4,8,16;33;3;3;4;1,4,8;\"")
    set(PROG_OUTPUT "1,4,8;4,8,16;33;3;3;4;1,4,8;")
else ()
    # Convert the string to a list of strings by replacing the carriage return with a semicolon
    string (REGEX REPLACE "[\r\n]+" ";" PROG_OUTPUT "${PROG_OUTPUT}")
endif ()

It runs for a bit until the compilation fails due to error: use of undeclared identifier 'real_C_LONG_DOUBLE_f'.
You guys got any ideas for how to get past this?

[401/3074] Building C object fortran\src\CMakeFiles\hdf5_f90cstub-shared.dir\H5_f.c.obj
FAILED: fortran/src/CMakeFiles/hdf5_f90cstub-shared.dir/H5_f.c.obj 
%BUILD_PREFIX%\Library\bin\clang-cl.exe  /nologo -DH5_BUILT_AS_DYNAMIC_LIB -D_BIND_TO_CURRENT_VCLIBS_VERSION=1 -D_CONSOLE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -Dhdf5_f90cstub_shared_EXPORTS -I%SRC_DIR%\src -I%SRC_DIR%\src\H5FDsubfiling -I%SRC_DIR%\build\src -I%SRC_DIR%\build\fortran -I%SRC_DIR%\build\fortran\shared -I%PREFIX%\Library\include -w  /DWIN32 /D_WINDOWS /O2 /Ob2 /DNDEBUG -MD -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var -Wno-c++-compat /W3 /wd4100 /wd4706 /wd4127 /showIncludes /Fofortran\src\CMakeFiles\hdf5_f90cstub-shared.dir\H5_f.c.obj /Fdfortran\src\CMakeFiles\hdf5_f90cstub-shared.dir\ -c -- %SRC_DIR%\fortran\src\H5_f.c
%SRC_DIR%\fortran\src\H5_f.c(187,16): error: use of undeclared identifier 'real_C_LONG_DOUBLE_f'
  187 |     if (sizeof(real_C_LONG_DOUBLE_f) == sizeof(float)) {
      |                ^
%SRC_DIR%\fortran\src\H5_f.c(191,21): error: use of undeclared identifier 'real_C_LONG_DOUBLE_f'
  191 |     else if (sizeof(real_C_LONG_DOUBLE_f) == sizeof(double)) {
      |                     ^
%SRC_DIR%\fortran\src\H5_f.c(196,21): error: use of undeclared identifier 'real_C_LONG_DOUBLE_f'
  196 |     else if (sizeof(real_C_LONG_DOUBLE_f) == sizeof(long double)) {
      |                     ^
3 errors generated.
[402/3074] Building C object fortran\src\CMakeFiles\hdf5_f90cstub-shared.dir\H5Af.c.obj
[403/3074] Building C object fortran\src\CMakeFiles\hdf5_f90cstub-shared.dir\H5Df.c.obj
ninja: build stopped: subcommand failed.

from https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1065872&view=logs&j=0ad3a718-cdde-5a77-9020-8beac94f6974&t=eda7a7f8-f584-55f5-338d-6077df40dfcc&l=2005

Update:

Okay, I think I solved it. Or at least I managed to get past the error message by adding the following to H5f90.h

#ifndef real_C_LONG_DOUBLE_f
#define real_C_LONG_DOUBLE_f long double
#endif

The next error is related to BIND(C) statements. Hm, are BIND(C) not supported by flang on windows?

C:\bld\flang-split_1725521580766\work\flang\lib\Optimizer\CodeGen\Target.cpp:102: not yet implemented: passing VALUE BIND(C) derived type for this target
LLVM ERROR: aborting

from https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1065902&view=logs&j=0ad3a718-cdde-5a77-9020-8beac94f6974&t=eda7a7f8-f584-55f5-338d-6077df40dfcc&l=2058

Or maybe it's BIND(C) and the use of passing argument by value like so?

subroutine mysub(x) bind(c)
    integer, value :: x  ! x is passed by value, not by reference
end subroutine mysub

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.
I do have some suggestions for making it better though...

For recipe/meta.yaml:

• Jinja2 variable references are suggested to take a {{<one space><variable name><one space>}} form. See lines [65].

[h-vetinari]

The not yet implemented message is unfortunately a pretty clear indicator that said fortran language feature is not yet available in flang. You can retry with flang 20 in ~February/March.

[Krande]

@h-vetinari Yup, it looks like it. I can raise an issue over at the llvm-project to see if it is on their radar. In the meantime to progress development on windows fortran support for hdf5 I opened #237 where I try to use the latest m2w64 libs. It looks quite promising from the local tests I did.

[Krande]

@h-vetinari Seems like this is already fixed in the flang main branch (hopefully on track for v20). See llvm/llvm-project#114035 (comment). I'll try to test this asap. If it works, then we can consider issueing a pre-release of flang v20 on a non-main conda-forge label.

[h-vetinari]

If it works, then we can consider issueing a pre-release of flang v20 on a non-main conda-forge label.

This is a nontrivial effort (not extreme either, but it needs a strong reason to do it). I probably won't get around to doing it, but if you want, you can do the following:

• pick a commit on LLVM main that looks OK (highly subjective of course)
• Take https://github.com/conda-forge/llvmdev-feedstock, merge main into dev, resolving all conflicts in favour of main (with the only exception of channel_targets: in CBC) then rerender
• do something like conda-forge/llvmdev-feedstock@61a9a86 for the new commit you've chosen
• open a PR that targets the dev branch and get it green (you might need to rebase the patches carried on the feedstock)
• if it's green and looks OK, I'll merge
• repeat the same exercise for all the components necessary until you get to flang-activation

You can expect this process to take at least a week (not work hours, but iterations to get things building and waiting for CI). I can help out if you're stuck somewhere, but that's about as much as I can promise for now.