Ongoing work on compiling code aster v16.4.X #16
Comments
|
Hi Kristoffer,
First of all, thank you for your contribution to the community! I am glad
that someone is working for Code_Aster in the conda-forge environment :)
I will try to look into your work as soon as possible and give you all the
feedback that I can
As you already found out, I spent some time recently with the packaging of
code_aster in conda-forge (I was asked by a research center to do so as
they use it for they research project) and I could contribute some packages
:
https://anaconda.org/conda-forge/code-aster (github
<https://github.com/conda-forge/code-aster-feedstock>)
https://anaconda.org/conda-forge/mfront
https://anaconda.org/conda-forge/miss3d
https://anaconda.org/conda-forge/libmed
(as well as others like freefem, getfem, ...)
I have currently different targets :
- add newer versions of python interpreter --> segfault issue
- add newer versions of code_aster (15.x, 16.x) --> build issue detected by
conda forge bots <https://github.com/conda-forge/code-aster-feedstock/pulls>
- add a windows (missing hdf5 fortran
<conda-forge/hdf5-feedstock#203>) / macos variant
- make all code_aster tests pass (for example this one
<https://github.com/conda-forge/code-aster-feedstock/blob/58e8378203915649ffe6928a4501966f027d13ee/recipe/meta.yaml#L85>
)
- add a parallel (mpi) variant
I do not know yet your current knowledge in building "conda-forge packages"
: I am more than willing to provide all the help that I can and save you as
much time as possible :-)
Conda-forge is in my opinion a step forward (both in complexity and final
result) with respect to creating a simple "conda" package, we can talk
about that if this interests you. If you are willing, I can add you among
the package maintainers so that you can contribute directly to those
projects.
First advice (you may already know that, in this case sorry for pointing
out) : you should work on a fork on your personal repository
<https://conda-forge.org/docs/maintainer/updating_pkgs.html#forking-and-pull-requests>
.
Second advice : use mamba, not conda. Faster, much faster. Also much better
dependency error reportings ;) Once installed, they are almost identical to
use (you can simply use "mamba" instead of "conda"). But you are free to do
as you prefer, of course.
Third advice : build locally before committing, this will save you time, I
can show you how if it helps you.
I have an hard time moving to a newer version of the python interpreter (I
really need to have a >=3.10 version because of dependencies constraints) :
it works perfectly using my local python but it doesn't with the one
provided with conda forge (as I wrote, there are additional constraints in
the forge to assure mutual compatibility among packages). I did isolate the
issue in some python parts of Code_Aster which I cannot find in open source
(I think that Matthieu coded those parts, I should ask him one of these
days).
I suspect that something is wrong with the code I had to put here in the
package because I could not find anywhere else (and it is required ad build
and at runtime):
https://github.com/ldallolio/code-aster-feedstock/tree/main/recipe/contrib/asrun
Code_Aster segfaults when building the command catalog, similar to this one
:
https://code-aster.org/forum2/viewtopic.php?id=1708
Please let me know I can help you, I will do my best :)
All the best,
Luca
Il giorno mar 22 ago 2023 alle ore 08:08 Kristoffer Andersen <
***@***.***> ha scritto:
… Hey @ldallolio <https://github.com/ldallolio>,
I have done some work on making a conda package for code aster v16.4.2
(with help from Mathieu Courtois). You can check the progress here
https://gitlab.com/codeaster/src/-/issues/1, The source code for all of
the compilation work currently done on v16.4.2 can be found here
<https://github.com/Krande/condapackaging/tree/code-aster/16.4.2/src/code_aster>
At the moment I've compiled and tested the v16.4.2 package for python 3.9,
3.10 and 3.11 and I am currently in the process of debugging the 5-10 %
failing tests (150-200 of 2182 tests are failing due to various errors).
Once all (or a sufficient number of) the tests pass, my plan is to create
a PR here and at the same time add the dependencies that currently aren't
on conda-forge (such as mgis
<https://github.com/thelfer/MFrontGenericInterfaceSupport> which I have
discussed with Thomas Helfer).
I would really appreciate any thoughts you might have on things I should
consider before making the PR, or if you have feedback/help you can provide
on the current state of compiling code please let me know?
Best Regards
Kristoffer
—
Reply to this email directly, view it on GitHub
<#16>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDPXKWLOYNUNLQEXFXTXWREENANCNFSM6AAAAAA3ZMAFE4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@ldallolio Great to hear! It seems our targets are aligned perfectly (next up on my list is also mpi and windows support of code aster) :) I have also converted completely to the mamba-bandwagon :) And I've found I also struggled with compiling the element catalogue I do have some experience with maintaining packages on conda-forge already, so hopefully I'll be able to contribute. So if you add me as a maintainer on code-aster when you have time I'll happily try to contribute as best I can! And maybe you can add me as a maintainer on the dependencies (such as libmed, mfront) as well? Btw. I have noticed that the latest (or at least among the latest) Either way, I added a test like this on my I will add a todo-list in my PR here of all the modifications I had to do in compiling the dependencies so that we can link them with PR's on the different feedstocks. Hopefully by the end of this week I will have solved most of the failing tests and I will make a PR! By the way, the 16.4.2 variant passed the |
|
Hi Kristoffer,
I am happy to hear that you have previous experience in conda-forge, things
will be quicker and simpler this way, glad to have you onboard!
I just added you as a maintainer in some projects (code-aster, libmed,
mfront, miss3d), please let me know if I can do more.
Thanks for pointing out boa, did not know it, seems great!
No idea about the cause of "missing MEDlibraryNumVersion", we definitely
need more tests there. It is great to know that the issue was fixed in
later versions, this really helps.
I will try to have a look at things from here, please let me know if I can
help in any other way, I will be happy to!
All the best,
Luca
Il giorno mar 22 ago 2023 alle ore 12:18 Kristoffer Andersen <
***@***.***> ha scritto:
… @ldallolio <https://github.com/ldallolio> Great to hear! It seems our
targets are aligned perfectly (next up on my list is also mpi and windows
support of code aster) :)
I have also converted completely to the mamba-bandwagon :) And I've found
boa for running fast conda-build iterations locally help significantly to
speed up workflows (even though it requires a boa convert
meta.yaml>recipe.yaml step)!
I also struggled with compiling the element catalogue catalo for any
interpreters > 3.9 for code aster <15.8.0. It was Mathieu who recommended
me to focus my efforts on v16.4.* (which is/will be the new stable version
apparently).
I do have some experience with maintaining packages on conda-forge
already, so hopefully I'll be able to contribute. So if you add me as a
maintainer on code-aster when you have time I'll happily try to contribute
as best I can! And maybe you can add me as a maintainer on the dependencies
(such as libmed, mfront) as well?
Btw. I have noticed that the latest (or at least among the latest) libmed
packages on conda-forge does not contain any reference to the
MEDlibraryNumVersion in libmed.so causing the code-aster compilation to
fail. I am not sure why though (it might be there is something wrong in the
code-aster waf)?
Either way, I added a test like this on my libmed packaging to ensure
that the MEDlibraryNumVersion is found in the compiled file:
https://github.com/Krande/condapackaging/blob/code-aster/16.4.2/src/code_aster/libmed/meta.yaml#L44-L49
test:
imports:
- med
commands:
- |
if [[ -z $(nm -D $CONDA_PREFIX/lib/libmed.so | grep MEDlibraryNumVersion) ]]; then
echo "MEDlibraryNumVersion is not present in libmed.so";
exit 1;
fi
I will add a todo-list in my PR here of all the modifications I had to do
in compiling the dependencies so that we can link them with PR's on the
different feedstocks?
By the way, the 16.4.2 variant passed the mfron01a test:
[image: image]
<https://user-images.githubusercontent.com/9642232/262317407-6eb4a82a-048e-45ac-a82b-8d48e3d8868f.png>
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDJ26JR5XXMKOU7FLODXWSBORANCNFSM6AAAAAA3ZMAFE4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Hey @ldallolio! Here's an update. I have now also compiled an MPI-build in addition to the sequential build for linux. Mathieu Courtois will attempt to compile Code Aster in a more updated Singularity container based on ubuntu 22.04 to help with debugging the remaining failed tests (see https://gitlab.com/codeaster/src/-/issues/1#note_1553448646). This will hopefully solve much of the remaining issues. For now the test results are
I am still waiting on conda-forge to accept my pull request for I haven't started looking at the work to be done for the windows compilation yet. But I think we'll be able to solve that in time as well :) Best Regards |
|
Hi Kristoffer,
Sorry about the late answer (I had some work going on at our old house in
Italy, wood and wall repairs among other things), now I am back to France
and I would like to speed up the process!
Thanks for your great news, a MPI version would be really helpful for a lot
of people, I think. Most of the tests seem to pass, which is wonderful.
I see that your mgis request is still open, hope it will be accepted as
soon as possible! Do you wish that we start working on a code_aster 16.4
upgrade ? I could start to integrate from your repository if you wish
Thank you once again, all the best
Luca
Il giorno sab 16 set 2023 alle ore 15:17 Kristoffer Andersen <
***@***.***> ha scritto:
… Hey @ldallolio <https://github.com/ldallolio>!
Here's an update. I have now also compiled an MPI-build in addition to the
sequential build for linux.
Mathieu Courtois will attempt to compile Code Aster in a more updated
Singularity container based on ubuntu 22.04 to help with debugging the
remaining failed tests (see
https://gitlab.com/codeaster/src/-/issues/1#note_1553448646). This will
hopefully solve much of the remaining issues.
For now the test results are
Version python mpi Numpy hdf5 Failed Tests Total Tests
16.4.2 3.11 no 1.23 1.10.6 8 2182
16.4.2 3.11 no 1.25.2 1.10.6 11 2182
16.4.2 3.11 yes 1.23 1.10.6 256 2252
16.4.2 3.11 yes 1.25.2 1.10.6 259 2252
I am still waiting on conda-forge to accept my pull request for mgis (conda-forge/staged-recipes#23818
(comment)
<conda-forge/staged-recipes#23818 (comment)>).
Once that gets accepted I will start to roll out PR's for the remaining
tests.
I haven't started looking at the work to be done for the windows
compilation yet. But I think we'll be able to solve that in time as well :)
Best Regards
Kristoffer
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDOERHLSUDJ6RBQ2HVTX2WRH7ANCNFSM6AAAAAA3ZMAFE4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
No worries :) Yes, it would be really great if you can start the work on integrating what I've done so far into the official conda-forge feedstocks! I have most certainly made mistakes and done strange things along the way in the different meta yaml's and build scripts. So feel free to ping me in the PR's whenever you come across anything you find that seems out of place :) We should also decide whether or not we want to create a dedicated conda-forge package for I'll try to ping the conda-forge admins again to see if they can expedite the PR review for Oh, and by the way! I have made minor progress on the windows variant of Code_Aster. But I had to use a newer Mingw compiler version than currently is available on conda-forge to successfully compile HDF5. From what I can tell (and the feedback I got here there is an ongoing process to update the windows fortran compilers. I got as far as getting working compilations of HDF5 and libmed. Medcoupling failed for some reason. But I digress. Let's fix the linux variants first! :) Best Regards |
|
Il giorno dom 8 ott 2023 alle ore 16:11 Kristoffer Andersen <
***@***.***> ha scritto:
No worries :) Yes, it would be really great if you can start the work on
integrating what I've done so far into the official conda-forge feedstocks!
I will do my best :)
I have most certainly made mistakes and done strange things along the way
in the different meta yaml's and build scripts. So feel free to ping me in
the PR's whenever you come across anything you find that seems out of place
:)
I will certainly share any question with you, thanks for all you have done
once again!
We should also decide whether or not we want to create a dedicated
conda-forge package for medcoupling or if we want to embed it with the
Code Aster package. Maybe it would be good to have a dedicated package? If
so we should also start the process of adding that package to conda-forge
as well. Besides medcoupling I think most of the other dependencies
already exist.
I would go for a separate package : I have seen many projects using
medcoupling only. Also, they have different licensing terms I think
I'll try to ping the conda-forge admins again to see if they can expedite
the PR review for mgis
If I can do anything to help just ask but it also took some months to me to
get the projets in conda_forge, thinks speed up after the project is in
Oh, and by the way! I have made minor progress on the windows variant of
Code_Aster. But I had to use a newer Mingw compiler version than currently
is available on conda-forge to successfully compile HDF5. From what I can
tell (and the feedback I got here
<https://matrix.to/#/!SOyumkgPRWoXfQYIFH:matrix.org/$169590581923890qByPT:matrix.org?via=matrix.org&via=gitter.im&via=cadair.com>
there is an ongoing process to update the windows fortran compilers.
That's also great news! Those compilers are pretty old indeed.
I got as far as getting working compilations of HDF5 and libmed.
Medcoupling failed for some reason. But I digress. Let's fix the linux
variants first! :)
I agree, first things first :)
Best Regards
Kristoffer
All the best,
Luca
… —
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDOVZN3OA3QYEA6QXMTX6KYCNAVCNFSM6AAAAAA3ZMAFE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJSGAZTQMJWHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Hi Kristoffer, |
|
That's great! Let me know if you run into any issues on the way and I'll answer as best I can, Best Regards |
|
Hey @ldallolio, just letting you know that I'll create a PR for a |
|
Hi Kristoffer,
I am still working on merging your contributions to conda forge for
code_aster so a medcoupling packager would be very welcome indeed!
Thanks in advance, if I can help don't hesitate to ask!
Have a nice day,
Luca
Il giorno lun 30 ott 2023 alle ore 09:58 Kristoffer Andersen <
***@***.***> ha scritto:
… Hey @ldallolio <https://github.com/ldallolio>, just letting you know that
I'll create a PR for a medcoupling to conda-forge now. Let me know if you
have already started the process and I'll focus my efforts elsewhere.
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDJCUWQ3EEYKN3AX6WLYB5T3VAVCNFSM6AAAAAA3ZMAFE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUG42TAMBTHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Btw. The mgis package is now merged! Do you want to be added as a maintainer? |
|
Kudos !!!! \(^__^)/
Il giorno lun 30 ott 2023 alle ore 10:24 Kristoffer Andersen <
***@***.***> ha scritto:
… Btw. The mgis <https://github.com/conda-forge/mgis-feedstock> package is
now merged!
- https://anaconda.org/conda-forge/mgis
Do you want to be added as a maintainer?
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDPRC5FNMKRDIJTV2A3YB5W4PAVCNFSM6AAAAAA3ZMAFE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUG44TEMRTGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Medcoupling PR up and running conda-forge/staged-recipes#24378 |
Krande
changed the title
|
Hi Kristoffer,
I could progress a bit towards a new code_aster 16 package these last few
days, I prepared a not-yet-working-but-almost-there pull-request :
#22
I did something slightly different from your "condapackaging" solution to
make it acceptable for conda-forge : metis and mumps are built and
statically linked with code_aster
Also, code_aster and medcoupling are installed into $PREFIX in a way that
should work without activating environments.
The package builds locally (using "./build-locally.py --debug") and tests
are ok in the build environment.
I could also build a local package with "mamba build -c conda-forge
./recipe" and install it locally using "mamba create -n testca16 -c
conda-forge -c local code-aster=16.4.9" and most tests are passing
(excluding umat, which is not a big deal)
The main issue seems to be that "./build-locally", without --debug, fails
(both locally and in the pull request checks
<https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/822354/logs/20>
).
The issue is during IMPR_RESU, because of med and/or medcoupling not
working properly :
"La bibliothèque "MED" n'est pas installée dans cette version de
Code_Aster."
I think that the issue is some missing med and/or medcoupling runtime
dependencies. Maybe you have already seen this ?
I will keep looking into it next week but I wanted to share this.
The only libraries that I cannot see at test time are astor and
pybind11-stubgen
(maybe they are needed by medcoupling at runtime ?)
I will try adding those and see what happens :)
Also, I tried linking statically med and hdf5 but failed, because
code_aster needs medcoupling now and medcoupling does not link with a
static med (even if recompiled with -fpic), I left some commented
instructions in the build.sh if you want to have a look
All the best,
Luca
Il giorno lun 30 ott 2023 alle ore 12:19 Kristoffer Andersen <
***@***.***> ha scritto:
… Medcoupling PR up and running conda-forge/staged-recipes#24378
<conda-forge/staged-recipes#24378>
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDJO2DWJOHHBW55DUPTYB6ELHAVCNFSM6AAAAAA3ZMAFE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUHE3TMOBRG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Hi Kristoffer,
I merged the pull request, since it will pass some tests at least with
python 3.8 and 3.9
https://anaconda.org/conda-forge/code-aster/files
Still not working (when MED tries to write files in IMPR_RESU) on python
3.10 and 3.11
I think I will also try to merge the libmed PR so that we have a new hdf,
maybe it will help...
If you have any idea about this it would be very welcome
All the best,
Luca
Il giorno dom 12 nov 2023 alle ore 16:56 Luca Dall'Olio <
***@***.***> ha scritto:
… Hi Kristoffer,
I could progress a bit towards a new code_aster 16 package these last few
days, I prepared a not-yet-working-but-almost-there pull-request :
#22
I did something slightly different from your "condapackaging" solution to
make it acceptable for conda-forge : metis and mumps are built and
statically linked with code_aster
Also, code_aster and medcoupling are installed into $PREFIX in a way that
should work without activating environments.
The package builds locally (using "./build-locally.py --debug") and tests
are ok in the build environment.
I could also build a local package with "mamba build -c conda-forge
./recipe" and install it locally using "mamba create -n testca16 -c
conda-forge -c local code-aster=16.4.9" and most tests are passing
(excluding umat, which is not a big deal)
The main issue seems to be that "./build-locally", without --debug, fails
(both locally and in the pull request checks
<https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/822354/logs/20>
).
The issue is during IMPR_RESU, because of med and/or medcoupling not
working properly :
"La bibliothèque "MED" n'est pas installée dans cette version de
Code_Aster."
I think that the issue is some missing med and/or medcoupling runtime
dependencies. Maybe you have already seen this ?
I will keep looking into it next week but I wanted to share this.
The only libraries that I cannot see at test time are astor and pybind11-stubgen
(maybe they are needed by medcoupling at runtime ?)
I will try adding those and see what happens :)
Also, I tried linking statically med and hdf5 but failed, because
code_aster needs medcoupling now and medcoupling does not link with a
static med (even if recompiled with -fpic), I left some commented
instructions in the build.sh if you want to have a look
All the best,
Luca
Il giorno lun 30 ott 2023 alle ore 12:19 Kristoffer Andersen <
***@***.***> ha scritto:
> Medcoupling PR up and running conda-forge/staged-recipes#24378
> <conda-forge/staged-recipes#24378>
>
> —
> Reply to this email directly, view it on GitHub
> <#16 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAPAUDJO2DWJOHHBW55DUPTYB6ELHAVCNFSM6AAAAAA3ZMAFE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUHE3TMOBRG4>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
|
Great work! Feel free to merge the libmed PR! I'll be back from vacation on Friday. Will take a look at the state of things then :) Best Regards |
|
Hey @ldallolio, I thought I might share that I have gotten further on compiling Code Aster for windows. There's still much to do, but now that most of the dependencies are compiled for windows on conda-forge You can follow the progress here: https://gitlab.com/krande/src/-/commits/win-support I have tried to list the different steps required for a quick and easy local debugging of the waf compilation here: https://gitlab.com/krande/src/-/blob/win-support/conda/conda_win_compilation.md Let me know if you have any thoughts on the matter :) |
|
Another update ,@ldallolio. I've managed to update the WAF build to pass the compilation all the individual c, c++ and fortran files and it has started on the linking phase of each library. I am seeing several linking issues that I need to resolve. Everything seems to be related to the linker unable to find fortran subroutines. For the linking phase and creation of Do you have any experience with this? |
|
Hi @Krande |
|
@ldallolio Yeah, name mangling is what I am currently suspecting as well. I have tried to force lowercase names for the fortran compiler using /names:lowercase, and when inspecting the symbols using I am also investigating if it might be about build order. As of now it always tries to link and build Any extra pair of eyes is more than welcome. I've been stuck on this for a few days now and haven't really progressed much :( |
|
Hi @Krande I just followed your instructions (Installing Intel Fortran etc.) but I ran into an issue, maybe you know how to get over it : it does not seem to find numpy headers during waf configure (see part of my Weird thing is : it seems to be looking in the right place on my disk : The include file is there too but I think that clang is not adding that directory to the include search path ? Did you have a similar issue in your build ? Thanks! `
|
|
Hey @ldallolio, I made a few commits just now to fix some issues in the Linux compilation. I believe I managed to get the windows build working again, so you could check if you are using the latest commit I made? (or I might be wrong and the windows configure step has regressed). See https://gitlab.com/krande/src/-/commit/b82346b495904137553333c408d2efeabf1f83dc for the last known working commit. And you can also doublecheck the github action I've set up for testing the windows builds here -> https://github.com/Krande/condapackaging/actions/workflows/ci-code-aster-src-win.yml) Additionally, could you do It also seems you are building a release version. I have only tested a debug version. So There might be some bugs in msvc win config that currently only works for the debug build. |
|
Hi @Krande , This is my mamba list result:
I looked into the gitlab actions, there seem to be some errors here and there but it might be something irrelevant and different. Another difference is that I use Visual Studio 2019 but here it seems to me that it is configure that is failing to find numpy headers (maybe a missing INCLUDE somewhere?) Thanks for looking into this, all the best |
|
That looks like your base mamba env. Can you do (if you haven't already, the setup I created will need to activate the I'll continue working on windows support today. So hopefully i will be able to help solve your issue! |
|
Hi @Krande , I just did it :-) here is the result on my pc :
Thanks for looking into this if you can! |
|
Okay, I see only minor differences in our python env's. Main difference is that I moved to python 3.11 (as we don't compile Code Aster for python 3.12 on linux yet). But I don't think that's the issue you're seeing. When comparing your original error message and the includes you are using, it seems you are missing an include path. From my numpy include check From yours Btw. It should be safe to move to the latest commit Hopefully, moving to the latest commit fixes it (and doing a If I where to speculate as to why the include path is not added, it might be that the Intel MKL deprecation warning is causing an issue with the numpy include path export. However, it's strange that I don't have that same deprecation warning when using the same numpy version (although on a different python version). Btw. How do you run waf configure? Are you using the "conda_build.bat"? |
|
Btw. I've made a little progress. Or rather, I've landed on the conclusion that trying to force the compilation of bibfor.dll first is not solving the issue. I ended up missing symbols from It seems more likely that it's a name mangling issue and/or a dependency resolving issue (either in the current WAF setup or the way I'm using the linker LINK.exe). I am going over the 30 unresolved symbols in the bibc.dll linking and it seems they all share the same definition setup. Take the following example missing symbol aster_core_module.cutgtmeCALL_UTGTME( &nbpar, nompar, valres, &codret );From #define CALL_UTGTME( a, b, c, d ) CALLPSPP( UTGTME, utgtme, a, b, c, d )
extern void DEFPSPP( UTGTME, utgtme, ASTERINTEGER *, char *, STRING_SIZE, ASTERDOUBLE *,
ASTERINTEGER * );Here is the definition of DEFPSPP and CALLPSPP From #ifdef ASTER_STRLEN_AT_END
...
#define DEFPSPP(UN,LN,a,b,lb,c,d) STDCALL(UN,LN)(a,b,c,d,lb)
#define CALLPSPP(UN,LN,a,b,c,d) F_FUNC(UN,LN)(a,b,c,d,strlen(b))
...
/* Appels et signatures avec strlen juste après le pointeur de chaine */
#else
...
#define DEFPSPP(UN,LN,a,b,lb,c,d) STDCALL(UN,LN)(a,b,lb,c,d)
#define CALLPSPP(UN,LN,a,b,c,d) F_FUNC(UN,LN)(a,b,strlen(b),c,d)
...
#endifCommon References (bibc/include/definition.h)#ifndef ASTER_NO_UNDERSCORE
#define F_FUNC( UN, LN ) LN##_
#else
#define F_FUNC( UN, LN ) LN
#endif/* STDCALL for "old" windows version */
#ifdef ASTER_ADD_STDCALL
#define STDCALL( UN, LN ) __stdcall F_FUNC( UN, LN )
#else
#define STDCALL( UN, LN ) F_FUNC( UN, LN )
#endifIt seems on linux, gcc handles this "behind the scenes". Ie. I can't see that all the Maybe the clue is to utilize the |
|
@ldallolio Hey. Good news. I believe I know how to solve the windows build. It seems to be a circular dependency issue. Ie. bibfor, bibc and bibcxx rely on symbols in eachother. After reading this it appears circular dependencies like this is fine on UNIX since they rely on So I need to create a new task which performs the ".lib+.exp generation step" that runs right before the first linking of the dll's. I tested this manually and it seems like it now manages to resolve the missing symbols. Now all I need to do is to make a conditional switch in the Code Aster WAF build for windows that adds this extra .lib generation step right before the linking phase. I'll have to figure out a reasonable order for the dependencies, but I'm thinking something like this
Obviously then it's a matter of enabling fortran on all the dependencies (which I've begun slightly in parallel here) :) AND of course it's the case that currently I have been compiling Code Aster using the intel fortran compiler |
|
Hi @Krande |
|
Hi @Krande , I cannot help so much in this phase, if I can do anything useful please let me know, I will do my best! Have a nice day, |
|
Hey @ldallolio, I am assuming you have installed the default c++ development tools using the visual studio installer? For reference I am using VS2022 build tools and I have installed the individual MSVC c++ version components
However, I don't recognize the error you're seeing. So I doubt that its due to a different MSVC c++ version. I believe that is the supported version by Can you share the full cmd terminal log from running the bat file? And maybe also how you call the batch file? (if you prefer you can send the logs to me directly if you don't want to share publicly). Btw. I believe I have a functional WAF build which generates the necessary .lib files and correctly rearranges the build order. Right now I have only a single unresolved reference I have also managed to compile HDF5 and LIBMED locally using the intel Fortran compiler on windows. Which means that all dependencies with fortran symbols on windows are possible to compile with MSVC abi compatibility. We are also getting help on fixing LLVM Flang to support HDF5 in HDFGroup/hdf5#4419 which hopefully will let us compile all of these soon on conda-forge as well! |
|
Hey @ldallolio, I believe I have fixed most of the issues with the Code Aster build itself. The issue I'm trying to figure out right now, is why we're not exporting fortran symbols for libmed (I managed to compile libmed successfully with fortran enabled. However, the symbols aren't exported :( ) I have tried to use I have tried different cmake flags and changes at https://github.com/Krande/med-copy/tree/conda-dev Let me know if you have any ideas? Best Regards |
|
Hi @Krande,
It is indeed surprising... The only thing I could find was to add a
/verbose to the link command, I suppose you already tried that one... Are
you using the intel fortran compiler ? Or flang ?
Maybe adding ENABLE_EXPORTS dans le Cmakefile ? As you know, I know almost
nothing on windows builds... :-(
https://cmake.org/cmake/help/v3.5/prop_tgt/ENABLE_EXPORTS.html
Another silly idea : we could create a minimal example (an hello world
fortran library), build it directly and/or with cmake and send it to the
corresponding forum ?
I can work on that if you think that it could help :)
All the best,
Luca
Il giorno mar 23 apr 2024 alle ore 17:20 Kristoffer Andersen <
***@***.***> ha scritto:
… Hey @ldallolio <https://github.com/ldallolio>, I believe I have fixed
most of the issues with the Code Aster build itself.
The issue I'm trying to figure out right now, is why we're not exporting
fortran symbols for libmed (I managed to compile libmed successfully with
fortran enabled. However, the symbols aren't exported :( )
I have tried to use dumpbin /symbols medfwrap.lib, but there aren't any
symbols in there (the file is only 176kB).
Similarly for the created dll, when I do dumpbin /exports medfwrap.dll I
cannot find any references to the fortran symbols I'm looking for :(
I have tried different cmake flags and changes at
https://github.com/Krande/med-copy/tree/conda-dev
Let me know if you have any ideas?
Best Regards
Kristoffer
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDNPANCBDU4ZD2O254DY6Z33ZAVCNFSM6AAAAAA3ZMAFE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZSGY3TIMZXGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I am using the intel fortran compiler. (flang failed unfortunately).
Thanks for the tip! I've tried something similar
Not a silly idea at all! That's likely our path of least resistance wrt solving it. I have made a minimal example for the circular dependency problem using I'll start tomorrow on adding a minimal cmake example with the missing fortran symbols in that repository.
Sure, I'll take all the help I can get :) But maybe you would like to check out the minimal WAF build example to see if it makes it easier to debug the MSVC issue you're having with the Code Aster build? |
|
Hey @ldallolio! Good news. I managed to track down the issue in libmed that resulted in no symbols being exported. It's a CMAKE bug, and it is described here: https://gitlab.kitware.com/cmake/cmake/-/issues/25598. I made a fix in Krande/med-copy@36bf272 where I explicitly define the symbols in a new file src/medfwrap_symbols.def.in. Then I manually pass these in using a I created locally a libmed conda package with the symbols using Intel Fortran compiler and https://github.com/Krande/libmed-feedstock/tree/win-fortran/recipe. However, I still need to align the symbol names convention. In Code Aster using Intel Fortran I am using |
|
Hey @ldallolio, just FYI: https://gitlab.com/codeaster/src/-/issues/1#note_1896122953 I think I have managed to solve the circular dependency issue and I am now building the dll's using WAF. Now I've reached a pybind11 issue, which seems to be caused by pybind11 not being initialized properly. I am currently investigating this. Best Regards |
|
Good news @ldallolio! I believe we're finally seeing progress on Code Aster for MSVC Windows :) Minor update: First attempt on MSVC -> Using Gives the following console output |
|
Hi @Krande,
Thanks for this great result and for sharing this news, I am really
impressed!!
In my opinion, tests might be randomly failing on windows because of "too
strict expected result precision check", I have seen that in the past.
Tests results are fixed in "non regression mode" under linux, where they
tend to stay almost the same but under windows results might be slightly
different.
My advice is to have a look at the failure in a couple of cases : if it is
simply a matter of a result change and not a big one, it means that
everything is probably ok (and that you cannot do anything about it).
Once again, thanks for this important result, it will help a lot of people!!
All the best,
Il giorno mer 8 mag 2024 alle ore 12:55 Kristoffer Andersen <
***@***.***> ha scritto:
… Good news @ldallolio <https://github.com/ldallolio>! I believe we're
finally seeing progress on Code Aster for MSVC Windows :)
(codeaster-deps) C:\Work\code\code-aster-src>run_ctest --resutest=temp/ctest -L submit -L sequential -LE need_data --timefactor=5.0 --only-failed-results -j 6
(codeaster-deps) C:\Work\code\code-aster-src>echo OFF
Active code page: 65001
only the results files of testcases in failure will be kept!
execute: rd /s /q C:\Work\code\code-aster-src\temp/ctest
Invalid switch - "ctest".
execute: ctest -j 6 -L sequential.*submit -LE need_data
Test project C:/Work/code/code-aster-src/temp/ctest
Start 102: ASTER_17.0.99_comp008b
Start 105: ASTER_17.0.99_comp008e
Start 115: ASTER_17.0.99_comp008o
Start 128: ASTER_17.0.99_comp010i
Start 176: ASTER_17.0.99_cont005e
Start 268: ASTER_17.0.99_forma01c
1/2203 Test #105: ASTER_17.0.99_comp008e ...........***Failed 2.51 sec
Start 339: ASTER_17.0.99_hpla100k
2/2203 Test #268: ASTER_17.0.99_forma01c ...........***Failed 2.51 sec
Start 361: ASTER_17.0.99_hplv106a
3/2203 Test #115: ASTER_17.0.99_comp008o ........... Passed 2.58 sec
Start 362: ASTER_17.0.99_hplv106b
4/2203 Test #102: ASTER_17.0.99_comp008b ........... Passed 2.61 sec
Start 363: ASTER_17.0.99_hplv106c
5/2203 Test #128: ASTER_17.0.99_comp010i ...........***Failed 2.98 sec
Start 364: ASTER_17.0.99_hplv108a
6/2203 Test #176: ASTER_17.0.99_cont005e ...........***Failed 3.93 sec
Start 365: ASTER_17.0.99_hplv108b
7/2203 Test #339: ASTER_17.0.99_hpla100k ...........***Failed 2.12 sec
Start 366: ASTER_17.0.99_hplv108c
8/2203 Test #363: ASTER_17.0.99_hplv106c ...........***Failed 2.27 sec
Using
from code_aster import CA
CA.close()
Gives the following console output
C:\Work\miniforge3\envs\codeaster-deps\python.exe C:\Work\code\code-aster-src\conda\test_install.py --simple
Ouverture en écriture du fichier ./vola.1
<INFO> Démarrage de l'exécution.
-- CODE_ASTER -- VERSION : DÉVELOPPEMENT (unstable) --
Version 17.0.99 modifiée le 08/05/2024
révision n/a - branche 'n/a'
Copyright EDF R&D 1991 - 2024
Exécution du : Wed May 8 12:49:36 2024
Nom de la machine :
MyWinMachine
Architecture : 64bit
Type de processeur : AMD64
Système d'exploitation :
Windows-10-10.0.22631-SP0
Langue des messages : nb (cp1252)
Version de Python : 3.11.0
Version de NumPy : 1.23.5
Parallélisme MPI : inactif
Parallélisme OpenMP : actif
Nombre de processus OpenMP utilisés : 1
Version de la librairie HDF5 : 1.14.4
Version de la librairie MED : 4.1.1
Version de la librairie MFront : 4.2.0
Librairie MUMPS : non disponible
Librairie PETSc : non disponible
Version de la librairie SCOTCH : 7.0.4
restarting from a previous execution...
Valeur initiale du temps CPU maximum = 86400 secondes
Valeur du temps CPU maximum passé aux
commandes = 77760 secondes
Réserve CPU prévue = 8640 secondes
Ouverture en lecture du fichier ./glob.1
Ajustement de la taille maximale des bases à 2048.00 Go.
Nom de la base : GLOBALE
Créée avec la version :
17.00.99
Nombre d'enregistrements utilisés : 152
Nombre d'enregistrements maximum
: 2684354
Nombre d'enregistrements par fichier : 15728
Longueur d'enregistrement
(octets) : 819200
Nombre d'identificateurs utilisés : 126
Taille maximum du
répertoire : 2000
Pourcentage d'utilisation du répertoire : 6 %
Ouverture en lecture du fichier ./glob.1
Ouverture en écriture du fichier ./vola.1
Fin de lecture (durée 0.000000 s.)
Mémoire limite pour l'allocation dynamique : 4608.00 Mo
ajouté à l'initialisation du processus :
249.94 Mo
Limite cible du processus : 4857.94 Mo
Taille limite des fichiers d'échange : 2048.00 Go
TMP_DIR <class 'pathlib.WindowsPath'>
CONDA_PREFIX <class 'pathlib.WindowsPath'>
ASTER_DIR <class 'pathlib.WindowsPath'>
BIN_DIR <class 'pathlib.WindowsPath'>
# Mémoire (Mo) : 250.00 / 250.00 / 167.58 / 167.58 (VmPeak / VmSize / Optimum / Minimum)
# Fin commande #1 user+syst: 0.08s (syst: 0.05s, elaps: 0.00s)
# ----------------------------------------------------------------------------------------------
.. _stg1_txt88
# ----------------------------------------------------------------------------------------------
# Commande #2 de
C:\Work\miniforge3\envs\codeaster-deps\Library\lib\aster\code_aster\CodeCommands\fin.py, ligne
88
FIN(INFO_RESU='NON',
RETASSAGE='NON')
Saving objects...
TMP_DIR <class 'pathlib.WindowsPath'>
CONDA_PREFIX <class 'pathlib.WindowsPath'>
ASTER_DIR <class 'pathlib.WindowsPath'>
BIN_DIR <class 'pathlib.WindowsPath'>
╔════════════════════════════════════════════════════════════════════════════════════════════════╗
║ <I> <CATAMESS_89> ║
║ ║
║ Liste des alarmes émises lors de l'exécution du calcul. ║
║ ║
║ Les alarmes que vous avez ║
║ choisies d'ignorer sont précédées de (*). ║
║ Nombre d'occurrences pour chacune des alarmes : ║
║ ║
║ aucune alarme ║
╚════════════════════════════════════════════════════════════════════════════════════════════════╝
---------------------------------------------------------------------------
Concepts de la base: G
Nom Type Taille (Mo) Nombre Nombre de
d'objets segments
TOTAL 30.49 45 106
&FOZERO 0.00 2 2
&&_NUM_C 0.00 1 1
&CATA 30.49 42 103
(*) sous-objets non accessible directement
---------------------------------------------------------------------------
Nom de la base : GLOBALE
Nombre d'enregistrements utilisés :
152
Nombre d'enregistrements maximum : 2684354
Nombre d'enregistrements par
fichier : 15728
Longueur d'enregistrement (octets) : 819200
Nombre total d'accès
en lecture : 133
Volume des accès en lecture : 103.91 Mo.
Nombre total d'accès en écriture : 223
Volume des accès en écriture :
174.22 Mo.
Nombre d'identificateurs utilisés : 126
Taille maximum du répertoire
: 2000
Pourcentage d'utilisation du répertoire : 6 %
Nom de la base : VOLATILE
Nombre d'enregistrements utilisés :
107
Nombre d'enregistrements maximum : 2684354
Nombre d'enregistrements par
fichier : 15728
Longueur d'enregistrement (octets) : 819200
Nombre total d'accès
en lecture : 0
Volume des accès en lecture : 0.00 Mo.
Nombre
total d'accès en écriture : 291
Volume des accès en écriture : 227.34
Mo.
Nombre d'identificateurs utilisés : 25
Taille maximum du répertoire
: 2000
Pourcentage d'utilisation du répertoire : 1 %
<I> <FIN> ARRET NORMAL DANS "FIN" PAR APPEL A "JEFINI".
<I> <FIN> MEMOIRE JEVEUX MINIMALE REQUISE POUR L'EXECUTION : 171.31 Mo
<I> <FIN> MEMOIRE JEVEUX OPTIMALE REQUISE POUR L'EXECUTION : 176.30 Mo
<I> <FIN> MAXIMUM DE MEMOIRE UTILISEE PAR LE PROCESSUS LORS DE L'EXECUTION : 259.79 Mo
<I> FERMETURE DES BASES EFFECTUEE
STATISTIQUES CONCERNANT L'ALLOCATION DYNAMIQUE :
TAILLE CUMULEE MAXIMUM : 176 Mo.
TAILLE CUMULEE LIBEREE : 22 Mo.
NOMBRE TOTAL D'ALLOCATIONS : 163
NOMBRE TOTAL DE LIBERATIONS : 163
APPELS AU MECANISME DE LIBERATION : 0
TAILLE MEMOIRE CUMULEE RECUPEREE : 0 Mo.
VOLUME DES LECTURES : 0 Mo.
VOLUME DES ECRITURES : 0 Mo.
MEMOIRE JEVEUX MINIMALE REQUISE POUR L'EXECUTION : 171.31 Mo
- IMPOSE DE NOMBREUX ACCES DISQUE
- RALENTIT LA VITESSE D'EXECUTION
MEMOIRE JEVEUX OPTIMALE REQUISE POUR L'EXECUTION : 176.30 Mo
- LIMITE LES ACCES DISQUE
- AMELIORE LA VITESSE D'EXECUTION
MAXIMUM DE MEMOIRE UTILISEE PAR LE PROCESSUS : 259.79 Mo
- COMPREND LA MEMOIRE CONSOMMEE PAR JEVEUX,
LE SUPERVISEUR PYTHON, LES LIBRAIRIES EXTERNES
<I> FIN D'EXECUTION LE : ME-08-MAI -2024 12:49:36
Signature of pickled file : db2b79f16030d5f871eae12c2d8f92044854d1aa8ac23098e1530240f1c86612
Signature of info file : 98586c4668893eb1a6a6cb880f4654029130feb0a04dd7d22af9f652be382c1d
Signature of Jeveux database: eee03bedc5ad4f78cc1aed30b0eaa9dd283abc612a539e68dffe44ce6117462b
********************************************************************************
* COMMAND : USER : SYSTEM : USER+SYS : ELAPSED *
********************************************************************************
* DEBUT : 0.08 : 0.05 : 0.12 : 0.00 *
* FIN : 0.06 : 0.09 : 0.16 : 0.00 *
* . check syntax : 0.00 : 0.00 : 0.00 : 0.00 *
* . fortran : 0.14 : 0.14 : 0.28 : 0.00 *
********************************************************************************
* TOTAL_JOB : 0.14 : 0.14 : 0.28 : 0.00 *
********************************************************************************
# Mémoire (Mo) : 259.79 / 83.00 / 176.30 / 171.31 (VmPeak / VmSize / Optimum / Minimum)
# Fin commande #2 user+syst: 0.06s (syst: 0.09s, elaps: 0.00s)
# ----------------------------------------------------------------------------------------------
Process finished with exit code 0
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDLWQODL6VBBVEMIVL3ZBIADPAVCNFSM6AAAAAA3ZMAFE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBQGMYDSNJZGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Thanks for the tip! Once my debugging is done, there are still some remaining tasks we'll need to solve if we want to distribute code aster for Windows on conda forge;
Oh, and I had to use the |
|
Hey, I've solved most of the core issues; I no longer need to run with The number of passing tests were at 82% earlier today, and I believe I know how to fix most of the remaining non-passing tests. Btw. One of the bugs that took me some time to hunt down was finally found in libmed. I fixed it, and I can now finally write my results to RMED on MSVC (I suspect the issue might be related to the issue we saw in medpro ref. ldallolio/medpro#1 (comment)) Anyways, with the libmed issue fixed, I can finally visualize the results using Paraview. So here's a figure of an eigenfrequency analysis using the MSVC compiled version :)
|
|
Hi Krande,
You have a great result there! I am pretty sure that we will see a lot of
downloads for this version as soon as it gets out :-)
Nice picture!
Il giorno mer 3 lug 2024 alle ore 18:03 Kristoffer Andersen <
***@***.***> ha scritto:
… Hey, I've solved most of the core issues now. The number of passing tests
were at 82% earlier today, and I believe I know how to fix most of the
remaining non-passing tests.
Btw. One of the bugs that took me some time to hunt down was in libmed. I
finally fixed it, and I can now finally write to RMED on MSVC (I suspect
the issue might be related to the issue we saw in medpro ref. ldallolio/medpro#1
(comment)
<ldallolio/medpro#1 (comment)>)
Anyways, with the libmed issue fixed, I can finally visualize the results
using Paraview. So here's a figure of an eigenfrequency analysis using the
MSVC compiled version :)
image.png (view on web)
<https://github.com/conda-forge/code-aster-feedstock/assets/9642232/7f8d7951-fb00-468a-8fd9-26767e43f95f>
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPAUDIEIPECFZ7WNWOCVYLZKQOD7AVCNFSM6AAAAAA3ZMAFE6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBWGY4TENJTGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hey @ldallolio,
I have done some work on making a conda package for code aster v16.4.2 (with help from Mathieu Courtois and by studying your work and the docker work by aether engineering). You can check the progress here https://gitlab.com/codeaster/src/-/issues/1, The source code for all of the compilation work currently done on v16.4.2 can be found here
At the moment I've compiled and tested the v16.4.2 package for python 3.9, 3.10 and 3.11 and I am currently in the process of debugging the 5-10 % failing tests (150-200 of 2182 tests are failing due to various errors).
Once all (or a sufficient number of) the tests pass, my plan is to create a PR here and at the same time add the dependencies that currently aren't on conda-forge (such as mgis which I have discussed with Thomas Helfer).
I would really appreciate any thoughts you might have on things I should consider before making the PR, or if you have feedback/help you can provide on the current state of compiling code please let me know?
Best Regards
Kristoffer
The text was updated successfully, but these errors were encountered: