Update fluent adapter for compatibility with preCICE versions >= 2.0.0

[BenjaminRodenberg]

The current version (https://github.com/precice/fluent-adapter/commits/master) of this adapter is heavily outdated and does not work with recent preCICE versions. This issue should provide an overview over the progress of this task.

API calls that have to be updated

• precicec_getReadNodesSize
• precicec_mapReadData
• precicec_getReadPositions

Possible Refactoring

• rename all_indices to vertexIDs
• Resolved in update_fluent191
• remove commented out code
• split up fsi.c? Is this allowed for ufs? Currently there is a lot of functionality in one single file.

Documentation of fsi.c

• fsi_init
• fsi_write_and_advance
• fsi_grid_motion
• fsi_plot_coords
• count_dynamic_threads
• gather_write_positions
• gather_read_positions
• read_displacements
• write_forces
• check_write_positions
• check_read_positions
• regather_read_positions Allow remeshing #4
• regather_write_positions Allow remeshing #4

Documentation of fsi_udf.c

• DEFINE_INIT
• DEFINE_ON_DEMAND
• DEFINE_GRID_MOTION
• All resolved in update_fluent191

Remove files

• Constants.h and SolverInterfaceC.h from repo as they are part of precice src/
• Reference here. Resolved in update_fluent191

[BenjaminRodenberg]

On branch update_fluent182 some first steps regarding compatibility of preCICE 1.2.0 and fluent 18.2 were undertaken.

Current state:

• update of Makefile for compatibility with fluent 18.2
• adapter at least compiles with recent fluent and preCICE (the adapter is far from working or being tested properly)
• outdated API calls were removed or clearly indicated as TODO

[lixiaole2017]

I use precice/fluent-adapter at update_fluent182, and precice-1.2.0( ps. I just add some debug code)
Config::Configuration config; This statement cannot be run in Fluent，What is the reason for this?
fluenterror.log

[BenjaminRodenberg]

See this report for more details.

[uekerman]

Files to remove (they are part of preCICE, not the adapter, if they need to be in the same folder as the adapter due to the way Fluent links things copy them over each time)

• https://github.com/precice/fluent-adapter/blob/update_fluent182/src/Constants.h
• https://github.com/precice/fluent-adapter/blob/update_fluent182/src/SolverInterfaceC.h

Remarks:

• the wave functionality has nothing to do with the adapter. Bernhard used this for one of his testcases (compare his thesis). We could probably also remove these files.

[uekerman]

What we need to do as well: document all functions. At least a few lines in front of every function.
E.g. what does this function do?

[mtree22]

I have a customer who needs to couple Fluent (20.2.0) to Abaqus for an FSI simulation. We are looking to preCICE as a possible answer. What can I do to help?

[IshaanDesai]

Hi @mtree22,

We worked on this adapter some time ago and were able to get to a semi-working state. Back then we were working with FLUENT 19.5 R3. There were several issues that we tried to resolve. The summary is as follows:

• The first challenge was to find an operating system where preCICE and Fluent could both be run. We initially explored CentOS 6 and 7 but later on moved to Ubuntu 16.04 and 18.04. Fluent does not officially support Ubuntu OS but we were able to install Fluent and run on Ubuntu systems.
• The master branch of the adapter has much more functionality than what is required for a FSI simulation. For a more updated version of the adapter I would ask you to refer to the branch update_fluent_19_5_R3.
• We have summarized the steps needed to integrate the adapter into Fluent and to start a Fluent simulation where a coupling to preCICE can be done.
• Back then we were not successful in performing a FSI simulation with the adapter. At some point the Ubuntu installation of Fluent 19.5 R3 would stop working and we lacked the Fluent expertise to dig further.

If you want to work with this adapter I can provide assistance from the adapter side. From the perspective of Fluent the adapter is essentially a set of User-Defined Functions (UDFs) which are called by Fluent at particular instances in each timestep and these UDFs then call the preCICE API commands.

[mtree22]

I have a working version of Fluent 20.2.0 on Scientific Linux 7.7 which I've used to run transient simulations all the way to completion. I also compiled preCICE 2.2.0 on this same OS. I can successfully run preCICE tutorials and other FSI simulations using OpenFOAM and CalculiX using this preCICE version and the OpenFOAM and CalculiX adapters. I'm hoping this covers the first challenge.

I will definitely switch to the update_fluent_19_5_R3 branch for my testing.

Do you happen to have a directory and case file setup that I can start from? Maybe something that aligns with the preCICE tutorials already? I'm thinking a fluid-fluent directory just like the fluid-openfoam directory that exists in tutorials/perpendicular-flap, or something like that... If not, I can start from scratch and make one myself.

Do you want me to report my findings here or somewhere else?

[IshaanDesai]

You have definitely already overcome the first challenge of having an OS on which both Fluent and preCICE work.

Do you happen to have a directory and case file setup that I can start from? Maybe something that aligns with the preCICE tutorials already? I'm thinking a fluid-fluent directory just like the fluid-openfoam directory that exists in tutorials/perpendicular-flap, or something like that... If not, I can start from scratch and make one myself.

In our initial attempts we were trying to get a 2D lid-driven cavity with a flexible bottom case to work. In this case the fluid cavity is simulated using Fluent and the flexible bottom is simulated using a dummy structural solver. I can point you to a very preliminary case setup. Please ask me if something is not clear in this case setup. I had not got this FSI case to work but I guess this is probably a good place to start rather than blank slate.

It would be very helpful if you reported your findings and we can do this here in the issue. That way we also have a record of our discussion which we can later refer to.

[mtree22]

I found your Cavity2D case shortly after I wrote that last comment. Yes, this definitely seems to be the first place to start.

I opened Cavity2D.cas in the fluent GUI just to see what it looks like, and it's complaining about the pre-compiled UDFs that exist the libudf sub-directory:

Auto-compilation skipped. Could not find input files required for creating "./libudf" library
corvidpost5.corvidtec.com: Opening library "/home/shelf1/motorsports/software/Utilities/source/precice/fluent-adapter/examples/Cavity2D/./libudf"...RPC RCX_SC_SET_ERR_MSG failed: RPC: Can't encode arguments

Error at host: The UDF library you are trying to load (./libudf) is not compiled for parallel use on the current platform (lnamd64).

No such file or directory
/home/shelf1/motorsports/software/Utilities/source/precice/fluent-adapter/examples/Cavity2D/./libudf/lnamd64/2d_host/libudf.so

Error at Node 0: The UDF library you are trying to load (./libudf) is not compiled for parallel use on the current platform (lnamd64).

No such file or directory
/home/shelf1/motorsports/software/Utilities/source/precice/fluent-adapter/examples/Cavity2D/./libudf/lnamd64/2d_node/libudf.so

corvidpost5.corvidtec.com: Opening library "/home/shelf1/motorsports/software/Utilities/source/precice/fluent-adapter/examples/Cavity2D/./libudf"...
Error: The UDF library you are trying to load (./libudf) is not compiled for parallel use on the current platform (lnamd64).

No such file or directory
/home/shelf1/motorsports/software/Utilities/source/precice/fluent-adapter/examples/Cavity2D/./libudf/lnamd64/2d_host/libudf.so
Error Object: #f

I'm thinking I have to delete what you have here and compile my own UDFs, right?

[IshaanDesai]

Yes you have to delete the folder libudf/ and follow the steps in the README file from the start. I think back then I pushed the pre-compiled UDFs unintentionally.

[mtree22]

Ok, I deleted libudf/ and used the README to compile my own UDFs. Now, I looked in steer-fluent.txt and found this command:

fluent 2ddp -g < steer-fluent.txt

BUT when I ran that, Fluent responded that it translated that to this command:

/opt/Software/ansys/v202/fluent/fluent20.2.0/bin/fluent -r20.2.0 2ddp -pshmem -host -t1 -mpi=intel -path/opt/Software/ansys/v202/fluent -ssh -cx corvidpost5.corvidtec.com:35966:33347

the -mpi=intel part of that command bothered me, since I compiled preCICE with openmpi, so instead I ran this:

fluent 2ddp -g -t1 -mpi=openmpi < steer-fluent.txt

That definitely progressed the sim further, but I was still met with an error. It looks like the error hits inside fsi_init when the precicec_createSolverInterface function is called.

I'm attaching several files. Any idea how to move past this?
fluent-999999-error.log
fluent-20220124-145048-20088.log
fluent-0-error.log
fluentCSM_debug.log
precice-config.txt

[IshaanDesai]

Firstly the mpi selection used to a problem before too so it is good to see that you found the relevant command for Fluent to be able to use openmpi with which preCICE is also compiled.

A possible problem I realized is in the configuration file. The line:

fluent-adapter/examples/Cavity2D/precice-config.xml

Line 44 in 714d77b

<time-window-size method="first-participant"/>

is problematic. Can you change it to <time-window-size value="0.001"/> and try again? Here 0.001 is just a dummy time step value for both participants. I will look into the adapter to see if the preCICE time step is directly passed on to Fluent or is there some computation in between.

[mtree22]

I changed line 44 to <time-window-size value="0.001"/> and re-ran the simulation. The same thing happened.

[IshaanDesai]

Okay, lets try to debug further. In the file fluent-0-error.log towards the end we see errors of the form:

 /opt/Software/ansys/v202/fluent/fluent20.2.0/lnamd64/2ddp_node/fluent_mpi.20.2.0(Init_Flow+0x4cd) [0xb4e3fd]
 /opt/Software/ansys/v202/fluent/fluent20.2.0/lnamd64/2ddp_node/fluent_mpi.20.2.0() [0x605337]
 /opt/Software/ansys/v202/fluent/fluent20.2.0/lnamd64/2ddp_node/fluent_mpi.20.2.0(eval+0x4b5) [0x206dbb5]
 /opt/Software/ansys/v202/fluent/fluent20.2.0/lnamd64/2ddp_node/fluent_mpi.20.2.0(PRF_Command_Start+0x131e) [0x5f1c1e]
 /opt/Software/ansys/v202/fluent/fluent20.2.0/lnamd64/2ddp_node/fluent_mpi.20.2.0(PRF_Node_repl+0x2e) [0x5f2ede]
 /opt/Software/ansys/v202/fluent/fluent20.2.0/lnamd64/2ddp_node/fluent_mpi.20.2.0(main+0x3ec) [0x607abc]
 /lib64/libc.so.6(__libc_start_main+0xf5) [0x2b533a1a0545]
 /opt/Software/ansys/v202/fluent/fluent20.2.0/lnamd64/2ddp_node/fluent_mpi.20.2.0() [0x513802]

This is suspicious because it indicates that Fluent is using an MPI shipped with it, or a Fluent compiled with a specific MPI. This could be problematic if we try to use it with a preCICE installation which is not compiled with the same MPI. Diagnostics can be done as follows:

1. Is Fluent shipped with an MPI? If yes, can we access it?
2. Can we compile preCICE with the MPI shipped with Fluent?

I am not sure if this is a problem, but it is definitely worth a shot.

[mtree22]

I was curious about something else based on the log files. In the fluent-20220124-145048-20088.log file it's clear that Fluent has kicked off 2 processes:

-------------------------------------------------------------------------------
ID    Hostname              Core  O.S.      PID    Vendor                  
-------------------------------------------------------------------------------
n0    corvidpost5.corvidte  1/64  Linux-64  22288  AMD Opteron(tm) 6386 SE 
host  corvidpost5.corvidte        Linux-64  21348  AMD Opteron(tm) 6386 SE 

MPI Option Selected: openmpi
-------------------------------------------------------------------------------

This made me wonder if Fluent was actually running in "parallel". This may be a concern because lines 86 and 87 of fsi.c overwrite the precice_process_id and comm_size variables to test "serial versions" only:
https://github.com/precice/fluent-adapter/blob/update_fluent_19_5_R3/examples/Cavity2D/libudf/src/fsi.c#L86

So, I commented out lines 86 and 87 and re-ran. This resulted in proceeding a little further into reading the precice-config.xml file. At least enough to show errors related to consistent and conservative mapping. I made the necessary changes in precice-config.xml, but the simulation still falls short of creating an interface object.

Now, to answer your questions:

1. Yes, I looked into the directories associated with the installation and it appears that Fleunt ships with a compliment of openmpi libraries. It seems that I can access the libraries.

2. I will try to compile preCICE with the MPI libraries shipped with Fluent.

It also appears that Fluent uses a wrapper to control which implementation and version of MPI that it uses. This means it's possible to run Fluent using a locally-installed version of OpenMPI by defining an OPENMPI_ROOT environment variable. I tried to run Fluent with the version of OpenMPI that I used in my preCICE compile (4.0.0), but apparently Fluent uses some deprecated OpenMPI functions that no longer exist in v4.0.0.

So, first I'm going to try and compile preCICE using the OpenMPI libraries that are included in the Fluent install, even though I don't know what version they are. If that doesn't work, I'll re-compile preCICE with an older version of OpenMPI and then use the OPENMPI_ROOT environment variable and hope that works better.

[IshaanDesai]

Firstly thanks for continuing to pursue this, it is really valuable to revive the adapter 😃

It was known that Fluent starts processes n0 and host and I had made the corresponding corrections in the adapter. My logic was that only one of the two processes actually does the coupling and hence needs to create the coupling interface. If both the processes create the interface object then it is a problem because later on only one of them can actually access data in Fluent. This was my observation some time ago so I am not entirely sure about it. Lets check this as we go along with the debugging.

I am eager to know what happens when you really manage to have the same OpenMPI for preCICE and Fluent. I expect this will resolve some cryptic errors for us.

[mtree22]

Ah, I was misunderstanding what was happening inside fsi.c with regard to the two processes. I think you'll find I misunderstand quite a bit... Regardless, I undid any of my changes (commenting out of lines, adding of printf statements) to fsi.c and re-compiled the UDF.

Then, I tried to go compile preCICE with a version of OpenMPI before 4.0.0, but I was having trouble getting PETSc to build, so I ventured into the world of Intel compilers and have landed on a consistent set of PETSc and preCICE that was build on Intel MPI 2018.4.274. Similar to the OPENMPI_ROOT environment variable, Fluent allows the use of an INTELMPI_ROOT variable to specify a custom Intel MPI version, so I set that to point to the same Intel MPI version. That definitely got us further than ever before!

Also, at some point I added a 2nd preCICE log file (fluentCSM_alldebug.log) that doesn't filter out any messages at all. It's probably overkill, but I thought it might be useful. run_fluent.txt is a copy of the shell script I use to kick off the sim.

fluentCSM_alldebug.log
fluentCSM_debug.log
precice-config.txt
run_fluent.txt
fluent-0-error.log
fluent-20220127-111934-6565.log

[IshaanDesai]

I also spent a lot of time misunderstanding stuff in this adapter so we are pretty much on the same page there 😉

It is great that you managed to get Fluent and preCICE to compile with the same MPI. Could you please document this somewhere? When we get the adapter to a working state, it would be essential to have these steps in the documentation for future users. At some point we will try to integrate this in the adapter documentation.

We never reached as far as you have right now, so now we will have to do some debugging together to figure out what is going wrong. I see that the error is now in the function set_mesh_positions. The problem appears to be in the following loop:

fluent-adapter/src/fsi.c

Lines 268 to 278 in 714d77b

	/* Count number of interface vertices and dynamic_thread_node_size */
	begin_f_loop(face, face_thread){
	if (PRINCIPAL_FACE_P(face,face_thread)){
	f_node_loop(face, face_thread, n){
	node = F_NODE(face, face_thread, n);
	printf("Count of wet_nodes_size = %d\n", wet_nodes_size);
	wet_nodes_size++;
	dynamic_thread_node_size[thread_index]++;
	}
	}
	} end_f_loop(face, face_thread);

Can you check if the the increment of dynamic_thread_node_size even happens? It could be that in some previous refactoring we messed up the initialization.

[mtree22]

I've been documenting a lot of what I did in the README.md file. Details of how to get Fluent to use any MPI are specified there. I forked this repo, so maybe we should move this discussion to a pull request sometime? That may make it easier for you to see the exact changes I've made and comment on them.

Regardless, I went ahead and added some printf statements between lines 274 and 275. I asked the code to spit out thread_index, and then dynamic_thread_node_size[thread_index]. That seems to return that the thread_index is 0, but there is no return of dynamic_thread_node_size[thread_index]. Instead, there is an error:

(999999) ERROR: domain.dynamic_threads == NULL

Which seems to come from a message on line 256.

I also noticed a comment from the preCICE log:

preCICE: No coupling scheme exists for the participant

I checked the precice-config.xml file, and I think the coupling scheme is defined appropriately, but maybe you can look it over? Could this be a source of the issue we're seeing in the loop?

Here are some new log files from this latest run:
fluent-20220128-151345-13608.log
fluentCSM_alldebug.log
fluentCSM_debug.log
fluent-0-error.log
fluent-999999-error.log

[IshaanDesai]

I've been documenting a lot of what I did in the README.md file. Details of how to get Fluent to use any MPI are specified there. I forked this repo, so maybe we should move this discussion to a pull request sometime? That may make it easier for you to see the exact changes I've made and comment on them.

Yes lets discuss the documentation and also all other changes in a pull request once we have something working here.

I checked the precice-config.xml using the config-visualizer and the configuration in itself does not look problematic:

One very trivial thing I noticed is that whenever you attach the precice-config file you attach a .txt file, but the original file is XML. Do you change the file format for some reason?

Another possible reason for the preCICE error could be that in the adapter we have only one of the two processes n0 and host connected to preCICE so if by chance due to some incorrect programming the other process tries to call something with preCICE we may see an error. I think this is unlikely as the error message would be very different than what we see.

Regarding the error of domain.dynamic_threads I really have no clue. The possibilities of what could have gone wrong are vast, mainly because of the intricate Fluent case setup required at the start. Do you by chance have a GUI version of Fluent working? We could then check the case setup in detail.
Otherwise I will try to find out more about this in the Fluent documentation and forums. Sorry I cannot be of direct help here with this particular issue.

[mtree22]

One very trivial thing I noticed is that whenever you attach the precice-config file you attach a .txt file, but the original file is XML. Do you change the file format for some reason?

Yes, when I try and upload a file with the .xml extension to these comments, GitHub yells at me as says that this filetype is not supported. The same thing happens with the fluent-2022*.log files. They are actually output in a .trn format. GitHub complains about that too. So, I just make a copy of them on my simulation directory and change the file extension. I'm certainly still running the simulation with a precice-config.xml file.

Do you by chance have a GUI version of Fluent working? We could then check the case setup in detail.

Yes, I can open the GUI and run from within there. What would you like me to check out in the GUI?

[IshaanDesai]

Yes, when I try and upload a file with the .xml extension to these comments, GitHub yells at me as says that this filetype is not supported. The same thing happens with the fluent-2022*.log files. They are actually output in a .trn format. GitHub complains about that too. So, I just make a copy of them on my simulation directory and change the file extension. I'm certainly still running the simulation with a precice-config.xml file.

Okay, I was not aware of GitHub complaining then! I had another question regarding the configuration. You said that at some point you corrected some stuff regarding the consistent / conservative mapping right? Can you please send over the new config file? I just want to make sure we are not making some mistakes there.

Yes, I can open the GUI and run from within there. What would you like me to check out in the GUI?

I think it would be a good idea to check if all the instructions from this section are being done correctly. Can you open the .cas file in the GUI and check this once please? I am aware that the section is incomplete but lets check up to the point written there.

[mtree22]

You said that at some point you corrected some stuff regarding the consistent / conservative mapping right? Can you please send over the new config file? I just want to make sure we are not making some mistakes there.

Of course, here is the config file I'm using:
precice-config.txt

And here is a diff file summarizing the changes I made:
precice-config_diff.txt

Can you open the .cas file in the GUI and check this once please?

Yes, I just opened Fluent GUI. Before opening the GUI, I defined the INTELMPI_ROOT environment variable, and Fluent recognizes it upon starting the GUI:

***
*** Using locally installed Intel MPI Library!
***
*** INTELMPI_ROOT=/opt/Software/intel_parallel_studio/2018u4/impi/2018.4.274/intel64
***
Starting /opt/Software/intel_parallel_studio/2018u4/impi/2018.4.274/intel64/bin/mpirun --rsh=ssh -f /tmp/fluent-appfile.mtree.43583 -genv FLUENT_ARCH lnamd64 -genv I_MPI_DEBUG 0 -genv I_MPI_PIN disable -genv I_MPI_ADJUST_REDUCE 2 -genv I_MPI_ADJUST_ALLREDUCE 2 -genv I_MPI_ADJUST_BCAST 1 -genv I_MPI_ADJUST_BARRIER 2 -genv I_MPI_ADJUST_ALLGATHER 2 -genv I_MPI_ADJUST_GATHER 2 -genv I_MPI_ADJUST_ALLTOALL 1 -genv I_MPI_ADJUST_SCATTER 2 -genv I_MPI_PLATFORM auto -genv PYTHONHOME /opt/Software/ansys/v202/fluent/fluent20.2.0/../../commonfiles/CPython/3_7/linx64/Release/python -genv FLUENT_PROD_Dir /opt/Software/ansys/v202/fluent/fluent20.2.0 -genv KMP_AFFINITY disabled -genv TMI_CONFIG /opt/Software/intel_parallel_studio/2018u4/impi/2018.4.274/intel64/etc/tmi.conf -n 1 -host corvidpost5.corvidtec.com /opt/Software/ansys/v202/fluent/fluent20.2.0/lnamd64/2ddp_node/fluent_mpi.20.2.0 node -mpiw intel -pic shmem -mport 10.1.211.5:10.1.211.5:35102:0

From here I can successfully read the Cavity2D.cas file. Upon loading that file I checked the following:

1. User-Defined -> Functions -> Manage
   
   I can see the libudf directory already loaded. I can also type libudf into the "Library Name" box and click load. Doing this re-loads the libudf library and displays the init, write_and_advance, gridmotions, and plot_coords functions. These same functions also appear in the GUI console output when the .cas file is loaded though.

2. User-Defined -> Function Hooks
   Looks like this:
   
   This looks to me a hook is already defined in Initialization. I can click the edit button next to Initialization and re-define the hook, but I can't edit any of the other places for a hook (Adjust, Execute at End, etc.).

3. User-Defined -> Memory
   Looks like this:
   
   I'm unsure if both of those are to be set to "1" per your instructions, but it seems fine to me.

4. Domain -> Mesh Models -> Dynamic Mesh
   Looks like this:
   
   With the following parameters for the moving_base dynamic mesh:
   

All of this seems to be in order, if you ask me.

[IshaanDesai]

All the GUI steps look to be in order. I think there is a problem with the object Domain* domain that the function fsi_init receives from Fluent. I am trying to find out more information about Domain in the UDF section of the Fluent manual but this may take some time.

The domain->dynamic_threads being NULL seems to be just wrong. Can you check if the domain object is even of the correct type? That would give us a hint if there is something fundamentally wrong with the case setup or something is going wrong in the adapter.

I know that what I am suggesting is very vague and could possibly not lead us anywhere, but we have never really got this far with the adapter in recent times so we have to really look everywhere possible to figure out the problem.

[IshaanDesai]

@mtree22 in addition to our discussion here over the last few days, lets try to have an extended discussion during the upcoming workshop on the user support day. We will upload the workshop schedule shortly and I will make it a point to have a time slot for us to talk.

[mtree22]

@IshaanDesai That sounds great! I'm already registered for the workshop and am looking forward to attending.

In the meantime I'm trying to test out the domain object at the different stages of the adapter code. I wrote a "debug_udf.c" file that only calls a DEFINE_INIT macro to read in the domain object, but before I call fsi_init, I have the code print to the screen some contents of domain. I was able to track down the source code for domain on my system, here:

opt/Software/ansys/v202/fluent/fluent20.2.0/src/storage/mem.h

where lines 639-808 defines the struct called "domain_struct" that is behind the Domain* domain pointer.

So, I'm trying to print values from domain like:

domain->id
domain->index
domain->level
domain->name
domain->material_name

To try and give myself a sense of whether or not the domain exists, and whether something happens to it within fsi_init at some point. I can at least confirm that Domain* domain is a pointer to a structure and that I can use the types of commands listed above to pull some values through -- the real meaning of those values is a bit cryptic at times, but at least I'm getting something other than NULL. I'll keep you updated if I find anything ground-breaking.

[IshaanDesai]

Hi @mtree22

Sorry for the delayed response, things are busy as the workshop is close.

So, I'm trying to print values from domain like:

domain->id
domain->index
domain->level
domain->name
domain->material_name

It is already promising that these values return some values other than NULL.

To try and give myself a sense of whether or not the domain exists, and whether something happens to it within fsi_init at some point. I can at least confirm that Domain* domain is a pointer to a structure and that I can use the types of commands listed above to pull some values through -- the real meaning of those values is a bit cryptic at times, but at least I'm getting something other than NULL. I'll keep you updated if I find anything ground-breaking.

From what you are describing my guess is that domain->dynamic_threads would have existed in a previous Fluent version and probably the name of the variable has since been changed. A good idea would be to check Fluent documentation in this regard. Maybe just something to try would be if something like domain->threads exists, or perhaps some similar names.

Unfortunately I cannot see these things for myself as I do not have a running Fluent installation. The workshop is going to be a good platform for us to try things together 😄

[precice-bot]

This issue has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/precice-workshop-2022-introduce-yourself/904/8

[mtree22]

From what you are describing my guess is that domain->dynamic_threads would have existed in a previous Fluent version and probably the name of the variable has since been changed.

I don't think this is the case. I can grep the /opt/Software/ansys/v202/fluent/fluent20.2.0/src/ directory for dynamic_threads and I find the following:

storage/mem.h:739:  struct dynamic_thread_struct *dynamic_threads;

line 739 of mem.h is in the middle of the domain_struct definition. This is the line that defines domain->dynamic_threads as being a pointer to a structure (a dynamic_thread_struct). So, I can then grep the same directory for 'dynamic_thread_struct' and I find the following:

mesh/dynamesh_tools.h:45:typedef struct dynamic_thread_struct

Upon viewing dynamesh_tools.h, I see that the dynamic_thread_struct is defined on lines 45 to 139

While looking at all of this, there are thread_struct variables as well. So, what's the difference between dynamic_thread_struct and thread_struct? Is one for cell zones that can move (dynamic_thread) and one for cell zones that cannot (thread)? What's the purpose of the UDF code that is in question? If calls to thread_struct will work, can we cycle through those and initialize the preCICE interface from the proper cell zone(s) instead of trying to look only at dynamic threads?

I'll continue working on this as much as possible before our meeting next Thursday (24 Feb). Also, do you mind if that meeting is during either of the last 2 blocks ( 13:00 - 15:00 or 16:00 - 18:00 CET)? I live in the eastern part of the United States, so I'm 6 hours behind Central European Time. If we schedule our meeting during that first block of time it will be an awful early morning for me (04:00).

[IshaanDesai]

@mtree22 lets continue this discussion via the workshop on Thursday. I will arrange for us to meet in the later two blocks, thats not a problem 👍

[mtree22]

@IshaanDesai @BenjaminRodenberg

I jumped back into this today in preparation for our discussion tomorrow. I've made a bit of progress, so I wanted to note it here so we can all start on the same page.

In order to pair down the problem, I wrote my own udf file (debug_udf.c). All it does is print a couple of messages and then steps into fsi_init() (from fsi.c). Everything up to this point has been focused on the domain structure that Fluent passes to the UDF and whether that domain has an appropriately defined dynamic_thread in it. While reading the fsi.c code, I noticed that a call to count_dynamic_threads() was commented out, so I uncommented this in hopes that it might shed more light on if a dynamic_thread was present and properly defined. This function seems to work just fine, and it finds a single dynamic_thread. The issue still remains within the set_mesh_positions function.

I researched a bit online and found these instructions, which I'm using to step through the udf code line-by-line until the error occurs. Using this, I found that the error occurs on line 308 of my current working version of fsi.c. Again, I've added A LOT of printf and message statements, so this line number no longer makes sense. Instead, this equates to line 298 in this branches fsi.c file:

291   begin_f_loop(face, face_thread){
292     if (PRINCIPAL_FACE_P(face,face_thread)){
293       f_node_loop(face, face_thread, n){
294         node = F_NODE(face, face_thread, n);
295         Message("(%d) Count of wet_nodes_size = %d\n", myid, wet_nodes_size);
296         wet_nodes_size++;
297         Message("(%d) thread_index: %d\n", myid, thread_index);
298         Message("(%d) Count of dynamic_thread_node_size[thread_index] = %d\n", myid,myid,  dynamic_thread_node_size[thread_index]);
299         dynamic_thread_node_size[thread_index]++;
300       }
301     }
302   } end_f_loop(face, face_thread);

I'm hoping to be able to share my screen with you to show you gdb in progress while we step through the code together. From here, maybe we find an easy fix and everything just works out. Or, maybe we move on to the next challenge and strategize on how to tackle it.

Talk to you tomorrow!

[mtree22]

We held a meeting as part of the preCICE workshop 2022 and learned some things about the Fluent adaptor. The current status is that a Fluent adaptor is not possible due to Boost library version incompatibilities. See here: #23 (comment)

[BenjaminRodenberg]

@mtree22 I updated the milestone with issues that are directly connected to getting the adapter running (https://github.com/precice/fluent-adapter/milestone/1). I would suggest to define preCICE 2.3.0 and Fluent 21.2.0 as our baseline until we get to a working state.

Could you also push your latest updates? Let's use this branch for our work: https://github.com/precice/fluent-adapter/tree/fluent_21_2_0. I think the other branches are becoming a bit confusing, because they are very outdated.