ansys.mapdl.reader can not read large CDB files

[Keynaan]

🔍 Before submitting the issue

• I have searched among the existing issues
• I am using a Python virtual environment

🐞 Description of the bug

It seems, that the CDB File reader in ansys.mapdl.reader can not read large files due to the old 32bit problematic.

Disclaimer: I am not that into C++, so take my suggestions with care!

I might have tracked it down to ansys/mapdl/reader/cython/_reader.pyx, where all libc.stdio imports their functions.
https://github.com/pyansys/pymapdl-reader/blob/3c54e59b196688ef99993d5aaf7398ce14abcfcd/ansys/mapdl/reader/cython/_reader.pyx#L162
Uses ftell to get the size, which might be incorrect for large files as shown in the following reproduction section.

I already tried the usual suggestions by using the following extra compile flags in setup.py to get 64bit functionality.

    extra_compile_args = ["-O3", "-w",                                                                                                                                                                                                                                                                                                                                     
                          "-D_FILE_OFFSET_BITS=64",                                                                                                                                                                                                                                                                                                                        
                          "-D_LARGEFILE64_SOURCE",                                                                                                                                                                                                                                                                                                                         
                          "-D_LARGE_FILE_SOURCE=1"]

Which hasn't changed anything as the shared library is compiled in 64bit mode anyway:

$ file ansys/mapdl/reader/_reader.cpython-38-x86_64-linux-gnu.so 
ansys/mapdl/reader/_reader.cpython-38-x86_64-linux-gnu.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

The linked libc is a 64bit shared object as well.

As always, thank you for the great work and hopefully it can be fixed easily.

📝 Steps to reproduce

After installation and loading of a rather large CDB file (about 3GB).

python -c "from ansys.mapdl import reader as pymapdl_reader; print(pymapdl_reader.Archive('/path/to/3GB/structure.CDB', parse_vtk=False, verbose=True))"

returns

ANSYS Archive File structure.CDB (About 2.5GB)
  Number of Nodes:              0
  Number of Elements:           0
  Number of Element Types:      0
  Number of Node Components:    0
  Number of Element Components: 0

fsize returns: -1638917353, which looks like a classic overflow.

Alltough a small file loads just fine:

ANSYS Archive File structure.CDB (About 106MB)
  Number of Nodes:              235343
  Number of Elements:           261265
  Number of Element Types:      3
  Number of Node Components:    3
  Number of Element Components: 1                                                                                                                                                                                                                                                                                                                                          

fsize returns: 111023960, which is the correct size.

💻 Which operating system are you using?

Linux

🐍 Which Python version are you using?

3.8

📦 Installed packages

aiohttp==3.7.4.post0                                                                                                                                                                                                                                                                                                                                                       
aiosignal==1.2.0                                                                                                                                                                                                                                                                                                                                                           
alabaster==0.7.12                                                                                                                                                                                                                                                                                                                                                          
ansys-api-mapdl==0.5.1                                                                                                                                                                                                                                                                                                                                                     
ansys-api-platform-instancemanagement==1.0.0b3                                                                                                                                                                                                                                                                                                                             
ansys-corba==0.1.1                                                                                                                                                                                                                                                                                                                                                         
ansys-mapdl-core==0.62.1                                                                                                                                                                                                                                                                                                                                                   
ansys-mapdl-reader==0.51.5                                                                                                                                                                                                                                                                                                                                                 
ansys-platform-instancemanagement==1.0.2                                                                                                                                                                                                                                                                                                                                   
appdirs==1.4.4                                                                                                                                                                                                                                                                                                                                                             
async-timeout==3.0.1                                                                                                                                                                                                                                                                                                                                                       
attrs==21.2.0                                                                                                                                                                                                                                                                                                                                                              
autobahn==21.3.1                                                                                                                                                                                                                                                                                                                                                           
Automat==20.2.0                                                                                                                                                                                                                                                                                                                                                            
Babel==2.10.1                                                                                                                                                                                                                                                                                                                                                              
cached-property==1.5.2                                                                                                                                                                                                                                                                                                                                                     
certifi==2021.10.8                                                                                                                                                                                                                                                                                                                                                         
cffi==1.14.6                                                                                                                                                                                                                                                                                                                                                               
cfgv==3.3.1                                                                                                                                                                                                                                                                                                                                                                
cftime==1.0.3.4                                                                                                                                                                                                                                                                                                                                                            
chardet==4.0.0                                                                                                                                                                                                                                                                                                                                                             
charset-normalizer==2.0.12                                                                                                                                                                                                                                                                                                                                                 
commonmark==0.9.1                                                                                                                                                                                                                                                                                                                                                          
constantly==15.1.0                                                                                                                                                                                                                                                                                                                                                         
coverage==6.3.1                                                                                                                                                                                                                                                                                                                                                            
cryptography==3.4.8                                                                                                                                                                                                                                                                                                                                                        
cycler==0.10.0                                                                                                                                                                                                                                                                                                                                                             
Cython==0.29.21                                                                                                                                                                                                                                                                                                                                                            
distlib==0.3.4                                                                                                                                                                                                                                                                                                                                                             
docutils==0.18.1                                                                                                                                                                                                                                                                                                                                                           
filelock==3.4.2                                                                                                                                                                                                                                                                                                                                                            
flake8==5.0.4                                                                                                                                                                                                                                                                                                                                                              
flatbuffers==2.0                                                                                                                                                                                                                                                                                                                                                           
fonttools==4.28.3                                                                                                                                                                                                                                                                                                                                                          
frozenlist==1.3.0
future==0.17.1                                                                                                                                                                                                                                                                                                                                                             
geomdl==5.3.1                                                                                                                                                                                                                                                                                                                                                              
git-lfs==1.6                                                                                                                                                                                                                                                                                                                                                               
googleapis-common-protos==1.56.3                                                                                                                                                                                                                                                                                                                                           
grpcio==1.47.0                                                                                                                                                                                                                                                                                                                                                             
h5py==2.10.0                                                                                                                                                                                                                                                                                                                                                               
hyperlink==21.0.0                                                                                                                                                                                                                                                                                                                                                          
identify==2.4.12                                                                                                                                                                                                                                                                                                                                                           
idna==3.2                                                                                                                                                                                                                                                                                                                                                                  
imageio==2.9.0                                                                                                                                                                                                                                                                                                                                                             
imagesize==1.3.0                                                                                                                                                                                                                                                                                                                                                           
importlib-metadata==4.12.0                                                                                                                                                                                                                                                                                                                                                 
incremental==21.3.0                                                                                                                                                                                                                                                                                                                                                        
iniconfig==1.1.1                                                                                                                                                                                                                                                                                                                                                           
Jinja2==3.0.3                                                                                                                                                                                                                                                                                                                                                              
joblib==0.14.0                                                                                                                                                                                                                                                                                                                                                             
kiwisolver==1.3.2                                                                                                                                                                                                                                                                                                                                                          
MarkupSafe==2.1.1                                                                                                                                                                                                                                                                                                                                                          
matplotlib==3.4.3                                                                                                                                                                                                                                                                                                                                                          
mccabe==0.7.0                                                                                                                                                                                                                                                                                                                                                              
meshio==4.4.6                                                                                                                                                                                                                                                                                                                                                              
mock==4.0.3                                                                                                                                                                                                                                                                                                                                                                
mpi4py==3.0.3                                                                                                                                                                                                                                                                                                                                                              
multidict==5.1.0                                                                                                                                                                                                                                                                                                                                                           
natsort==8.1.0                                                                                                                                                                                                                                                                                                                                                             
netCDF4==1.5.3                                                                                                                                                                                                                                                                                                                                                             
nodeenv==1.6.0                                                                                                                                                                                                                                                                                                                                                             
numpy==1.21.2                                                                                                                                                                                                                                                                                                                                                              
packaging==21.3                                                                                                                                                                                                                                                                                                                                                            
petsc4py==3.14.0                                                                                                                                                                                                                                                                                                                                                           
pexpect==4.8.0                                                                                                                                                                                                                                                                                                                                                             
Pillow==8.3.1                                                                                                                                                                                                                                                                                                                                                              
platformdirs==2.4.1                                                                                                                                                                                                                                                                                                                                                        
pluggy==1.0.0                                                                                                                                                                                                                                                                                                                                                              
pre-commit==2.9.3                                                                                                                                                                                                                                                                                                                                                          
protobuf==3.19.4                                                                                                                                                                                                                                                                                                                                                           
protoc-gen-swagger==0.1.0                                                                                                                                                                                                                                                                                                                                                  
ptyprocess==0.7.0
py==1.11.0                                                                                                                                                                                                                                                                                                                                                                 
pybind11==2.5.0                                                                                                                                                                                                                                                                                                                                                            
pycodestyle==2.9.1                                                                                                                                                                                                                                                                                                                                                         
pycparser==2.20                                                                                                                                                                                                                                                                                                                                                            
pyflakes==2.5.0                                                                                                                                                                                                                                                                                                                                                            
Pygments==2.11.2                                                                                                                                                                                                                                                                                                                                                           
pyiges==0.2.1                                                                                                                                                                                                                                                                                                                                                              
pyparsing==2.4.7                                                                                                                                                                                                                                                                                                                                                           
pytecplot==1.4.2                                                                                                                                                                                                                                                                                                                                                           
pytest==6.2.5                                                                                                                                                                                                                                                                                                                                                              
pytest-cov==3.0.0                                                                                                                                                                                                                                                                                                                                                          
pytest-mock==3.7.0                                                                                                                                                                                                                                                                                                                                                         
python-dateutil==2.8.2                                                                                                                                                                                                                                                                                                                                                     
pytz==2022.1                                                                                                                                                                                                                                                                                                                                                               
pyvista==0.31.3                                                                                                                                                                                                                                                                                                                                                            
PyYAML==6.0                                                                                                                                                                                                                                                                                                                                                                
pyzmq==22.3.0                                                                                                                                                                                                                                                                                                                                                              
recommonmark==0.7.1                                                                                                                                                                                                                                                                                                                                                        
requests==2.27.1                                                                                                                                                                                                                                                                                                                                                           
rich==12.3.0                                                                                                                                                                                                                                                                                                                                                               
scikit-learn==0.23.2                                                                                                                                                                                                                                                                                                                                                       
scipy==1.5.4                                                                                                                                                                                                                                                                                                                                                               
scooby==0.5.7                                                                                                                                                                                                                                                                                                                                                              
setuptools-scm==6.3.2                                                                                                                                                                                                                                                                                                                                                      
simple-pid==1.0.1                                                                                                                                                                                                                                                                                                                                                          
six==1.16.0                                                                                                                                                                                                                                                                                                                                                                
snowballstemmer==2.2.0                                                                                                                                                                                                                                                                                                                                                     
Sphinx==3.3.1                                                                                                                                                                                                                                                                                                                                                              
sphinx-rtd-theme==0.5.2                                                                                                                                                                                                                                                                                                                                                    
sphinxcontrib-applehelp==1.0.2                                                                                                                                                                                                                                                                                                                                             
sphinxcontrib-devhelp==1.0.2                                                                                                                                                                                                                                                                                                                                               
sphinxcontrib-htmlhelp==2.0.0                                                                                                                                                                                                                                                                                                                                              
sphinxcontrib-jsmath==1.0.1                                                                                                                                                                                                                                                                                                                                                
sphinxcontrib-qthelp==1.0.3                                                                                                                                                                                                                                                                                                                                                
sphinxcontrib-serializinghtml==1.1.5                                                                                                                                                                                                                                                                                                                                       
testfixtures==6.18.3                                                                                                                                                                                                                                                                                                                                                       
threadpoolctl==2.0.0                                                                                                                                                                                                                                                                                                                                                       
toml==0.10.2                                                                                                                                                                                                                                                                                                                                                               
tomli==1.2.2                                                                                                                                                                                                                                                                                                                                                               
tox==3.24.5                                                                                                                                                                                                                                                                                                                                                                
tqdm==4.62.2                                                                                                                                                                                                                                                                                                                                                               
transforms3d==0.3.1                                                                                                                                                                                                                                                                                                                                                        
Twisted==21.7.0                                                                                                                                                                                                                                                                                                                                                            
txaio==21.2.1                                                                                                                                                                                                                                                                                                                                                              
typing-extensions==3.10.0.2                                                                                                                                                                                                                                                                                                                                                
urllib3==1.26.9                                                                                                                                                                                                                                                                                                                                                            
virtualenv==20.13.0                                                                                                                                                                                                                                                                                                                                                        
vtk==9.0.3                                                                                                                                                                                                                                                                                                                                                                 
wslink==1.0.7                                                                                                                                                                                                                                                                                                                                                              
yarl==1.6.3                                                                                                                                                                                                                                                                                                                                                                
zipp==3.8.0                                                                                                                                                                                                                                                                                                                                                                
zope.interface==5.4.0

[akaszynski]

@Keynaan, sorry for the delayed response. I'm looking into this right now.

[akaszynski]

Perfectly reproduced this. Coming out with a fix shortly (example requires the fix, so don't run it just yet):

>>> from ansys.mapdl.reader import Archive, save_as_archive
>>> import pyvista as pv
>>> d = 250
>>> grid = pv.UniformGrid(dims=(d, d, d)).cast_to_unstructured_grid()
>>> filename = '/tmp/tmp.cdb'
>>> save_as_archive(filename, grid)
>>> print('File size:', os.path.getsize(filename)//1024**2, 'MB')
File size: 3519 MB
>>> Archive(filename)
ANSYS Archive File tmp.cdb
  Number of Nodes:              0
  Number of Elements:           0
  Number of Element Types:      0
  Number of Node Components:    0
  Number of Element Components: 0

Thanks for your excellent work identifying the issue and reporting your debugging env.

[akaszynski]

Resolved as of ansys-mapdl-reader==0.52.1.

Thanks again for pointing out this issue.

[Keynaan]

Perfect! It works. Thank you.