[Bug]: The program exited abnormally while processing the super large model

[verybigzhouhai]

What happened?

Hi,
The program exited abnormally while processing the super large model!

Recently, I encountered an unexpected program exit while working on a model containing 300000 components. The error message is shown in the following figure.

Aborted()
web-ifc-api-node.js:5175
(node:57980) UnhandledPromiseRejectionWarning: RuntimeError: Aborted(). Build with -sASSERTIONS for more info.
at abort (F:\CJXX_WORK\Project\bimrun23dtiles\web-ifc\web-ifc-api-node.js:5179:19)
at _abort (F:\CJXX_WORK\Project\bimrun23dtiles\web-ifc\web-ifc-api-node.js:7139:11)
at :wasm-function[52]:0x16ff
at :wasm-function[345]:0x37baf
at :wasm-function[523]:0x76527
at :wasm-function[127]:0xf9b1
at :wasm-function[879]:0xbdbc4
at :wasm-function[889]:0xbe056
at :wasm-function[213]:0x225b9
at :wasm-function[1325]:0xe21f8
(Use node --trace-warnings ... to show where the warning was created)
warning.js:43
(node:57980) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
warning.js:43
(node:57980) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

For this, I created a simple testing program that only writes simple triangular data. When modelCount=150000, the program cannot execute correctly. When modelCount=100000, the ifc file can be successfully output.
It should be noted that program abnormal exit does not only occur during saveFile, but also during WriteLine when the modelCount is large enough (300000)
test code:
`

const fs = require('fs');
const path = require("path");
const { FILE_DESCRIPTION,Schemas,logical,NewIfcModel, ms,IfcAPI, Handle, IFC4, IFC2X3, IFCBUILDINGSTOREY, IFCPROPERTYSINGLEVALUE, IFCSIUNIT, EMPTY, IFCPROPERTYSET, IFCOWNERHISTORY, IFCRELDEFINESBYPROPERTIES } = require("../web-ifc/web-ifc-api-node.js");
const ifcapi = new IfcAPI();

async function createIFC() {

    const filename = "F:\\temp\\ifc\\output\\testexport.ifc"
    await ifcapi.Init();
    
    if (!fs.existsSync(filename)) {
        if (!fs.existsSync(path.dirname(filename))) {
            fs.mkdirSync(path.dirname(filename), {recursive: true});
        }
    }

    const modelOpton = {
        schema: Schemas.IFC2X3,  // ifc版本
        name: "test.ifc",
        description: ["1", "2"],
        authors: ["3", "4"],
        organizations: ["5", "6"],
        authorization: "78",
    }

    const loaderSettings = {}

    let modelID = ifcapi.CreateModel(modelOpton, loaderSettings);

    const modelCount = 150000;
    for (let i = 0; i < modelCount; i++) {
        console.log(`${i}/${modelCount}`)
        for (let j = 0; j < 50; j++) {
            let cartPoint1 = new IFC2X3.IfcCartesianPoint([1,2,3]);
            let cartPoint2 = new IFC2X3.IfcCartesianPoint([4,5,6]);
            let cartPoint3 = new IFC2X3.IfcCartesianPoint([7,8,9]);

            let array = [cartPoint1, cartPoint2, cartPoint3];

            let poly = new IFC2X3.IfcPolyLoop(array);
            ifcapi.WriteLine(modelID, poly);
        }
                
    }
    // save file
    fs.writeFileSync(filename, ifcapi.SaveModel(modelID));
    console.log("ifc exported successfully!")
}

function generatorUUID(){
    return Math.random().toString(36).substr(2, 9)+Math.random().toString(36).substr(2, 9)+Math.random().toString(36).substr(2, 4)
}

createIFC()

`

Version

0.0.46

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Anything else?

No response

[verybigzhouhai]

Hi, @beachtom ,Is there any new progress on this issue

[beachtom]

Yes I have identified the cause but still need to code up the fix

[verybigzhouhai]

@beachtom I am glad that the cause of the problem has been identified. Is there an expected timeline? This is important to me. Unfortunately, I am not a C++programmer, otherwise I could have made some contributions. thanks

[beachtom]

I am hoping in the next week. Sent from Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Zhouhai ***@***.***> Sent: Friday, January 5, 2024 3:08:58 AM To: IFCjs/web-ifc ***@***.***> Cc: Thomas Beach ***@***.***>; Mention ***@***.***> Subject: Re: [IFCjs/web-ifc] [Bug]: The program exited abnormally while processing the super large model (Issue #538) External email to Cardiff University - Take care when replying/opening attachments or links. Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni. @beachtom<https://github.com/beachtom> I am glad that the cause of the problem has been identified. Is there an expected timeline? This is important to me. Unfortunately, I am not a C++programmer, otherwise I could have made some contributions. thanks — Reply to this email directly, view it on GitHub<#538 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGIIJHVAO5CG5DP3JFLFLQLYM5VEVAVCNFSM6AAAAABBCNWVDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZYGA3DENBUGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[verybigzhouhai]

Hi, @beachtom ,Sorry to ask again, is there any progress on this issue?

[beachtom]

This is quite a lot more tricky that we thought. Essentially, it seems like you are overfilling the WASM memory so we need to write the data out when that happens.

[beachtom]

So this should now be fixed. I also added a method to write files out using a callback, which makes it a lot quicker in the case of huge files (see the WebIfcApi.spec.ts test lines 536 as an example)

[verybigzhouhai]

@beachtom Thank you very much for making the modifications to this issue. I have tested it and found that the program still exits abnormally when modelCount=30,0000.

[beachtom]

So this is a different problem - you are now hitting I think a hard limit of 4GB which is what WASM can hold in memory. 150000 is a 2GB IFC file - so 300000 will be a 4GB file so this makes about sense. Realistically is there a use case for a model this big?

[verybigzhouhai]

@beachtom , For complex BIM models, this should be very common. In my two projects, there were over 300000 components, and I could only export 100000. Due to the different number of triangles for each component, in another project, there were 40000 components, but I could only export 8000. Of course, since the model has a large amount of vertex data that can be reused, I have not yet considered this in the program. If there is a 4GB WASM memory limit, how about using this as a node C++ plugin? We usually don't operate such a large IFC file on the web, but in node.js programs, 4GB can easily become a limit

[beachtom]

So the limit only applies for created lines.

So if you are loading a model - you cannot add more than 4GB of IFC lines to it.

Can you describe what exactly you want to do so I can suggest the best way to achieve it - or see how we can fix it.

[verybigzhouhai]

In our system, it is necessary to export existing geometric data to IFC format. These raw data may come from 3D model formats such as. rvt/fbx \ obj, etc. The exported IFC files can be shared with others, so my usage will only be to create IFC files from scratch, not to load existing IFC files. Moreover, most of our model data is complex building information models, so there are many model details, There is a large amount of triangular data

[beachtom]

OK now I understand. I will look into this further.

Do you have any way of predicting the size of the model before you start exporting to IFC?

[verybigzhouhai]

I think the size can be determined by the amount of vertex data, which is known to me before exporting.

[verybigzhouhai]

I may need to use the C++version temporarily, and I noticed that the ifcLoader only has the RemoveLine method and no method to add rows. What if I quickly use the loader to add rows? I have looked at the WriteLine method in web-ifc-api.cpp. Do I need to customize a method for adding rows to ifcLoader? So that I can create, modify, and output IFC files in web-ifc-test.cpp

[beachtom]

So I am working on a node API version. But in the meantime my suggested approach would be to generate two IFC files - start the second one at a higher expressID and then as a post-processing step merge them

[verybigzhouhai]

Okay, I got it！谢谢!

[verybigzhouhai]

I tried to generate a certain amount of data into different files, and when a file is generated, I call ifcap.CloseModel (modelID), and then call ifcapi again Create a new model and obtain a new modelID, but during this process, the memory still dynamically increases, eventually reaching 4GB. Then, the program exits. Did the CloseModel method correctly release the memory? I don't quite understand whether the IfcTokenStream and IfcFileStream sections need to release memory.
I added a parameter to control the number of lines in each file, and now it can output 315000 lines, which is not much better than before.

    const fs = require('fs');
const path = require("path");
const { FILE_DESCRIPTION,Schemas,logical,NewIfcModel, ms,IfcAPI, Handle, IFC4, IFC2X3, IFCBUILDINGSTOREY, IFCPROPERTYSINGLEVALUE, IFCSIUNIT, EMPTY, IFCPROPERTYSET, IFCOWNERHISTORY, IFCRELDEFINESBYPROPERTIES } = require("../web-ifc/web-ifc-api-node.js");

const ifcapi = new IfcAPI();

async function createIFC() {

    const filename = "F:\\temp\\ifc\\output\\testexport.ifc"
    await ifcapi.Init();
    
    if (!fs.existsSync(filename)) {
        if (!fs.existsSync(path.dirname(filename))) {
            fs.mkdirSync(path.dirname(filename), {recursive: true});
        }
    }

    const maxModelCountSingleFile = 10000;

    const modelOpton = {
        schema: Schemas.IFC2X3,  // ifc版本
        name: "test.ifc",
        description: ["1", "2"],
        authors: ["3", "4"],
        organizations: ["5", "6"],
        authorization: "78",
    }

    const loaderSettings = {
    } 

    const modelCount = 1000000;
    let modelID = 0;
    let maxExpressID = 1;
    for (let i = 0; i < modelCount; i++) {
        if (i % maxModelCountSingleFile === 0) {
            // save file
            if (i != 0) {
                fs.writeFileSync(filename.replace(".ifc", `${modelID}.ifc`), ifcapi.SaveModel(modelID))
                ifcapi.CloseModel(modelID)
            }

            modelID = ifcapi.CreateModel(modelOpton, loaderSettings);
            console.log(modelID)
        }
        console.log(`${i}/${modelCount}`)
        for (let j = 0; j < 50; j++) {
            let cartPoint1 = new IFC2X3.IfcCartesianPoint([modelID,modelID,modelID]);
            cartPoint1.expressID = maxExpressID++;
            let cartPoint2 = new IFC2X3.IfcCartesianPoint([modelID,modelID,modelID]);
            cartPoint2.expressID = maxExpressID++;
            let cartPoint3 = new IFC2X3.IfcCartesianPoint([modelID,modelID,modelID]);
            cartPoint3.expressID = maxExpressID++;

            let array = [cartPoint1, cartPoint2, cartPoint3];

            let poly = new IFC2X3.IfcPolyLoop(array);
            poly.expressID = maxExpressID++;
            ifcapi.WriteLine(modelID, poly);
        }
               
    }

    

    // save file
    fs.writeFileSync(filename.replace(".ifc", `${modelID}.ifc`), ifcapi.SaveModel(modelID)); 
    console.log("ifc exported successfully!")
}

function generatorUUID(){
    return Math.random().toString(36).substr(2, 9)+Math.random().toString(36).substr(2, 9)+Math.random().toString(36).substr(2, 4)
}

createIFC()

[beachtom]

Let me look into this - the memory should be being cleared

[beachtom]

There was a memory leak - I have just fixed it.

If you try what is on main

[verybigzhouhai]

Do I just need to call Ifcap.CloseModel(modelID) to free memory? I downloaded the latest main branch code and compiled it, but the problem still exists. Do you have test cases

[beachtom]

There was another memory leak - I think that is fixed now. Can you let me know

[verybigzhouhai]

The problem still exists, and the total number of rows written has not changed much

[verybigzhouhai]

It should be noted that I do not load the model from a file, but instead create a model from the original. So, does the judgment on line 21 of the IfcTokenChunk.cpp file prevent us from releasing memory
https://github.com/IFCjs/web-ifc/blob/8b74ba536a36fce927fa49e75b27291f066191c0/src/cpp/parsing/IfcTokenChunk.cpp#L21

[beachtom]

Well spotted - let me test

[verybigzhouhai]

How to start with a higher expressID? Is the approach here correct?#538 (comment) I seem to have found that incorrect content has been exported in the new file, and I haven't found any patterns yet. It's still being tested.

[beachtom]

So the memory leak is fixed!

To start with a higher ID create an object

let cartPoint1 = ifcApi.CreateIfcEntity(newModID,IFCCARTESIANPOINT,[ifcApi.CreateIfcType(newModID,IFCLENGTHMEASURE,1),ifcApi.CreateIfcType(newModID,IFCLENGTHMEASURE,2),ifcApi.CreateIfcType(newModID,IFCLENGTHMEASURE,3)]);

then override the ID

cartPoint.expressID = XXX;

If this doesn't work let me know the issue and the output and I will fix

[verybigzhouhai]

I have some new findings, the second exported ifc file is not quite correct. As shown in the following figure.

const fs = require('fs');
const path = require("path");
const { FILE_DESCRIPTION,Schemas,logical,NewIfcModel, ms,IfcAPI,IFCCARTESIANPOINT,IFCFACE,IFCFACEOUTERBOUND,IFCLENGTHMEASURE,IFCPOLYLOOP, Handle, IFC4, IFC2X3, IFCBUILDINGSTOREY, IFCPROPERTYSINGLEVALUE, IFCSIUNIT, EMPTY, IFCPROPERTYSET, IFCOWNERHISTORY, IFCRELDEFINESBYPROPERTIES } = require("../web-ifc/web-ifc-api-node.js");

const ifcapi = new IfcAPI();

async function createIFC() {

    const filename = "F:\\temp\\ifc\\output\\testexport.ifc"
    await ifcapi.Init();
    
    if (!fs.existsSync(filename)) {
        if (!fs.existsSync(path.dirname(filename))) {
            fs.mkdirSync(path.dirname(filename), {recursive: true});
        }
    }

    const modelOpton = {
        schema: Schemas.IFC2X3,  // ifc版本
        name: "test.ifc",
        description: ["1", "2"],
        authors: ["3", "4"],
        organizations: ["5", "6"],
        authorization: "78",
    }

    const loaderSettings = {
        // LINEWRITER_BUFFER: 100,
        // MEMORY_LIMIT: 4294967295
    } 

    const modelCount = 2;
    let modelID = 0;
    let maxExpressID = 1;
    for (let i = 0; i < modelCount; i++) {

        modelID = ifcapi.CreateModel(modelOpton, loaderSettings);

        console.log(`${modelID}/${modelCount}`)
        const faces = [];

        let cartPoint1 = ifcapi.CreateIfcEntity(modelID,IFCCARTESIANPOINT,[ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,1),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,2),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,3)]);
        cartPoint1.expressID = maxExpressID++;
        let cartPoint2 = ifcapi.CreateIfcEntity(modelID,IFCCARTESIANPOINT,[ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,4),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,5),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,6)]);
        cartPoint2.expressID = maxExpressID++;
        let cartPoint3 = ifcapi.CreateIfcEntity(modelID,IFCCARTESIANPOINT,[ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,7),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,8),ifcapi.CreateIfcType(modelID,IFCLENGTHMEASURE,9)]);
        cartPoint3.expressID = maxExpressID++;
        let array = [cartPoint1, cartPoint2, cartPoint3];
        let poly = ifcapi.CreateIfcEntity(modelID,IFCPOLYLOOP, array);
        poly.expressID = maxExpressID++;
        ifcapi.WriteLine(modelID, poly);

        const faceOuterBound = ifcapi.CreateIfcEntity(modelID,IFCFACEOUTERBOUND, poly, true)
        faceOuterBound.expressID = maxExpressID++;
        ifcapi.WriteLine(modelID, faceOuterBound);

        const face = ifcapi.CreateIfcEntity(modelID,IFCFACE, [faceOuterBound])
        face.expressID = maxExpressID++;
        faces.push(face);
        ifcapi.WriteLine(modelID, face);
        
        fs.appendFileSync(filename, ifcapi.SaveModel(modelID)); 
        fs.appendFileSync(filename, "\n---------------------\n"); 
        ifcapi.CloseModel(modelID)
    }

    
    console.log("ifc exported successfully!")

}


createIFC()

[beachtom]

Yes that is a bug you have uncovered. I will take a look hopefully today

[verybigzhouhai]

I have some new findings that multiple exported rows come from duplicate writes, as there is a mutual referencing relationship between IFCCARTESIANPOINT, IfcFaceOuterBound, IfcFace, and IfcClosedShell, so duplicate writes occur in WriteLine.
https://github.com/IFCjs/web-ifc/blob/c860b2ca43dce268b581e3248d834c1501e77143/src/ts/web-ifc-api.ts#L598
There are two questions here:
The first question, in the example provided above, only those with modelId>0 will have duplicate outputs. The first model will not have this situation in any case, but they all have duplicate writes.
The second issue is that when I discover that an object has previous references, we can proactively not execute a writeline on it.
However, there is a problem here where many elements are referenced multiple times, such as IfcBuildingStorey and IfcBuildingElementProxy, so it is still impossible to completely avoid them. Alternatively, we can add a property to IfcEntity to determine whether to automatically execute the writeline or cancel the automatic writeline, It is entirely up to the user to decide.

[beachtom]

Yeah your findings match mine. I am not quite sure what is causing it at the moment

[beachtom]

I've fixed the duplicate lines issue!

[verybigzhouhai]

Great! 发自我的iPhone

…

------------------ Original ------------------ From: Tom Beach ***@***.***&gt; Date: Sat,Feb 3,2024 6:27 PM To: IFCjs/web-ifc ***@***.***&gt; Cc: Zhouhai ***@***.***&gt;, Author ***@***.***&gt; Subject: Re: [IFCjs/web-ifc] [Bug]: The program exited abnormally whileprocessing the super large model (Issue #538) I've fixed the duplicate lines issue! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

[verybigzhouhai]

Thank you. I'll test it later. 发自我的iPhone

…

------------------ Original ------------------ From: Tom Beach ***@***.***&gt; Date: Sat,Feb 3,2024 6:27 PM To: IFCjs/web-ifc ***@***.***&gt; Cc: Zhouhai ***@***.***&gt;, Author ***@***.***&gt; Subject: Re: [IFCjs/web-ifc] [Bug]: The program exited abnormally whileprocessing the super large model (Issue #538) I've fixed the duplicate lines issue! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

[verybigzhouhai]

Great, the program is running well now. I have successfully created a super large IFC file.
Thank you very much for your help!
At the end of this question, my small suggestion is to add two parameters to the saveFile to determine whether additional information about ifc needs to be output. Because when we export models in batches, in most cases we still need to merge them into one file. If we output each model as a standard ifc file, then we still need to handle a large amount of file merging in the end, which requires time and performance investment.
Divide an IFC into three parts. In the first model, only two parts 1 and 2 need to be exported, in the last model, only two parts 2 and 3 need to be exported, and in other models, only the second part needs to be exported. In this way, we can directly write the content returned by the saveFile function to the same file without the need for subsequent processing.

Of course, this is just a suggestion. Thank you again!

[verybigzhouhai]

@beachtom Hi, I'm sorry, there are still some issues here. This fix 4f80542 seems to have caused a new memory leak. The memory kept growing slowly until it was not used enough.

[beachtom]

ok let me check that out