Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Traverse object with Buffer field is SLOW #9

Open
jekanik opened this issue May 22, 2024 · 5 comments
Open

Traverse object with Buffer field is SLOW #9

jekanik opened this issue May 22, 2024 · 5 comments

Comments

@jekanik
Copy link

jekanik commented May 22, 2024

Hi @ljharb!
We are using traverse in case where we need to remove sensitive or inappropriate data from object before logging it.
And there are cases when we have Buffer field in this object. We assuming that this field is sensitive and trying to filter it out.
But, unfortunately, CPU profile showed that service spends ~500ms of CPU time inside of traverse.

How to reproduce:

    const obj = {
        data: readFileSync(join(__dirname, 'image.webp')), // image is 1.8MB
    };

    console.time('test_timer');

    const updated = traverse(obj).map(function () {
        if (this.key !== undefined && this.key === 'data') {
            this.update('secret');
        }
    });

    console.timeEnd('test_timer');

Logs:
v. 0.6.6

test_timer: 331 ms

v. 0.6.9

test_timer: 116 ms

Update of the lib makes it better but still it's weird to have 100+ms latency for such case.

Could you please advice, if we are misusing lib or something else?
Thanks

@jekanik
Copy link
Author

jekanik commented May 22, 2024

Small addition - CPU profile (meaningful part)
image

@ljharb
Copy link
Owner

ljharb commented May 23, 2024

I'm assuming that when you say "ByteBuffer" you're referring to a node Buffer object?

@ljharb
Copy link
Owner

ljharb commented May 23, 2024

Potentially #2 is related.

@jekanik jekanik changed the title Traverse object with ByteBuffer field is SLOW Traverse object with Buffer field is SLOW May 23, 2024
@jekanik
Copy link
Author

jekanik commented May 23, 2024

I'm assuming that when you say "ByteBuffer" you're referring to a node Buffer object?

Indeed. My Java experience accidentally kicked in :)

Potentially #2 is related.

Tbh I'd prefer not to copy 1.8MB buffer :)
P.S.: As I see map copies object, but not forEach. And still with forEach timing is similar.

Thanks for response @ljharb

@ljharb
Copy link
Owner

ljharb commented Jun 18, 2024

To be clear, the issue isn't about copying, it's about traversing objects that are "meant" to be treated as leaves/atoms, like a Buffer or a URL (see #10).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants