Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add RW locking to better manage concurrency #8997

Merged
merged 2 commits into from
Jul 17, 2024

Conversation

tjquinno
Copy link
Member

Description

Resolves #8980

The Helidon metrics implementation which uses Micrometer involves several interrelated data structures. Updates to these have always been protected by a lock, but the referenced issue reveals an oversight: read-only operations (which did not participate in locking) could fail in odd ways because the backing data structure for a returned list or set could change.

This PR enhances the locking scheme to use a ReadWriteReentrantLock.

Notable points:

  • Methods which read from the internal data structures now use the read lock.
  • Methods which update the data structures now use the write lock.
  • One method, getOrCreate, starts with a read lock and "promotes" to the write lock as needed. Of course RWRL does not itself support that but the changes to getOrCreate do. See the comments in the revision to the code for details.

Documentation

Bug fix; no doc update.

@tjquinno tjquinno self-assigned this Jul 16, 2024
@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Jul 16, 2024
Copy link
Member

@ljnelson ljnelson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of gotchas

lock.readLock().lock();
List<io.helidon.metrics.api.Meter> temp;
try {
temp = new ArrayList<>(meters.values());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm; this idiom with the streaming below it seems verbose. What about:

return List.copyOf(meters.values()).stream()
    .map(...)
    ....

…all here under lock instead?

Copy link
Member Author

@tjquinno tjquinno Jul 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copying the array inside the lock but doing the rest of the work outside lets us release the lock sooner. Maybe not a huge difference but we might as well hold the lock for as short a time as we can.

return scopeMembership.keySet();
lock.readLock().lock();
try {
return new HashSet<>(scopeMembership.keySet());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it OK if someone casts the returned Iterable to a Set and starts adding things to it? If not, consider:

return Set.copyOf(scopeMembership.keySet());

Copy link
Member Author

@tjquinno tjquinno Jul 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's OK as far as I'm concerned.

We've copied the keys into a new set that is not backed by the original map, so the calling code can do what it wants with the returned set without disturbing our internal set.

Or did I miss your point?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just meant: what are the semantics of the return value? I tend to prefer: "the return value is immutable". You're saying, I think: "the return value's mutability is undefined". Either can be true; just wanted clarification.

}
return result;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Watch mutability.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, we create a new result set that is not backed by our internal set, so I'm not sure why we'd restrict what the caller can do with the return value.

try {
onAddListeners.forEach(listener -> listener.accept(result));
} catch (Exception e) {
// Ignored.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ding ding ding. My alarm bells go off; worthy of a log at least I would think

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[I responded to this comment earlier but maybe I didn't save it.]

I've removed the try/catch block. Everywhere else where we invoke the listeners we (intentionally) let any exceptions bubble up. We should do the same here.

}

lock.lock();
/*
"Promote" our lock from read to write.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

…although you don't have any lock at this point?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, not at that moment--hence the quotes around "promote" because in fact just a few lines above we had acquired the read lock and the effect we want is to promote the lock from read to write as the comment a few lines below explains.

I've rewritten the comments to remove any possible confusion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Acquiring WL and then checking condition again to account for the interim no lock was held.

@tjquinno tjquinno requested a review from ljnelson July 17, 2024 13:19
@tjquinno tjquinno merged commit 793ffac into helidon-io:main Jul 17, 2024
12 checks passed
@tjquinno tjquinno deleted the 4.x-metrics-locking branch July 17, 2024 15:58
barchetta pushed a commit to barchetta/helidon that referenced this pull request Jul 17, 2024
* Add RW locking to better manage concurrency

Signed-off-by: Tim Quinn <[email protected]>

* Address review comments

---------

Signed-off-by: Tim Quinn <[email protected]>
barchetta added a commit that referenced this pull request Jul 17, 2024
* Add RW locking to better manage concurrency
---------

Signed-off-by: Tim Quinn <[email protected]>
Co-authored-by: Tim Quinn <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OCA Verified All contributors have signed the Oracle Contributor Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4.x MeterRegistry#meters() can throw IllegalStateException
3 participants