Fix/meta/full index #3177

carpawell · 2025-02-26T01:52:40Z

No description provided.

codecov · 2025-02-26T14:34:58Z

Codecov Report

Attention: Patch coverage is 73.80952% with 77 lines in your changes missing coverage. Please review.

Project coverage is 23.35%. Comparing base (414d3e9) to head (a39907e).

Files with missing lines	Patch %	Lines
pkg/services/meta/containers.go	80.97%	28 Missing and 15 partials ⚠️
cmd/neofs-node/meta.go	0.00%	21 Missing ⚠️
pkg/services/meta/blocks.go	77.50%	6 Missing and 3 partials ⚠️
pkg/services/meta/meta.go	0.00%	2 Missing ⚠️
pkg/local_object_storage/metabase/version.go	50.00%	0 Missing and 1 partial ⚠️
pkg/services/meta/notifications.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3177      +/-   ##
==========================================
+ Coverage   23.13%   23.35%   +0.21%     
==========================================
  Files         760      760              
  Lines       60498    60708     +210     
==========================================
+ Hits        13999    14179     +180     
- Misses      45504    45517      +13     
- Partials      995     1012      +17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

roman-khimov · 2025-02-26T14:48:30Z

pkg/services/meta/blocks.go

-			l.Error("handling object notification", zap.Error(err))
-			continue
-		}
+		wg.Go(func() error {


Random order?

to me it is more like this: #3178
imo we should discuss it this way, otherwise order means nothing

roman-khimov · 2025-02-26T14:52:13Z

pkg/services/meta/containers.go

@@ -44,71 +48,195 @@ func (s *containerStorage) drop() error {
 	return nil
 }

-func (s *containerStorage) putObject(e objEvent) error {
+func (s *containerStorage) putObject(e objEvent, h objectsdk.Object) error {


This better be separated, you don't need h for the base index and it should operate in the same fashion. Then you can pass the changeset (in whatever fashion) to the other routine maintaining full index that will fetch things (potentially concurrently) and process them (likely more sequentially).

what do you target? the whole meta service is a concurrent routine that does not block any operations. i do not know how to implement if i do not understand requirements

It can still be concurrent to the main index, there is zero contradiction here.

cthulhu-rider · 2025-02-27T08:03:53Z

pkg/services/meta/blocks.go

+		wg.Go(func() error {
+			err := m.handleObjectNotification(ctx, s, ev)
+			if err != nil {
+				return fmt.Errorf("handling %s/%s object notification: %w", ev.cID, ev.oID, err)


previously we logged all errors, now only the earliest one. I tink they all deserve attention, so suggest to keep logging here, and ignore on Wait()

will fix depening of what we will decide with @roman-khimov about object handing order in the thread above

cthulhu-rider · 2025-02-27T08:11:27Z

pkg/services/meta/containers.go

 	s.m.Lock()
 	defer s.m.Unlock()

-	newKVs := make(map[string][]byte)
+	dbKVs := make(map[string][]byte)
+	mptKVs := make(map[string][]byte)


we can add its els directly to s.opsBatch, right?

we can but dbKVs and commonKVs are still required so imo it is ok to keep them the same. this map should be like ~10 fields long, i do not thinks it even escaped. also it consists of string which made this change almost useless

but ok

cthulhu-rider · 2025-02-27T08:14:02Z

pkg/services/meta/containers.go

+		commonKVs[string(append([]byte{0, deletedIndex}, commsuffix...))] = e.deletedObjects
+		clearMpt, clearDB := deleteObjectsOps(s.db, e.deletedObjects)
+		maps.Copy(mptKVs, clearMpt)
+		maps.Copy(dbKVs, clearDB)


these maps can be accepted and set by deleteObjectsOps so we dont need to copy them

makes sence with latest changes

cthulhu-rider · 2025-02-27T08:16:57Z

pkg/services/meta/containers.go

+	phy := hasParent || (fPart.IsZero() && parID.IsZero())
+	pldHash, ok := h.PayloadChecksum()
+	if !ok {
+		return errors.New("missing payload checksum")


i'd handle all error cases on the func top

get object's fields in a sep func to pass many args to this func? how it helps?

cthulhu-rider · 2025-02-27T08:27:30Z

cmd/neofs-node/meta.go

+
+	var hPrm getsvc.HeadPrm
+	hPrm.SetHeaderWriter(&hw)
+	hPrm.WithAddress(addr)


Suggested change

hPrm.WithAddress(addr)

hPrm.WithAddress(oid.NewAddress(cID, oID))

cthulhu-rider · 2025-02-27T08:57:04Z

pkg/services/meta/containers.go

+
+				resDB[string(keyToDrop)] = nil
+
+				if vInt, isInt := parseInt(string(attrV)); isInt {


again can panic

cthulhu-rider · 2025-02-27T09:00:15Z

pkg/services/meta/containers.go

+					keyToDrop = append(keyToDrop, attrToIntIndex)
+					keyToDrop = append(keyToDrop, attributeDelimiter...)
+					keyToDrop = keyToDrop[:len(keyToDrop)+intValLen]
+					putBigInt(keyToDrop, vInt)


wrong place to put

cthulhu-rider · 2025-02-27T09:03:09Z

pkg/services/meta/meta.go

 	// List returns node's containers that support chain-based meta data and
 	// any error that does not allow listing.
 	List() (map[cid.ID]struct{}, error)
 	// IsMineWithMeta checks if the given CID has meta enabled and current
 	// node belongs to it.
 	IsMineWithMeta(cid.ID) (bool, error)
+	// Head returns actual object header from the NeoFS network (non-local
+	// objects should also be returned). Missing, removed object statuses
+	// must be reported according to API statuses from SDK.


this statement is not needed actually, Meta does not care what error is returned

whay do you mean? an object can be removed, not found, etc. i can imagine future status logic based on response

cthulhu-rider · 2025-02-27T09:03:57Z

pkg/services/meta/meta.go

 	// List returns node's containers that support chain-based meta data and
 	// any error that does not allow listing.
 	List() (map[cid.ID]struct{}, error)
 	// IsMineWithMeta checks if the given CID has meta enabled and current
 	// node belongs to it.
 	IsMineWithMeta(cid.ID) (bool, error)
+	// Head returns actual object header from the NeoFS network (non-local


objects are immutable so header cannot be other than actual

"actual" means exist/lost. it hightly relates this issue: #3140

cthulhu-rider · 2025-02-27T09:06:36Z

pkg/services/meta/notifications.go

-	err := s.putObject(e)
+	h, err := m.net.Head(ctx, e.cID, e.oID)
+	if err != nil {
+		return fmt.Errorf("HEAD rpc error: %w", err)


there is no guarantee that the RPC has been actually done - some preliminary errors are possible, or only local execution. Simply

Suggested change

return fmt.Errorf("HEAD rpc error: %w", err)

return fmt.Errorf("HEAD object: %w", err)

well, changed but why?

Make underlying KV database store the same index structure as in the new metabase structure for search V2. Exclude repetitive fields from the database in the old MPT form. Receive object headers via internal object get (head) service. Closes #3139. Signed-off-by: Pavel Karpy <[email protected]>

This functionality will be shared for some time meta-service feature is being developed. Refs #3139. Signed-off-by: Pavel Karpy <[email protected]>

Refs #3139. Signed-off-by: Pavel Karpy <[email protected]>

carpawell · 2025-03-01T01:47:13Z

https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/2664-1740774980/index.html

carpawell self-assigned this Feb 26, 2025

carpawell force-pushed the fix/meta/full-index branch from cb36362 to 981f5cf Compare February 26, 2025 14:32

carpawell marked this pull request as ready for review February 26, 2025 14:41

carpawell requested review from roman-khimov and cthulhu-rider as code owners February 26, 2025 14:41

roman-khimov reviewed Feb 26, 2025

View reviewed changes

carpawell force-pushed the fix/meta/full-index branch from 981f5cf to 3299a41 Compare February 26, 2025 20:37

carpawell mentioned this pull request Feb 26, 2025

Validate (refuse) events from chain by metaservice based on current state #3178

Open

carpawell force-pushed the fix/meta/full-index branch from 3299a41 to 5088635 Compare February 26, 2025 20:43

carpawell requested a review from roman-khimov February 26, 2025 20:48

cthulhu-rider reviewed Feb 27, 2025

View reviewed changes

carpawell force-pushed the fix/meta/full-index branch 2 times, most recently from 9943362 to f7ebf89 Compare February 27, 2025 14:49

carpawell marked this pull request as draft February 27, 2025 17:09

carpawell added 3 commits February 28, 2025 22:06

metabase: export PutMetadataForObject

de3e075

This functionality will be shared for some time meta-service feature is being developed. Refs #3139. Signed-off-by: Pavel Karpy <[email protected]>

node/meta: add meta indexes compatibility test

a39907e

Refs #3139. Signed-off-by: Pavel Karpy <[email protected]>

carpawell force-pushed the fix/meta/full-index branch from 5d22263 to a39907e Compare February 28, 2025 19:07

carpawell requested a review from cthulhu-rider March 1, 2025 01:46

carpawell marked this pull request as ready for review March 1, 2025 01:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix/meta/full index #3177

Fix/meta/full index #3177

carpawell commented Feb 26, 2025

codecov bot commented Feb 26, 2025 •

edited

Loading

roman-khimov Feb 26, 2025

carpawell Feb 26, 2025

roman-khimov Feb 26, 2025

carpawell Feb 26, 2025 •

edited

Loading

roman-khimov Feb 26, 2025

cthulhu-rider Feb 27, 2025

carpawell Feb 27, 2025 •

edited

Loading

cthulhu-rider Feb 27, 2025

carpawell Feb 27, 2025

cthulhu-rider Feb 27, 2025

carpawell Feb 27, 2025

cthulhu-rider Feb 27, 2025

carpawell Feb 27, 2025

cthulhu-rider Feb 27, 2025

cthulhu-rider Feb 27, 2025

cthulhu-rider Feb 27, 2025

cthulhu-rider Feb 27, 2025

carpawell Feb 27, 2025 •

edited

Loading

cthulhu-rider Feb 27, 2025

carpawell Feb 27, 2025

cthulhu-rider Feb 27, 2025

carpawell Feb 27, 2025

carpawell commented Mar 1, 2025

	hPrm.WithAddress(addr)
	hPrm.WithAddress(oid.NewAddress(cID, oID))


		resDB[string(keyToDrop)] = nil

		if vInt, isInt := parseInt(string(attrV)); isInt {

	return fmt.Errorf("HEAD rpc error: %w", err)
	return fmt.Errorf("HEAD object: %w", err)

Fix/meta/full index #3177

Are you sure you want to change the base?

Fix/meta/full index #3177

Conversation

carpawell commented Feb 26, 2025

codecov bot commented Feb 26, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carpawell Feb 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carpawell Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carpawell Feb 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carpawell commented Mar 1, 2025

codecov bot commented Feb 26, 2025 •

edited

Loading

carpawell Feb 26, 2025 •

edited

Loading

carpawell Feb 27, 2025 •

edited

Loading

carpawell Feb 27, 2025 •

edited

Loading