Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Networking: Multi-node libp2p testing ⚙️ #131

Closed
8 tasks done
Tracked by #331
zeeshanlakhani opened this issue May 3, 2023 · 0 comments · Fixed by #461
Closed
8 tasks done
Tracked by #331

Networking: Multi-node libp2p testing ⚙️ #131

zeeshanlakhani opened this issue May 3, 2023 · 0 comments · Fixed by #461
Assignees
@bgins
Labels
networking Features, functionality involving networking runtime Related to IPVM runtime specifics

Comments

@zeeshanlakhani
Copy link
Contributor

zeeshanlakhani commented May 3, 2023 •
edited by bgins
Loading

Summary

Test swarm, gossip, mdns, node-to-node connections, and event handling.

Where to steal from: https://github.com/libp2p/rust-libp2p/blob/master/protocols/kad/src/behaviour/test.rs, and poss

Goals

  • Test receipt gossip and DB update
  • Test receipt use over network (from dialed peer)
  • Test rendezvous setup (3 nodes)
  • Test MDNS dialing
  • Test nodes added/removed from DHT
  • Test Node Dialing
  • Test workflow info and receipt records retrieved from the DHT
  • Test workflow info retrieved directly from provider

ETA: 2023-11-10
@zeeshanlakhani zeeshanlakhani added rust runtime Related to IPVM runtime specifics labels May 3, 2023
@zeeshanlakhani zeeshanlakhani added good first issue Good for newcomers help wanted Extra attention is needed labels Jun 17, 2023
@zeeshanlakhani zeeshanlakhani added this to the MVP Development milestone Aug 21, 2023
@zeeshanlakhani zeeshanlakhani changed the title Re-use libp2p2 testing apparatus to test code around gossip, records libp2p2 testing apparatus to test code around gossip, records Aug 28, 2023
@bgins bgins self-assigned this Sep 26, 2023
@bgins bgins mentioned this issue Sep 26, 2023
Closed
4 tasks
@zeeshanlakhani zeeshanlakhani changed the title libp2p2 testing apparatus to test code around gossip, records Networking: Multi-node libp2p testing Sep 27, 2023
@zeeshanlakhani zeeshanlakhani removed the good first issue Good for newcomers label Sep 27, 2023
@zeeshanlakhani zeeshanlakhani mentioned this issue Sep 27, 2023
Open
18 tasks
@zeeshanlakhani zeeshanlakhani removed this from the MVP Development milestone Sep 27, 2023
@bgins bgins mentioned this issue Sep 27, 2023
Merged
13 tasks
@zeeshanlakhani zeeshanlakhani added networking Features, functionality involving networking and removed rust help wanted Extra attention is needed labels Sep 29, 2023
@zeeshanlakhani zeeshanlakhani changed the title Networking: Multi-node libp2p testing Networking: Multi-node libp2p testing ⚙️ Sep 29, 2023
zeeshanlakhani pushed a commit that referenced this issue Oct 10, 2023
@bgins
test: Add network tests (#335)
b0bda83 
# Description

This pull request implements the following features:

- [x] Add RPC listens on address test
- [x] Add Websocket listens on address test
- [x] Add libp2p generates peer ID test
- [x] Add libp2p known peers connect test
- [x] Add libp2p peers connect after mDNS discovery test
- [x] Add `console_subscriber_port` setting
- [x] Update swarm event logs so `peer_id` is always the first label (we
test logs assuming so)
- [x] Update logs for swarm peer connection closed event to debug and
move `peer_id` to a label
- [x] Add `kill_homestar` function (to abstract out some repeated code)
- [x] Turn off tokio console when running tests
- [x] Reorganize integration test fixture ports into a sparse sequence
across fixtures (gives us room to add tests, but keeps them somewhat in
order).

## Link to issue

Implements #131

Solving for  

- [X] Test MDNS dialing
- [X] Test Node Dialing

## Type of change

- [x] New feature (non-breaking change that adds functionality)
- [x] Refactor (non-breaking change that updates existing functionality)

## Test plan (required)

We are adding a test to check on existing functionality.
@bgins bgins mentioned this issue Oct 24, 2023
Merged
22 tasks
@bgins bgins mentioned this issue Nov 1, 2023
Closed
zeeshanlakhani added a commit that referenced this issue Nov 7, 2023
@bgins @zeeshanlakhani
refactor: Improve rendezvous protocol usage (#399)
5b5b2ff 
# Description

This pull request implements the following features:

- [x] Add event handler cache implementation
- [x] Renew rendezvous registration at expiration
- [x] Re-discover from rendezvous servers on a set interval
- [x] Renew discovered peer registrations when they expire
- [x] Remove behavior where we add other peers to `external_addresses`
- [x] Add check to not dial ourselves on rendezvous discovery
- [x] Update dialing on rendezvous discovery to only dial peers we
aren't already connected to
- [x] Add rendezvous server `Dialing`, `PeerRegistered`, and
`RegistrationExpired` event debug logs
- [x] Separate `enable_rendezvous` config into
`enable_rendezvous_server` and `enable_rendezvous_client` configs
(server is opt-in, client is opt-out)
- [x] Add `max_connected_peers` config
- [x] Add `rendezvous_registration_ttl` and
`rendezvous_discovery_interval` configs
- [x] Test rendezvous register, discover, and connect
- [x] Test disconnect after rendezvous connect
- [x] Test registration expires and re-registration on expiration
- [x] Test discovered registration expires and rediscovery attempted at
expiration
- [x] Test rediscovery on rendezvous discovery interval
- [x] Add `extract_timestamps_where` and `count_lines_where` test
utilities
- [x] Update libp2p deprecated Kademlia and SwarmBuilder interfaces
- [x] Update integration test fixtures to avoid overlapping ports

## Link to issue

Implements #131.

## Type of change

- [x] New feature (non-breaking change that adds functionality)
- [x] Refactor (non-breaking change that updates existing functionality)
- [x] Comments have been added/updated

## Test plan (required)

We are adding tests to check existing functionality and a test to check
registration renewals.

---------

Co-authored-by: Zeeshan Lakhani <[email protected]>
@bgins bgins mentioned this issue Nov 8, 2023
Merged
7 tasks
zeeshanlakhani added a commit that referenced this issue Nov 10, 2023
@bgins @zeeshanlakhani
feat: Add pubsub receipt sharing notifications (#418)
0a8d525 
# Description

This PR implements the following changes:

- [x] Add receipt published and received notifications
- [x] Update receipt sharing log messages
- [x] Add receipt sharing integration test
- [x] Add `remove_db` and `wait_for_socket_connection` test utilities
- [x] Update JSON `peer_id` key to `peerId`

## Link to issue

Implements #131 

## Type of change

- [x] New feature (non-breaking change that adds functionality)
- [x] Refactor (non-breaking change that updates existing functionality)

## Test plan (required)

This PR includes an integration test that checks for gossiped receipts
in websocket notifications and logs.

---------

Co-authored-by: Zeeshan Lakhani <[email protected]>
hugomrdias pushed a commit that referenced this issue Nov 14, 2023
@bgins @zeeshanlakhani @hugomrdias
feat: Add pubsub receipt sharing notifications (#418)
da42d01 
# Description

This PR implements the following changes:

- [x] Add receipt published and received notifications
- [x] Update receipt sharing log messages
- [x] Add receipt sharing integration test
- [x] Add `remove_db` and `wait_for_socket_connection` test utilities
- [x] Update JSON `peer_id` key to `peerId`

## Link to issue

Implements #131 

## Type of change

- [x] New feature (non-breaking change that adds functionality)
- [x] Refactor (non-breaking change that updates existing functionality)

## Test plan (required)

This PR includes an integration test that checks for gossiped receipts
in websocket notifications and logs.

---------

Co-authored-by: Zeeshan Lakhani <[email protected]>
zeeshanlakhani added a commit that referenced this issue Nov 29, 2023
@bgins @zeeshanlakhani
feat: Add pubsub receipt sharing notifications (#418)
476c47f 
# Description

This PR implements the following changes:

- [x] Add receipt published and received notifications
- [x] Update receipt sharing log messages
- [x] Add receipt sharing integration test
- [x] Add `remove_db` and `wait_for_socket_connection` test utilities
- [x] Update JSON `peer_id` key to `peerId`

## Link to issue

Implements #131 

## Type of change

- [x] New feature (non-breaking change that adds functionality)
- [x] Refactor (non-breaking change that updates existing functionality)

## Test plan (required)

This PR includes an integration test that checks for gossiped receipts
in websocket notifications and logs.

---------

Co-authored-by: Zeeshan Lakhani <[email protected]>
@bgins bgins mentioned this issue Dec 4, 2023
Merged
35 tasks
bgins added a commit that referenced this issue Jan 11, 2024
@bgins @QuinnWilton
feat: Add DHT and RequestResponse notifications (#461)
5c876c5 
# Description

This PR implements the following changes:

- [x] Add DHT notifications
  - [x] Put receipt and workflow info to DHT notifications
  - [x] Got receipt and workflow info from DHT notifications
  - [x] Receipt and workflow info quorum success notifications
  - [x] Receipt and workflow info quorum failure notifications
- [x] Add request-response notifications
  - [x] Sent workflow info to peer
  - [x] Received workflow info from peer
- [x] Update notifications to accept `Ipld` data, not just strings
- [x] Refactor swarm `FoundEvent` into `DecodedRecord` and `FoundEvent`
(to include extra information on `FoundEvent` without overloading
decoding mechanism)
- [x] Add better error for timed out records (records with the code
"Timeout")
- [x] Fix `IndexedResources` decoding and add/update unit tests to check
it
- [x] Fix flaky `test_libp2p_receipt_gossip_serial` integration test
- [x] Remove no connected peer checks (we dial up a peer if they are not
connected but stored in the DHT)
- [x] Rename `check_lines_for` test utility to `check_for_line_with`
- [x] Remove unused `handler_timeout_fn`s
- [x] Move `defaults.toml` from `homestar-runtime/fixtures` to
`homestar-runtime/config`
- [x] Update `ReceivedReceiptPubsub` notification "peerId" field to
"publisher" to better reflect the role of the peer (breaking change for
clients).
- [x] Remove receipt persistence on receiving workflow info (to be
replaced with another mechanism)
- [x] Break `retrieve_from_query` into `retrieve_from_dht` and
`retrieve_from_provider`
- [x] Update retrieve workflow info from provider mechanism to trigger
on `retrieve_from_dht` error or timeout
- [x] Update DHT behavior to only add addresses manually with the
`kad::BucketInserts::Manual` configuration
- [x] Handle swarm `RequestResponse` `ResponseSent` event with a debug
log (instead of leaving it uncaught)
- [x] Increase threads allocated per test

## Link to issue

Closes #131
Closes #475

## Type of change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Refactor (non-breaking change that updates existing functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to not work as expected)

The breaking change is the update to the `ReceivedReceiptPubsub`
notification.

## Test plan (required)

We've added tests to check:
  - [x] Put receipt and workflow info to DHT
  - [x] Got workflow info from DHT notifications
  - [x] Receipt and workflow info quorum success
  - [x] Receipt and workflow info quorum failure
  - [x] Provider sent workflow info to peer
  - [x] Peer received workflow info from a provider
  - [x] Peer received workflow info indirectly through a recursive request
(Test written, but ignored until we can isolate nodes)

We also have the starts for a test of got receipt from DHT that will be
used with future updates to receipt retrieval (in the
`test_libp2p_dht_records_serial` test, but commented out)

---------

Co-authored-by: Quinn Wilton <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bgins bgins

Labels
networking Features, functionality involving networking runtime Related to IPVM runtime specifics
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Add DHT and RequestResponse notifications ipvm-wg/homestar
3 participants
@zeeshanlakhani @bgins @mriise