Static sharding & Postgres

[SionoiS]

Hi,

@Ivansete-status and me are in need of a fleet for static sharding and the new DB.

Configuring nwaku for static shard is just a matter of changing the pubsub topics but for the DB I'm not sure, I'll let Ivan explain.

We seek guidance on this, O great infra master :P

I was thinking of copying waku2.test then modify it for our needs? What else do we need?

[Ivansete-status]

Thanks for submitting that @SionoiS !
I think it is a good approach to have the wakuv2.test as a reference.
Re the Postgres feature, we need to have a separate node that will host the Postgres itself. Then, in this "staging" fleet we will have four nodes (3 nwaku + 1 postgres.)

cc - @jm-clius

[SionoiS]

Thanks for submitting that @SionoiS ! I think it is a good approach to have the wakuv2.test as a reference. Re the Postgres feature, we need to have a separate node that will host the Postgres itself. Then, in this "staging" fleet we will have four nodes (3 nwaku + 1 postgres.)

cc - @jm-clius

Maybe more then 4 nodes. Depends if testing only 2 shards is sufficient (probably not).

[jm-clius]

Thanks for submitting that @SionoiS ! I think it is a good approach to have the wakuv2.test as a reference. Re the Postgres feature, we need to have a separate node that will host the Postgres itself. Then, in this "staging" fleet we will have four nodes (3 nwaku + 1 postgres.)
cc - @jm-clius

Maybe more then 4 nodes. Depends if testing only 2 shards is sufficient (probably not)

Number of nodes doesn't have to equal the number of shards. In fact all nodes should support the same shards and likely ~10 in total. I think we can come up with more precise requirements here. Let's discuss in nwaku-PM call and be specific here (i.e. which shards, how to set up the postgresql settings and what services should be enabled here).

[SionoiS]

Static Sharding

Cluster 16 is reserved for status main net according to RFC52 shall we use cluster 18 for test net?
For shards indices, 896+ for controls, 768+ for 1on1, 128+ for small communities, 16+ for large communities and maybe more then 1 shard per category? Let say 3.

Topics;

/waku/2/rs/18/16
/waku/2/rs/18/17
/waku/2/rs/18/18
/waku/2/rs/18/128
/waku/2/rs/18/129
/waku/2/rs/18/130
/waku/2/rs/18/768
/waku/2/rs/18/769
/waku/2/rs/18/770
/waku/2/rs/18/896
/waku/2/rs/18/897
/waku/2/rs/18/896

Protocols

At least one Store node to test new DB, LightPush, Filter, Peer Exchange, RLN and Relay

Otherwise will use the default values from wakuv2-test.yml

[Ivansete-status]

@jakubgs - Re Postgres we need to pass a param analog to the next one, to each nwaku node:

store-message-db-url = "postgres://postgres:test123@localhost:5432/postgres"

Obviously, the user (test1234) and the password (5432) should be changed and cannot be written in GitHub. We need to keep them secret :)

On the other hand, the localhost should be replaced by the IP of the instance that hosts the Postgres database.

We should be able to interact with the Postgres database from within its own host through the psql command.

As we've mentioned, we'll only have one Postgres instance for now. I suggest having it in the AMS data center.

Thanks in advance!

[jakubgs]

You know, just because you open an issue in a repo doesn't mean I will see it. You actually have to ping me, like the comment above did, otherwise I will never see it.

[jakubgs]

On the other hand, the localhost should be replaced by the IP of the instance that hosts the Postgres database.

Lol, really, I wouldn't have guessed :D. Hahahaha

[jakubgs]

We already have a Postgres setup for status-go nodes:
https://github.com/status-im/infra-role-status-go/blob/3e52ec695ac4c807ca01b28f908a943b8a0b545b/templates/docker-compose.yml.j2#L55-L80

I assume in this case it would also be optional and enabled by a flag.

[jakubgs]

Also, can someone explain to em what this PR was for and why it was closed? Is it related?

• wakuv2-shards: init new fleet for static sharding #75

[SionoiS]

Also, can someone explain to em what this PR was for and why it was closed? Is it related?

• New fleet Initialization #75

At first I thought I was going to setup the fleet but the process seams to be that we ask and specify what we need to you @jakubgs

Is that how it works usually?

[jakubgs]

I welcome PRs, since they save me time in the long run. That PR looked pretty sensible, maybe a few things to fix.

[SionoiS]

I welcome PRs, since they save me time in the long run. That PR looked pretty sensible, maybe a few things to fix.

I'll make the changes then we can continue.

[SionoiS]

Sorry for the mess @jakubgs nobody noticed the problem with the config.

[jakubgs]

Shit happens when you move fast.

[Ivansete-status]

As per the Postgres configuration, we need the next topology in the wakuv2.shards fleet:

One single Postgres DB instance running in Amsterdam DC.
The three nodes should be connected to the ams db instance.

A parameter similar to the next one should be passed to the nwaku nodes.

--store-message-db-url='postgres://postgres:test123@localhost:5432/postgres'

[jakubgs]

@Ivansete-status is this request separate from:

• Static sharding fleets for Status Communities infra-status#2

Or is it just an early version of that issue and now outdated?

I don't think this repo can hold such a drastically different setup.
Unless we actually want to apply it to all other stages in this fleet.

[Ivansete-status]

Hey @jakubgs !
This is a separate request. In this case, the purpose of having such a layout is to ensure we test the next use case:
"Multiple writers writing to a single postgresql DB in a real network environment".

This fleet is for testing purposes. If that setup doesn't fit well with the rest, we can seek another one.

( cc @jm-clius )

[jakubgs]

Isn't this just a test stage for what you want to do in:

• Static sharding fleets for Status Communities infra-status#2

It seems like we just got confused by not knowing your future goal and put this fleet here for lack of a better place.
But really, it should belong, in the new repo where the fleet requested by status-im/infra-status#2 will be created.

[Ivansete-status]

The main purpose is to have a testing environment where we can validate and measure the use case I mentioned in my previous comment. The goal is to test it before that use case becomes real for the Status apps.

[Ivansete-status]

@jakubgs @yakimant - from the next repo, a nwaku node instance + Postgres can be started with docker compose up.

https://github.com/waku-org/nwaku-compose

Checking the Grafana dashboard from localhost:3000 (admin - admin)

[jm-clius]

Afaik the wakuv2.shards fleet is being used to dogfood some aspects related to Status static sharding, but dogfooding efforts will focus on the new status.shards fleet as soon as this is ready. I therefore propose to deprioritise the postgresql installation for wakuv2.shards and instead focus on status.shards.

[yakimant]

@jm-clius, @Ivansete-status, I just decomissioned wakuv2.shards: #88

I guess this Issues should be closed now in favour of:

• waku.test (Autosharding Test Fleet #82)
• shards.test (Static sharding fleets for Status Communities infra-status#2)

fleets