Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: force mongodb connection to direct on localdev #7022

Merged
merged 1 commit into from
Jan 15, 2024

Conversation

KenLSM
Copy link
Contributor

@KenLSM KenLSM commented Jan 11, 2024

Problem

On local dev, several BE APIs will fail with Server selection timed out after 30000 from MongoDB.

Solution

Add directConnection=true to force all db operations to one single host designated in the connection URI.

Background
The APIs failures were not completely deterministic, however some APIs fails much more constantly than other. APIs that never failed were POST requests, and specifically those that causes a DB write. While, those that failed consistently were APIs that were targeting read: secondary (while read: secondaryPreferred didn't fail).

Aside: Definitions

secondary
Operations read only from the secondary members of the set.

secondaryPreferred
Operations typically read data from secondary members of the replica set. If the replica set has only one single primary member and no other members, operations read data from the primary member.


The above explains the phenomenon of why it:

  1. didn't fail on deployed environments (staging, uat, ...)
  2. only certain APIs fail

Now what?
We have multiple approaches here:

  1. Setup replicaSet on our localdev following the bitnami guide
  2. Force directConnection such that all operations are routed to primary replica, even for those that specifically targets secondary
  3. Update all secondary to secondaryPreferred

For option 1, it incurs 2x more memory consumption as each additional DB docker instance adds ~200MB, and we have to spin up the secondary + the arbiter (to co-ordinate leader election) node -- I've tried setting this up, but didn't manage to successfully find the correct configuration after a few hours.

For option 2, all traffic will be channeled to the singular DB on docker. This is chosen as it is the simplest fix while mimicking our prior local dev behaviour.

For option 3, it will cause some reads that we intend to only be on secondary to be channeled to the primary node. i.e., generation of document stats on the landing page which is non-mission critical.

@KenLSM KenLSM requested review from justynoh and wanlingt January 11, 2024 17:46
@KenLSM KenLSM enabled auto-merge (squash) January 11, 2024 17:47
@KenLSM KenLSM requested a review from tshuli January 12, 2024 02:25
Copy link
Contributor

@wanlingt wanlingt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks for investigating this!

@KenLSM KenLSM merged commit 17124f2 into develop Jan 15, 2024
30 checks passed
@KenLSM KenLSM deleted the fix/mongodb-server-conn branch January 15, 2024 06:52
@KenLSM KenLSM mentioned this pull request Jan 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants