fix: force mongodb connection to direct on localdev #7022
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
On local dev, several BE APIs will fail with
Server selection timed out after 30000
from MongoDB.Solution
Add
directConnection=true
to force all db operations to one single host designated in the connection URI.Background
The APIs failures were not completely deterministic, however some APIs fails much more constantly than other. APIs that never failed were POST requests, and specifically those that causes a DB
write
. While, those that failed consistently were APIs that were targetingread: secondary
(whileread: secondaryPreferred
didn't fail).Aside: Definitions
The above explains the phenomenon of why it:
staging
,uat
, ...)Now what?
We have multiple approaches here:
replicaSet
on our localdev following the bitnami guidedirectConnection
such that all operations are routed toprimary
replica, even for those that specifically targetssecondary
secondary
tosecondaryPreferred
For option 1, it incurs 2x more memory consumption as each additional DB docker instance adds
~200MB
, and we have to spin up the secondary + the arbiter (to co-ordinate leader election) node -- I've tried setting this up, but didn't manage to successfully find the correct configuration after a few hours.For option 2, all traffic will be channeled to the singular DB on docker. This is chosen as it is the simplest fix while mimicking our prior local dev behaviour.
For option 3, it will cause some reads that we intend to only be on secondary to be channeled to the primary node. i.e., generation of document stats on the landing page which is non-mission critical.