Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

redeploying a kubernetes service results in temporary outage #22

Closed
rhs opened this issue Apr 20, 2017 · 2 comments
Closed

redeploying a kubernetes service results in temporary outage #22

rhs opened this issue Apr 20, 2017 · 2 comments

Comments

@rhs
Copy link
Contributor

rhs commented Apr 20, 2017

when I redeploy a kubernetes service sitting behind a given ambassador route, there seem to be 5-10 seconds of downtime... this doesn't happen if I'm connecting directly to the external ip of the service
@kflynn
Copy link
Member

kflynn commented Apr 26, 2017 •
edited
Loading

The issue seems to be that, since Envoy only queries the SDS every 15 seconds by default, it can have an endpoint that's stale if the endpoint goes away right after the query (as can happen when the service is updated).

The simplest way to work around this for Ambassador-as-an-API-gateway is likely just to let Kubernetes handle load balancing for the moment. That costs a lot of features, but it's dirt simple.

Other things we could do:

  • drop the 15-second poll to 1 second
  • document that you must do rolling upgrades to avoid the outage (ugh)
  • teach Envoy how to accept updates over a websocket asynchronously ( ;) and also ugh)

@kflynn
Copy link
Member

kflynn commented May 3, 2017

Closing since we switched away from SDS for launch. Also see #47.

@kflynn kflynn closed this as completed May 3, 2017
@vaibhavrtk vaibhavrtk mentioned this issue Feb 13, 2019
Closed
kflynn pushed a commit that referenced this issue Apr 10, 2019
@richarddli
Merge pull request #22 from datawire/config
fb2a979 
delete obsolete docs
LukeShu added a commit that referenced this issue Sep 6, 2019
@LukeShu
Merge pull request #24 from datawire/lukeshu/merged-sidecar-pt1
754ff2a 
This is a bundle of "stable" work pulled out from working toward #22
that I figure I should try to land sooner rather than later.  None of
the commits are really thematically related to eachother.  See
individual commits for details.
rhs added a commit that referenced this issue Oct 9, 2019
@rhs
Merge pull request #22 from datawire/k8s-namespaces
5e8a52d 
Support Kubernetes namespaces
acookin added a commit that referenced this issue Aug 18, 2021
@acookin
envoy v2 fixes maybz (#22)
5cf7440 
* envoy v2 fixes maybz

* envoy v2 fixes maybz
acookin added a commit that referenced this issue Aug 24, 2021
@acookin
envoy v2 fixes maybz (#22)
1fb9b96 
* envoy v2 fixes maybz

* envoy v2 fixes maybz

Signed-off-by: Flynn <[email protected]>
acookin added a commit that referenced this issue Aug 24, 2021
@acookin @aidanhahn
@Alice-Lilith @kflynn
[v2.0.2-ea] Release Branch (#3701)
b23ceab 
* ensure pod log uploading when pytests fail

- incorrect if() clause in yaml caused failure to upload pod logs,
  making it hard to diagnose failures

Signed-off-by: Alice Nodelman <[email protected]>

* fixup links in readme (#3682)

Signed-off-by: Alix Cook <[email protected]>

* fix chart readme (#3690)

Signed-off-by: Alix Cook <[email protected]>

* Document similar implementation in DevPortal (#3691)

* Document similar implementation in DevPortal

Signed-off-by: alex <[email protected]>

* Added comment to the AmbassadorMapping CRD docs attribute

Signed-off-by: alex <[email protected]>

* make generate

Signed-off-by: Alix Cook <[email protected]>

Co-authored-by: Alix Cook <[email protected]>

* split out t_mappingtests into namespace sets (default and plain)

    - some of the mappingtests are designed to run in the default namespace,
      some in the plain namespace.  Split up these tests into different py
      files to make it easier to understand and execute
    - update pytest harness to start an httpbin for each namespace
      defined in tests under execution
    - update plain test gold file to use correct namespace

Signed-off-by: Alice Nodelman <[email protected]>

* add devel (#3696)

Signed-off-by: Alix Cook <[email protected]>

* Update for v2.0.2-ea (#3698)

Signed-off-by: Alix Cook <[email protected]>

* Update for v2.0.2-ea

Signed-off-by: Alix Cook <[email protected]>

* update envoy commit

Signed-off-by: Aidan Hahn <[email protected]>

* dont push manifests if IS_PRIVATE

Signed-off-by: Flynn <[email protected]>

* updates for embargoed

Signed-off-by: Flynn <[email protected]>

* envoy v3 should be default

Signed-off-by: Flynn <[email protected]>

* run make generate

Signed-off-by: Flynn <[email protected]>

* Update for v1.14.0.

* dont push manifests if IS_PRIVATE
* updates for embargoed
* Update for v1.14.0

Signed-off-by: Alix Cook <[email protected]>

* fixup changelog

Signed-off-by: Flynn <[email protected]>

* gotest fix (#16)

* gotest fix

* undo

Signed-off-by: Flynn <[email protected]>

* remove tests that were commented out on master

Signed-off-by: Aidan Hahn <[email protected]>
Signed-off-by: Flynn <[email protected]>

* enable deprecated apis

Signed-off-by: Flynn <[email protected]>

* pytest corrections (#18)

Signed-off-by: Aidan Hahn <[email protected]>
Signed-off-by: Flynn <[email protected]>

* gzip fix (#19)

* gzip fix

* update ambex

* update ambex

* update circle

Signed-off-by: Flynn <[email protected]>

* pytest envoy v2 test fixes (#21)

Signed-off-by: Flynn <[email protected]>

* envoy v2 fixes maybz (#22)

* envoy v2 fixes maybz

* envoy v2 fixes maybz

Signed-off-by: Flynn <[email protected]>

* also push rc for emissary privaet

Signed-off-by: Flynn <[email protected]>

* allow chunked length (#24)

* expose chunked length

Signed-off-by: AliceProxy <[email protected]>

improve test coverage

Signed-off-by: AliceProxy <[email protected]>

Remove v2listener and v2Test

Signed-off-by: AliceProxy <[email protected]>

add setting to protos

Signed-off-by: AliceProxy <[email protected]>

make generate

Signed-off-by: AliceProxy <[email protected]>

* comment out test

* test fixes

* UNDO THIS

* mkay

* comment out test

Co-authored-by: AliceProxy <[email protected]>
Signed-off-by: Flynn <[email protected]>

* update pf gomod

Signed-off-by: Flynn <[email protected]>

* extauth needs "ignore_case" for auth and request headers

Signed-off-by: Aidan Hahn <[email protected]>
Signed-off-by: Flynn <[email protected]>

* yank deprecated config option

Signed-off-by: Aidan Hahn <[email protected]>
Signed-off-by: Flynn <[email protected]>

* update t_chunked_length test

Signed-off-by: Aidan Hahn <[email protected]>
Signed-off-by: Flynn <[email protected]>

* Log ambex snapshots

Signed-off-by: Flynn <[email protected]>

* ambex logging cleanup

Signed-off-by: Flynn <[email protected]>

* Update goldfiles for V3 #1 (from d15d3b4)

Signed-off-by: Flynn <[email protected]>

* fix fake_mapping_cors_test

Signed-off-by: Flynn <[email protected]>

* fix chunked length

Signed-off-by: Flynn <[email protected]>

* fix envoy_api_version

Signed-off-by: Flynn <[email protected]>

* make go-mod-tidy

Signed-off-by: Flynn <[email protected]>

* Update t_chunked_length for V2

Signed-off-by: Flynn <[email protected]>

* Update gold files

Signed-off-by: Flynn <[email protected]>

* CHANGELOG

Signed-off-by: Flynn <[email protected]>

* make generate

Signed-off-by: Flynn <[email protected]>

* Temporary fix for repo name.

Signed-off-by: Flynn <[email protected]>

* remove fake mapping test, make generate

Signed-off-by: Flynn <[email protected]>

* bump envoy in 2.0.2

* fix manifest stuff

* fix is private thing again

Co-authored-by: Alice Nodelman <[email protected]>
Co-authored-by: Alex Gervais <[email protected]>
Co-authored-by: Aidan Hahn <[email protected]>
Co-authored-by: Flynn <[email protected]>
Co-authored-by: Aidan Hahn <[email protected]>
Co-authored-by: AliceProxy <[email protected]>
Co-authored-by: Flynn <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rhs @kflynn