Fix event not registered on shutdown #4400

arcusfelis · 2024-11-19T22:02:18Z

This PR addresses "MIM-2316 event_not_registered for c2s_xmpp_element_size_out when stopping the node error".

When we call mongooseimctl stop (and there are some websocket connections), we see this error in logs:
{error,#{what => event_not_registered,labels => #{},event_name => c2s_xmpp_element_size_out} .

Code in question is this one in mongoose_listener:

stop() ->
    Listeners = mongoose_config:get_opt(listen),
    lists:foreach(fun stop_listener/1, Listeners),
    mongoose_instrument:tear_down(instrumentation(Listeners)).

So, basically it looks like there are some c2s processes alive on the node after we call tear_down. So, metric c2s_xmpp_element_size_out was there, but not here anymore.

Proposed changes include:

Try to reproduce the bug, most likely you will need websocket connections when you stop the server (or stop the listener?). - Done. Just having a websocket connection is enough.
Try to fix the bug by ensuring that there are no c2s processes after the listener is stopped. Make sure no processes could spawn up. - there was already logic for it, it was using ranch and c2s supervisor. Moving stopping based on the supervisor above before stopping the listeners fixes the issue. But there still a chance that new connections would be established. So, we 1) - disable acceptors. 2) - stop c2s processes. 3) - stop listeners completely. instead.

Global distribution:

mysql_redis preset was using the same redis for both reg and mim clusters. This means that session from reg was visible in mim cluster over ejabberd_sm_redis. So, eve pid was visible in ejabberd_sm on the mim cluster too. We use different redis database numbers for different clusters now.
added a test with a websocket client to global distribution too.
added debug logging into make_error_reply.

Shutdown:

mongoose_c2s_sup only contains websocket clients sessions (maybe bosh too).
ranch contains regular XMPP connections.

Shutdown logic now is:

suspend all ranch acceptors.
drain connections from both ranch and c2s supervisor - could take some time.
shotdown listeners and remove instrimentation metrics.

Use ranch:wait_for_connections(Ref, '==', 0), because it is more optimized, than just pulling a list of pids and checking its length in the loop.

Ideas for future:

can we use websocket process as a c2s process? so we have one process in the system per user. Could be nice savings in Websocket oriented setups.
Move this code around a bit, so we can have mongooseimctl suspend_listener and mongooseimctl drain commands, which could be useful for implementing graceful shutdowns in k8s or ansible scripts.

And websocket clients

Disallow new connections by calling suspend_listeners Call broadcast_c2s_shutdown_sup Call mongoose_listener:stop This should unify behaviour of c2s and websockets when shutting down the node

mongoose-im · 2024-11-19T22:13:04Z

elasticsearch_and_cassandra_27 / elasticsearch_and_cassandra_mnesia / 8d032a3
Reports root/ big
OK: 471 / Failed: 2 / User-skipped: 49 / Auto-skipped: 0

gdpr_SUITE:retrieve_personal_data:retrieve_personal_data_mam:retrieve_personal_data_mam_cassandra:retrieve_mam_muc

{error,{#{what => invalid_response_code,expected_type => ok,
      response_code => {exit_status,1}},
    [{graphql_helper,assert_response_code,2,
             [{file,"/home/circleci/project/big_tests/tests/graphql_helper.erl"},
              {line,258}]},
     {graphql_helper,get_ok_value,2,
             [{file,"/home/circleci/project/big_tests/tests/graphql_helper.erl"},
              {line,241}]},
     {gdpr_SUITE,request_and_unzip_personal_data,3,
           [{file,"/home/circleci/project/big_tests/tests/gdpr_SUITE.erl"},
            {line,1692}]},
     {gdpr_SUITE,'-retrieve_mam_muc/1-fun-2-',4,
           [{file,"/home/circleci/project/big_tests/tests/gdpr_SUITE.erl"},
            {line,652}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1794}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1303}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1235}]}]}}

-              -spec broadcast_c2s_shutdown_listeners() -> ok.
-              broadcast_c2s_shutdown_listeners() ->
+              -spec suspend_listeners(typed_listeners()) -> ok.
+              suspend_listeners(TypedListeners) ->

+                  TypedListeners = get_typed_listeners(),
+                  suspend_listeners(TypedListeners),
+                  StoppedCount = broadcast_c2s_shutdown_sup(),
+                  StoppedCount2 = broadcast_c2s_shutdown_to_regular_c2s_connections(TypedListeners),

Fix event not registered on shutdown #4400

Fix event not registered on shutdown #4400

Conversation

arcusfelis commented Nov 19, 2024 • edited Loading

mongoose-im commented Nov 19, 2024 • edited Loading

codecov bot commented Nov 19, 2024 • edited Loading

Codecov Report

mongoose-im commented Nov 28, 2024 • edited Loading

mongoose-im commented Nov 29, 2024 • edited Loading

chrzaszcz left a comment

Choose a reason for hiding this comment

chrzaszcz Nov 29, 2024

Choose a reason for hiding this comment

chrzaszcz Nov 29, 2024

Choose a reason for hiding this comment

mongoose-im commented Dec 2, 2024 • edited Loading

mongoose-im commented Dec 2, 2024 • edited Loading

chrzaszcz left a comment

Choose a reason for hiding this comment

arcusfelis commented Nov 19, 2024 •

edited

Loading

mongoose-im commented Nov 19, 2024 •

edited

Loading

codecov bot commented Nov 19, 2024 •

edited

Loading

mongoose-im commented Nov 28, 2024 •

edited

Loading

mongoose-im commented Nov 29, 2024 •

edited

Loading

mongoose-im commented Dec 2, 2024 •

edited

Loading

mongoose-im commented Dec 2, 2024 •

edited

Loading