Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mnesia crashes and recovers on secondary nodes in clustered configuration #2

Open
beardedeagle opened this issue Jul 30, 2018 · 1 comment
Open

Mnesia crashes and recovers on secondary nodes in clustered configuration #2

beardedeagle opened this issue Jul 30, 2018 · 1 comment
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@beardedeagle
Copy link
Owner

beardedeagle commented Jul 30, 2018 •
edited
Loading

When mnesiac is configured to run in a clustered env, mnesia crashes on any node brought up after the first node. mnesia does recover and this lib is STILL usable in production (I use it in production), but this should be solved. It LOOKS like a issue with how the schema is being initialized. Trace output below.

config:

config :libcluster,
  topologies: [
    vanguard: [
      strategy: Cluster.Strategy.Epmd,
      config: [
        hosts: [
          :"[email protected]",
          :"[email protected]"
        ]
      ]
    ]
  ]

config :mnesia, dir: to_charlist(Path.join(File.cwd!(), System.get_env("DB_DIR") || "priv/data"))

config :mnesiac,
  stores: [
    Vanguard.Db.ProcStore,
    Vanguard.Db.OwnerStore,
    Vanguard.Db.HandlerStore,
    Vanguard.Db.StatusStore,
    Vanguard.Db.CheckStore
  ],
  schema_type: :disc_copies,
  table_load_timeout: 600_000

test01:

~> PORT=4001 DB_DIR=priv/test01 iex --erl '-mnesia debug trace verbose' --name [email protected] --cookie test -S mix phx.server
Erlang/OTP 21 [erts-10.0.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Mnesia('[email protected]'): mnesia_monitor starting: <0.267.0>
Mnesia('[email protected]'): Version: "4.15.4"
Mnesia('[email protected]'): Env access_module: mnesia
Mnesia('[email protected]'): Env allow_index_on_key: false
Mnesia('[email protected]'): Env auto_repair: true
Mnesia('[email protected]'): Env backup_module: mnesia_backup
Mnesia('[email protected]'): Env debug: trace
Mnesia('[email protected]'): Env dir: "/Users/rlthompson/GitHub/vanguard/priv/test01"
Mnesia('[email protected]'): Env dump_disc_copies_at_startup: true
Mnesia('[email protected]'): Env dump_log_load_regulation: false
Mnesia('[email protected]'): Env dump_log_time_threshold: 180000
Mnesia('[email protected]'): Env dump_log_update_in_place: true
Mnesia('[email protected]'): Env dump_log_write_threshold: 1000
Mnesia('[email protected]'): Env event_module: mnesia_event
Mnesia('[email protected]'): Env extra_db_nodes: []
Mnesia('[email protected]'): Env ignore_fallback_at_startup: false
Mnesia('[email protected]'): Env fallback_error_function: {mnesia,lkill}
Mnesia('[email protected]'): Env fold_chunk_size: 100
Mnesia('[email protected]'): Env max_wait_for_decision: infinity
Mnesia('[email protected]'): Env schema_location: opt_disc
Mnesia('[email protected]'): Env core_dir: false
Mnesia('[email protected]'): Env pid_sort_order: false
Mnesia('[email protected]'): Env no_table_loaders: 2
Mnesia('[email protected]'): Env dc_dump_limit: 4
Mnesia('[email protected]'): Env send_compressed: 0
Mnesia('[email protected]'): Env schema: []
Mnesia('[email protected]'): Mnesia debug level set to trace
Mnesia('[email protected]'): mnesia_subscr starting: <0.268.0>
Mnesia('[email protected]'): mnesia_locker starting: <0.269.0>
Mnesia('[email protected]'): mnesia_recover starting: <0.270.0>
Mnesia('[email protected]'): mnesia_tm starting: <0.271.0>
Mnesia('[email protected]'): Schema initiated from: default
Mnesia('[email protected]'): mnesia_controller starting: <0.273.0>
Mnesia('[email protected]'): mnesia_late_loader starting: <0.274.0>
Mnesia('[email protected]'): Intend to load tables: []
[info] [swarm on [email protected]] [tracker:init] started
[warn] [libcluster:vanguard] unable to connect to :"[email protected]"
[info] [mnesiac:[email protected]] mnesiac starting...
Mnesia('[email protected]'): Mnesia already started, 0 seconds
Mnesia('[email protected]'): Transaction log dump initiated by schema_prepare: already_dumped
Mnesia('[email protected]'): Transaction log dump initiated by schema_update: {needs_dump,1}
Mnesia('[email protected]'): write performed by {tid,3,<0.318.0>} on record:
        {schema,schema,
                [{name,schema},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,schema},
                 {attributes,[table,cstruct]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224666330000,-576460752303423483,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929224,771269}}}}]}
Mnesia('[email protected]'): write performed by {tid,4,<0.334.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['[email protected]']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,4,<0.334.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['[email protected]']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Getting table 'Elixir.ProcStore' (ram_copies) from disc: {dumper,
                                                                                      create_table}
Mnesia('[email protected]'): Creating index for 'Elixir.ProcStore' 3 ordered
Mnesia('[email protected]'): write performed by {tid,5,<0.338.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,5,<0.338.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Getting table 'Elixir.OwnerStore' (disc_copies) from disc: {dumper,
                                                                                        create_table}
Mnesia('[email protected]'): write performed by {tid,6,<0.344.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,6,<0.344.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Getting table 'Elixir.HandlerStore' (disc_copies) from disc: {dumper,
                                                                                          create_table}
Mnesia('[email protected]'): Creating index for 'Elixir.HandlerStore' 3 ordered
Mnesia('[email protected]'): Creating index for 'Elixir.HandlerStore' 7 ordered
Mnesia('[email protected]'): write performed by {tid,7,<0.350.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,7,<0.350.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Getting table 'Elixir.StatusStore' (disc_copies) from disc: {dumper,
                                                                                         create_table}
Mnesia('[email protected]'): Creating index for 'Elixir.StatusStore' 3 ordered
Mnesia('[email protected]'): Creating index for 'Elixir.StatusStore' 4 ordered
Mnesia('[email protected]'): write performed by {tid,8,<0.356.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,8,<0.356.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): Getting table 'Elixir.CheckStore' (disc_copies) from disc: {dumper,
                                                                                        create_table}
Mnesia('[email protected]'): Creating index for 'Elixir.CheckStore' 3 ordered
Mnesia('[email protected]'): Creating index for 'Elixir.CheckStore' 4 ordered
Mnesia('[email protected]'): Creating index for 'Elixir.CheckStore' 10 ordered
[info] [mnesiac:[email protected]] mnesiac started
Interactive Elixir (1.7.1) - press Ctrl+C to exit (type h() ENTER for help)

test02:

 ~> PORT=4000 DB_DIR=priv/test02 iex --erl '-mnesia debug trace verbose' --name [email protected] --cookie test -S mix phx.server
Erlang/OTP 21 [erts-10.0.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Mnesia('[email protected]'): mnesia_monitor starting: <0.267.0>
Mnesia('[email protected]'): Version: "4.15.4"
Mnesia('[email protected]'): Env access_module: mnesia
Mnesia('[email protected]'): Env allow_index_on_key: false
Mnesia('[email protected]'): Env auto_repair: true
Mnesia('[email protected]'): Env backup_module: mnesia_backup
Mnesia('[email protected]'): Env debug: trace
Mnesia('[email protected]'): Env dir: "/Users/rlthompson/GitHub/vanguard/priv/test02"
Mnesia('[email protected]'): Env dump_disc_copies_at_startup: true
Mnesia('[email protected]'): Env dump_log_load_regulation: false
Mnesia('[email protected]'): Env dump_log_time_threshold: 180000
Mnesia('[email protected]'): Env dump_log_update_in_place: true
Mnesia('[email protected]'): Env dump_log_write_threshold: 1000
Mnesia('[email protected]'): Env event_module: mnesia_event
Mnesia('[email protected]'): Env extra_db_nodes: []
Mnesia('[email protected]'): Env ignore_fallback_at_startup: false
Mnesia('[email protected]'): Env fallback_error_function: {mnesia,lkill}
Mnesia('[email protected]'): Env fold_chunk_size: 100
Mnesia('[email protected]'): Env max_wait_for_decision: infinity
Mnesia('[email protected]'): Env schema_location: opt_disc
Mnesia('[email protected]'): Env core_dir: false
Mnesia('[email protected]'): Env pid_sort_order: false
Mnesia('[email protected]'): Env no_table_loaders: 2
Mnesia('[email protected]'): Env dc_dump_limit: 4
Mnesia('[email protected]'): Env send_compressed: 0
Mnesia('[email protected]'): Env schema: []
Mnesia('[email protected]'): Mnesia debug level set to trace
Mnesia('[email protected]'): mnesia_subscr starting: <0.268.0>
Mnesia('[email protected]'): mnesia_locker starting: <0.269.0>
Mnesia('[email protected]'): mnesia_recover starting: <0.270.0>
Mnesia('[email protected]'): mnesia_tm starting: <0.271.0>
Mnesia('[email protected]'): Schema initiated from: default
Mnesia('[email protected]'): mnesia_controller starting: <0.273.0>
Mnesia('[email protected]'): mnesia_late_loader starting: <0.274.0>
Mnesia('[email protected]'): Intend to load tables: []
[info] [swarm on [email protected]] [tracker:init] started
[info] [swarm on [email protected]] [tracker:ensure_swarm_started_on_remote_node] nodeup [email protected]
[info] [swarm on [email protected]] [tracker:cluster_wait] joining cluster..
[info] [swarm on [email protected]] [tracker:cluster_wait] syncing with [email protected]
[info] [swarm on [email protected]] [tracker:awaiting_sync_ack] received sync acknowledgement from [email protected], syncing with remote registry
[info] [swarm on [email protected]] [tracker:awaiting_sync_ack] local synchronization with [email protected] complete!
[info] [swarm on [email protected]] [tracker:resolve_pending_sync_requests] pending sync requests cleared
[info] [libcluster:vanguard] connected to :"[email protected]"
[info] [mnesiac:[email protected]] mnesiac starting...
Mnesia('[email protected]'): mnesia_controller terminated: shutdown
Mnesia('[email protected]'): mnesia_tm terminated: shutdown
Mnesia('[email protected]'): mnesia_recover terminated: shutdown
Mnesia('[email protected]'): mnesia_locker terminated: shutdown
Mnesia('[email protected]'): mnesia_subscr terminated: shutdown
Mnesia('[email protected]'): mnesia_monitor terminated: shutdown
[info] Application mnesia exited: :stopped
Mnesia('[email protected]'): mnesia_monitor starting: <0.329.0>
Mnesia('[email protected]'): Version: "4.15.4"
Mnesia('[email protected]'): Env access_module: mnesia
Mnesia('[email protected]'): Env allow_index_on_key: false
Mnesia('[email protected]'): Env auto_repair: true
Mnesia('[email protected]'): Env backup_module: mnesia_backup
Mnesia('[email protected]'): Env debug: trace
Mnesia('[email protected]'): Env dir: "/Users/rlthompson/GitHub/vanguard/priv/test02"
Mnesia('[email protected]'): Env dump_disc_copies_at_startup: true
Mnesia('[email protected]'): Env dump_log_load_regulation: false
Mnesia('[email protected]'): Env dump_log_time_threshold: 180000
Mnesia('[email protected]'): Env dump_log_update_in_place: true
Mnesia('[email protected]'): Env dump_log_write_threshold: 1000
Mnesia('[email protected]'): Env event_module: mnesia_event
Mnesia('[email protected]'): Env extra_db_nodes: []
Mnesia('[email protected]'): Env ignore_fallback_at_startup: false
Mnesia('[email protected]'): Env fallback_error_function: {mnesia,lkill}
Mnesia('[email protected]'): Env fold_chunk_size: 100
Mnesia('[email protected]'): Env max_wait_for_decision: infinity
Mnesia('[email protected]'): Env schema_location: opt_disc
Mnesia('[email protected]'): Env core_dir: false
Mnesia('[email protected]'): Env pid_sort_order: false
Mnesia('[email protected]'): Env no_table_loaders: 2
Mnesia('[email protected]'): Env dc_dump_limit: 4
Mnesia('[email protected]'): Env send_compressed: 0
Mnesia('[email protected]'): Env schema: []
Mnesia('[email protected]'): Mnesia debug level set to trace
Mnesia('[email protected]'): mnesia_subscr starting: <0.330.0>
Mnesia('[email protected]'): mnesia_locker starting: <0.331.0>
Mnesia('[email protected]'): mnesia_recover starting: <0.332.0>
Mnesia('[email protected]'): mnesia_tm starting: <0.333.0>
Mnesia('[email protected]'): Schema initiated from: default
Mnesia('[email protected]'): mnesia_controller starting: <0.335.0>
Mnesia('[email protected]'): mnesia_late_loader starting: <0.336.0>
Mnesia('[email protected]'): Intend to load tables: []
Mnesia('[email protected]'): Mnesia started, 0 seconds
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): write performed by {tid,3,<0.340.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['[email protected]']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          '[email protected]'}},
                 {version,{{2,0},[]}}]}
Mnesia('[email protected]'): write performed by {tid,3,<0.340.0>} on record:
        {schema,schema,
                [{name,schema},
                 {type,set},
                 {ram_copies,['[email protected]']},
                 {disc_copies,['[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,schema},
                 {attributes,[table,cstruct]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224666330000,-576460752303423483,1},
                          '[email protected]'}},
                 {version,{{4,0},{'[email protected]',{1532,929235,924960}}}}]}
Mnesia('[email protected]'): Table 'Elixir.ProcStore' is loaded on '[email protected]'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('[email protected]'): Table 'Elixir.OwnerStore' is loaded on '[email protected]'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('[email protected]'): Table 'Elixir.HandlerStore' is loaded on '[email protected]'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('[email protected]'): Table 'Elixir.StatusStore' is loaded on '[email protected]'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('[email protected]'): Table 'Elixir.CheckStore' is loaded on '[email protected]'. s=unknown, r=nowhere, lc=false, f=false, m=true
Mnesia('[email protected]'): Logging mnesia_up '[email protected]'
Mnesia('[email protected]'): Logging mnesia_up '[email protected]'
Mnesia('[email protected]'): Transaction log dump initiated by schema_prepare: already_dumped
Mnesia('[email protected]'): Transaction log dump initiated by schema_update: {needs_dump,1}
Mnesia('[email protected]'): write performed by {tid,5,<0.342.0>} on record:
        {schema,schema,
                [{name,schema},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]','[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,schema},
                 {attributes,[table,cstruct]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224666330000,-576460752303423483,1},
                          '[email protected]'}},
                 {version,{{5,0},{'[email protected]',{1532,929235,977055}}}}]}
Mnesia('[email protected]'): write performed by {tid,6,<0.358.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['[email protected]','[email protected]']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,93952}}}}]}
Mnesia('[email protected]'): Getting table 'Elixir.ProcStore' (ram_copies) from node '[email protected]': {dumper,
                                                                                                         {add_table_copy,
                                                                                                          {tid,
                                                                                                           6,
                                                                                                           <0.358.0>}}}
Mnesia('[email protected]'): Creating index for 'Elixir.ProcStore' 3 ordered
Mnesia('[email protected]'): Table 'Elixir.ProcStore' copied from '[email protected]' to '[email protected]'
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,6,<0.358.0>} on record:
        {schema,'Elixir.ProcStore',
                [{name,'Elixir.ProcStore'},
                 {type,set},
                 {ram_copies,['[email protected]','[email protected]']},
                 {disc_copies,[]},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.ProcStore'},
                 {attributes,[name,pid]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224852085000,-576460752303421727,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,93952}}}}]}
Mnesia('[email protected]'): write performed by {tid,7,<0.363.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]','[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,135578}}}}]}
Mnesia('[email protected]'): Getting table 'Elixir.OwnerStore' (disc_copies) from node '[email protected]': {dumper,
                                                                                                           {add_table_copy,
                                                                                                            {tid,
                                                                                                             7,
                                                                                                             <0.363.0>}}}
Mnesia('[email protected]'): Table 'Elixir.OwnerStore' copied from '[email protected]' to '[email protected]'
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,7,<0.363.0>} on record:
        {schema,'Elixir.OwnerStore',
                [{name,'Elixir.OwnerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]','[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.OwnerStore'},
                 {attributes,[owner_id,name,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224869816000,-576460752303421631,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,135578}}}}]}
Mnesia('[email protected]'): write performed by {tid,8,<0.370.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]','[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,188492}}}}]}
Mnesia('[email protected]'): Getting table 'Elixir.HandlerStore' (disc_copies) from node '[email protected]': {dumper,
                                                                                                             {add_table_copy,
                                                                                                              {tid,
                                                                                                               8,
                                                                                                               <0.370.0>}}}
Mnesia('[email protected]'): Creating index for 'Elixir.HandlerStore' 3 ordered
Mnesia('[email protected]'): Creating index for 'Elixir.HandlerStore' 7 ordered
Mnesia('[email protected]'): Table 'Elixir.HandlerStore' copied from '[email protected]' to '[email protected]'
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,8,<0.370.0>} on record:
        {schema,'Elixir.HandlerStore',
                [{name,'Elixir.HandlerStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]','[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{7,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.HandlerStore'},
                 {attributes,[handler_id,owner_id,handler_data,timeout,
                              handler_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224884256000,-576460752303421535,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,188492}}}}]}
Mnesia('[email protected]'): write performed by {tid,9,<0.377.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]','[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,239534}}}}]}
Mnesia('[email protected]'): Getting table 'Elixir.StatusStore' (disc_copies) from node '[email protected]': {dumper,
                                                                                                            {add_table_copy,
                                                                                                             {tid,
                                                                                                              9,
                                                                                                              <0.377.0>}}}
Mnesia('[email protected]'): Creating index for 'Elixir.StatusStore' 3 ordered
Mnesia('[email protected]'): Creating index for 'Elixir.StatusStore' 4 ordered
Mnesia('[email protected]'): Table 'Elixir.StatusStore' copied from '[email protected]' to '[email protected]'
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,9,<0.377.0>} on record:
        {schema,'Elixir.StatusStore',
                [{name,'Elixir.StatusStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]','[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.StatusStore'},
                 {attributes,[status_id,check_id,owner_id,status,response,
                              retention,last_ran]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224900376000,-576460752303423355,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,239534}}}}]}
Mnesia('[email protected]'): write performed by {tid,10,<0.384.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]','[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,292209}}}}]}
Mnesia('[email protected]'): Getting table 'Elixir.CheckStore' (disc_copies) from node '[email protected]': {dumper,
                                                                                                           {add_table_copy,
                                                                                                            {tid,
                                                                                                             10,
                                                                                                             <0.384.0>}}}
Mnesia('[email protected]'): Creating index for 'Elixir.CheckStore' 3 ordered
Mnesia('[email protected]'): Creating index for 'Elixir.CheckStore' 4 ordered
Mnesia('[email protected]'): Creating index for 'Elixir.CheckStore' 10 ordered
Mnesia('[email protected]'): Table 'Elixir.CheckStore' copied from '[email protected]' to '[email protected]'
Mnesia('[email protected]'): Transaction log dump skipped (optional): schema_prepare
Mnesia('[email protected]'): write performed by {tid,10,<0.384.0>} on record:
        {schema,'Elixir.CheckStore',
                [{name,'Elixir.CheckStore'},
                 {type,set},
                 {ram_copies,[]},
                 {disc_copies,['[email protected]','[email protected]']},
                 {disc_only_copies,[]},
                 {load_order,0},
                 {access_mode,read_write},
                 {majority,false},
                 {index,[{3,ordered},{4,ordered},{10,ordered}]},
                 {snmp,[]},
                 {local_content,false},
                 {record_name,'Elixir.CheckStore'},
                 {attributes,[check_id,owner_id,handler_id,check_data,dc,
                              interval,timeout,check_state,host,created_at]},
                 {user_properties,[]},
                 {frag_properties,[]},
                 {storage_properties,[]},
                 {cookie,{{1532929224917072000,-576460752303421407,1},
                          '[email protected]'}},
                 {version,{{3,0},{'[email protected]',{1532,929236,292209}}}}]}
[info] [mnesiac:[email protected]] mnesiac started
Interactive Elixir (1.7.1) - press Ctrl+C to exit (type h() ENTER for help)
@beardedeagle beardedeagle added bug Something isn't working help wanted Extra attention is needed labels Jul 30, 2018
@beardedeagle beardedeagle changed the title Mnesia crashes on secondary nodes in clustered configuration Mnesia crashes and recovers on secondary nodes in clustered configuration Jul 31, 2018
@beardedeagle beardedeagle mentioned this issue Aug 8, 2019
Closed
5 tasks
@beardedeagle
Copy link
Owner Author

I believe I have a solution to this, but will need to test with the #53 release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@beardedeagle