Comments (12)
Commands worked for first table, but failed when i tried it with Tab = emqx_ds_session.
Since based on the discussion above, it appears this is some kind of inconsistent database state, I've exported the data, blow away the cluster PVCs after stopping it, and restarted it then imported the data export.
Followed by checking if the schemas are still listed from the diagnostic command above. This appears to have resolved the issue.
I thank you for your time in responding to this.
from emqx.
Any suggestions on how to proceed would be appreciated, since I have absolutely no idea how it got into this state.
from emqx.
could you share the commands you use?
do you run the force leave commands on one of the alive nodes?
running_nodes =>
['emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local'],
It should print something, shouldn't be silent.
from emqx.
If you are not running production,
Assume the node emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local
is the stale node that you want to remove from the cluster.
Have you tried something like following?
emqx_ctl cluster force-leave 'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local'
from emqx.
That's exactly it, when I run the force-leave command, it does nothing with no output.
emqx@emqx-core-f7b96944f-0:/opt/emqx$ emqx_ctl cluster force-leave 'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local'
emqx@emqx-core-f7b96944f-0:/opt/emqx$
And yes, I run the force-leave commands on one of the alive nodes as the stopped nodes are from instances that no longer exist because of updates. (The emqx operator creates new instances on changes.)
from emqx.
could you check the container logs, it may print to the container logs or emqx logs?
from emqx.
Huh, you're right, it did log an error to the container log.
2024-04-11T14:45:22.489186+00:00 [error] msg: ctl_command_crashed, function: cluster, module: emqx_mgmt_cli, reason: {error,{no_exists,emqx_enhanced_authn_scram_mnesia}}, stacktrace: [{mria_lib,ensure_ok,1,[{file,"mria_lib.erl"},{line,200}]},{emqx_mgmt_cli,cluster,1,[{file,"emqx_mgmt_cli.erl"},{line,120}]},{emqx_ctl,run_command,2,[{file,"emqx_ctl.erl"},{line,128}]},{erpc,execute_call,4,[{file,"erpc.erl"},{line,589}]}]
from emqx.
could you run
emqx eval 'io:format("~p",[ets:tab2list(schema)])'
and send us the outputs?
from emqx.
emqx@emqx-core-f7b96944f-0:/opt/emqx$ emqx eval 'io:format("~p",[ets:tab2list(schema)])'
[{schema,emqx_shared_subscription,
[{name,emqx_shared_subscription},
{type,bag},
{ram_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_shared_subscription},
{attributes,[group,topic,subpid]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065685467428384,-576460752303423261,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,209748}}}}]},
{schema,emqx_retainer_index,
[{name,emqx_retainer_index},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,retained_index},
{attributes,[key,expiry_time]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{dets,[{auto_save,1000}]},
{ets,
[compressed,
{read_concurrency,true},
{write_concurrency,true}]}]},
{cookie,
{{1693065687469494982,-576460752303422813,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{19,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,225842}}}}]},
{schema,emqx_delayed,
[{name,emqx_delayed},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,true},
{record_name,delayed_message},
{attributes,[key,delayed,msg]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065678111567111,-576460752303423007,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{2,61},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,237902}}}}]},
{schema,emqx_enhanced_authn_scram_mnesia,
[{name,emqx_enhanced_authn_scram_mnesia},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,user_info},
{attributes,[user_id,stored_key,server_key,salt,is_superuser]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,
{{1693065684176525817,-576460752303423357,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{15,10},
{'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
{1701,660370,348172}}}}]},
{schema,emqx_trie,
[{name,emqx_trie},
{type,ordered_set},
{ram_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_trie},
{attributes,[key,count]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
{cookie,
{{1693065685361027907,-576460752303423230,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,200749}}}}]},
{schema,emqx_app,
[{name,emqx_app},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_app},
{attributes,
[name,api_key,api_secret_hash,enable,desc,expired_at,created_at]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065678325425274,-576460752303422911,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,252630}}}}]},
{schema,emqx_exclusive_subscription,
[{name,emqx_exclusive_subscription},
{type,set},
{ram_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,exclusive_subscription},
{attributes,[topic,clientid]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
{cookie,
{{1693065685725162238,-576460752303423070,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,234938}}}}]},
{schema,emqx_route_filters,
[{name,emqx_route_filters},
{type,ordered_set},
{ram_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,routeidx},
{attributes,[entry,unused]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{ets,[{read_concurrency,true},{write_concurrency,auto}]}]},
{cookie,
{{1695117719694849871,-576460752303417790,1},
'emqx@emqx-core-699f577dd8-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,217434}}}}]},
{schema,emqx_admin_jwt,
[{name,emqx_admin_jwt},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_admin_jwt},
{attributes,[token,username,exptime,extra]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
{cookie,
{{1693065678755233574,-576460752303422815,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,253496}}}}]},
{schema,emqx_banned_rules,
[{name,emqx_banned_rules},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,banned},
{attributes,[who,by,reason,at,until]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,
{{1712595007635830501,-576460752303422896,1},
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{7,3},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,241171}}}}]},
{schema,cluster_rpc_mfa,
[{name,cluster_rpc_mfa},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,cluster_rpc_mfa},
{attributes,[tnx_id,mfa,created_at,initiator]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065681678657421,-576460752303423485,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,256395}}}}]},
{schema,emqx_routing_node,
[{name,emqx_routing_node},
{type,set},
{ram_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,routing_node},
{attributes,[name,const]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,
{{1693065685544069539,-576460752303423165,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,213404}}}}]},
{schema,emqx_ds_session,
[{name,emqx_ds_session},
{type,set},
{ram_copies,[]},
{disc_copies,[]},
{disc_only_copies,[]},
{rocksdb_copies,
['emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,session},
{attributes,[id,iterators]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065686470964222,-576460752303422335,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{15,10},
{'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
{1701,660370,165129}}}}]},
{schema,emqx_admin,
[{name,emqx_admin},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_admin},
{attributes,[username,pwdhash,description,role,extra]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
{cookie,
{{1693065679931265846,-576460752303423484,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,254543}}}}]},
{schema,bpapi,
[{name,bpapi},
{type,set},
{ram_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,bpapi},
{attributes,[key,version]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065687063855530,-576460752303422910,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,258793}}}}]},
{schema,emqx_ds_iterator_ref,
[{name,emqx_ds_iterator_ref},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,[]},
{disc_only_copies,[]},
{rocksdb_copies,
['emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,iterator_ref},
{attributes,[ref_id,it_id,start_time]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1695117720120122338,-576460752303417630,1},
'emqx@emqx-core-699f577dd8-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{15,10},
{'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
{1701,660370,131478}}}}]},
{schema,emqx_activated_alarm,
[{name,emqx_activated_alarm},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,true},
{record_name,activated_alarm},
{attributes,[name,details,message,activate_at]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065685986815759,-576460752303422431,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{2,61},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,255649}}}}]},
{schema,emqx_acl,
[{name,emqx_acl},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_acl},
{attributes,[who,rules]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,
{{1693065683472063593,-576460752303423294,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,255685}}}}]},
{schema,emqx_psk,
[{name,emqx_psk},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,psk_entry},
{attributes,[psk_id,shared_secret,extra]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,
{{1693065678067829202,-576460752303423103,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,231205}}}}]},
{schema,emqx_channel_registry,
[{name,emqx_channel_registry},
{type,bag},
{ram_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,channel},
{attributes,[chid,pid]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
{cookie,
{{1693065687167321371,-576460752303422239,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,237971}}}}]},
{schema,emqx_banned,
[{name,emqx_banned},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,banned},
{attributes,[who,by,reason,at,until]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,
{{1693065685843327984,-576460752303423069,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,244950}}}}]},
{schema,mria_schema,
[{name,mria_schema},
{type,ordered_set},
{ram_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,mria_schema},
{attributes,[mnesia_table,shard,storage,config]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065677861775661,-576460752303423327,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,262931}}}}]},
{schema,cluster_rpc_commit,
[{name,cluster_rpc_commit},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,cluster_rpc_commit},
{attributes,[node,tnx_id]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065682855116971,-576460752303423358,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,257318}}}}]},
{schema,'$mria_rlog_sync',
[{name,'$mria_rlog_sync'},
{type,set},
{ram_copies,[]},
{disc_copies,[]},
{disc_only_copies,[]},
{null_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,'$mria_rlog_sync'},
{attributes,[reply_to,shard,extra]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065678029072848,-576460752303423231,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,259774}}}}]},
{schema,schema,
[{name,schema},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,schema},
{attributes,[table,cstruct]},
{user_properties,
[{mnesia_backend_types,
[{rocksdb_copies,mnesia_rocksdb},
{null_copies,mria_mnesia_null_storage}]}]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065677601354987,-576460752303423390,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{171,4},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,200725}}}}]},
{schema,emqx_route,
[{name,emqx_route},
{type,bag},
{ram_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_copies,[]},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,route},
{attributes,[topic,dest]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{ets,[{read_concurrency,true},{write_concurrency,true}]}]},
{cookie,
{{1693065685628943507,-576460752303423166,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,219865}}}}]},
{schema,emqx_dashboard_monitor,
[{name,emqx_dashboard_monitor},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,true},
{record_name,emqx_monit},
{attributes,[time,data]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065679589317207,-576460752303422719,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{2,61},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,237940}}}}]},
{schema,emqx_retainer_index_meta,
[{name,emqx_retainer_index_meta},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,retained_index_meta},
{attributes,[key,read_indices,write_indices,reindexing,extra]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{dets,[{auto_save,1000}]},
{ets,
[compressed,
{read_concurrency,true},
{write_concurrency,true}]}]},
{cookie,
{{1693065687310684089,-576460752303422909,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{19,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,224323}}}}]},
{schema,emqx_authn_scram_mnesia,
[{name,emqx_authn_scram_mnesia},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,user_info},
{attributes,[user_id,stored_key,server_key,salt,is_superuser]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,
{{1700095721479745429,-576460752303422206,1},
'emqx@emqx-core-794b5bd5b5-1.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{14,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,250613}}}}]},
{schema,emqx_deactivated_alarm,
[{name,emqx_deactivated_alarm},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,true},
{record_name,deactivated_alarm},
{attributes,[activate_at,name,details,message,deactivate_at]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065686101778828,-576460752303423388,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{2,61},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,237925}}}}]},
{schema,emqx_telemetry,
[{name,emqx_telemetry},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,telemetry},
{attributes,[id,uuid]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065688182842201,-576460752303422143,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,208394}}}}]},
{schema,emqx_trace,
[{name,emqx_trace},
{type,set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,emqx_trace},
{attributes,
[name,type,filter,enable,payload_encode,extra,start_at,end_at]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[]},
{cookie,
{{1693065687234401845,-576460752303423005,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,205979}}}}]},
{schema,emqx_retainer_message,
[{name,emqx_retainer_message},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,retained_message},
{attributes,[topic,msg,expiry_time]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,
[{dets,[{auto_save,1000}]},
{ets,
[compressed,
{read_concurrency,true},
{write_concurrency,true}]}]},
{cookie,
{{1693065687345372571,-576460752303422814,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{19,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,222014}}}}]},
{schema,emqx_authn_mnesia,
[{name,emqx_authn_mnesia},
{type,ordered_set},
{ram_copies,[]},
{disc_copies,
['emqx@emqx-core-f7b96944f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local']},
{disc_only_copies,[]},
{load_order,0},
{access_mode,read_write},
{majority,false},
{index,[]},
{snmp,[]},
{local_content,false},
{record_name,user_info},
{attributes,[user_id,password_hash,salt,is_superuser]},
{user_properties,[]},
{frag_properties,[]},
{storage_properties,[{ets,[{read_concurrency,true}]}]},
{cookie,
{{1693065684858268518,-576460752303422591,1},
'emqx@emqx-core-846d4f7977-0.emqx-headless.home-network.svc.cluster.local'}},
{version,
{{18,18},
{'emqx@emqx-core-f7b96944f-0.emqx-headless.home-network.svc.cluster.local',
{1712,595312,251641}}}}]}]ok
from emqx.
Thanks for the info, so the problem is there is one deprecated table (emqx_enhanced_authn_scram_mnesia
) since 5.3.1 was used by old version of EMQX on old nodes. However this table are no longer in use by new nodes.
Now when you call nodes to leave, the cluster complains because the old deleted nodes has the copy which shouldn't be deleted as they are the only nodes holding the original copy of the data.
Did you upgrade the EMQX cluster with emqx-operator? could you recall the command you use for us to reproduce?
ping @Rory-Z to get some attention as it was deplyed and managed by Emqx-operator.
on the other hand, while we are investigating are you willing to run some hacking command to get these nodes out of your cluster?
I won't recommend to do that if you have live nodes. prefer to reproduce the issue in a staging env and try out the hacking commands.
from emqx.
I do not believe I will be able to reliably reproduce this on demand now, given that I updated to 5.3.1 over 5 months ago. And for the most part I just update the EMQX CRD spec in git and it gets applied to my k8s cluster automatically where the operator proceeds to update the emqx cluster.
I have no issue with running command hacks on it as it's a homelab environment and not anything production-critical. It would be appreciated as it would likely be easier then having to set up all the authentication credentials again for MQTT and updating various IoT devices using them.
from emqx.
without reproducing the issue it is hard to find a proper commands for you to remove the dead nodes.
if possible, I think it is easier just to recreate your home lab cluster.
But if you want to try we have some hacks: NOTE: it may corrupt you entire data cause cluster refuse to come up.
- In the emqx container, Open emqx remote console with
emqx remote_console
- copy & paste following commands
DeadNodes = ['emqx@emqx-core-767989788f-0.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-1.emqx-headless.home-network.svc.cluster.local',
'emqx@emqx-core-767989788f-2.emqx-headless.home-network.svc.cluster.local'].
rr(mnesia_schema).
Tab = emqx_enhanced_authn_scram_mnesia.
lists:foreach(fun(Node) -> mnesia:add_table_copy(Tab, Node, ram_copies) end, mria:running_nodes()),
[{K,#cstruct{disc_copies = DC, ram_copies = RC} = V}] = ets:lookup(mnesia_gvar, {Tab, cstruct}).
ets:insert(mnesia_gvar, {K,V#cstruct{disc_copies = DC--DeadNodes, ram_copies = RC--DeadNodes}}).
{atomic, ok} = mnesia:delete_table(Tab).
-
then exit the remote console with `Ctl-G' then 'q' enter...
-
then retry with force-leave commands...
-
you will hit other tables, then just repeat 1-4 with other tables. Maybe it will be easier with another layer of scripts.
from emqx.
Related Issues (20)
- Can't add users on touch screen device HOT 7
- Action output to external HTTP server - authorization header issue HOT 2
- The client disconnects after publish message whose topic starts with '$' HOT 2
- Not getting Emqx dashboard HOT 1
- EMQX Clustering - Message Replication HOT 4
- STOMP GW does not send heartbeat to client HOT 4
- EMQX cluster cannot restart after persistence HOT 1
- After STOMP enables authentication, authentication fails when the account password carries a colon
- runq_overload everyday for few minutes HOT 4
- Receive Maximum Not Sent in Bridge CONNACK HOT 4
- ~10ms latency on publishing and receiving message on the same machine on windows HOT 6
- webhook监听上下线事件时,事件时序有误,原因不明(connected and disconnected events may out of order) HOT 3
- Authentication fails periodically and restart fixes it HOT 9
- For my project required kafka and kerberos integration with EMQX opensource,So any how is it possible to do these integration with Emqx OPenSource version HOT 2
- Get acknowledgement from subscriber(s) after publish messages HOT 2
- Restored retained message have no payload HOT 2
- error messages received and MQTT broker keep running up and down, very HOT 4
- Several protocol violations or bugs in EMQX HOT 26
- Upgrade to Openssl 3.0 or higher HOT 7
- LDAP server treated as down due to wrong error returned than expected? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emqx.