Crash from DistributedAsyncInsertDirectoryQueue::getStatus

[filimonov]

[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.914437 [ 22000 ] <Fatal> BaseDaemon: (version 23.8.11.29.altinitystable (altinity build), build id: C485509DB13B40C787D7AC10C9FD049303477677, git hash: 8e9be16153250c8307730c9b88c8517181f494ce) (from thread 21737) (query_id: fa7bef29-a36b-4872-89ab-0d6d1458bdc1) (query: SELECT hostName(), `database`, `table`, `data_path`, `is_blocked`, `error_count`, `data_files`, `data_compressed_bytes`, `broken_data_files`, `broken_data_compressed_bytes`, `last_exception`, `last_exception_time` FROM `system`.`distribution_queue` WHERE `data_files` > 0) Received signal Segmentation fault (11)
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.914503 [ 22000 ] <Fatal> BaseDaemon: Address: 0xe0. Access: read. Address not mapped to object.
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.914547 [ 22000 ] <Fatal> BaseDaemon: Stack trace: 0x00007f1a42ddcfc4 0x0000000012adf865 0x0000000012649295 0x0000000010b84141 0x0000000010a2a661 0x00000000125d3fb9 0x0000000011eed96b 0x0000000011ed51c8 0x0000000011ed45f4 0x0000000011f78996 0x0000000011f798a7 0x00000000122bfe15 0x00000000122bb5b5 0x0000000013137519 0x00000000131498f9 0x0000000015b42834 0x0000000015b43a31 0x0000000015c7a667 0x0000000015c7893c 0x00007f1a42dda609 0x00007f1a42cff353
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.914596 [ 22000 ] <Fatal> BaseDaemon: 2. ? @ 0x00007f1a42ddcfc4 in ?
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.917409 [ 22000 ] <Fatal> BaseDaemon: 3. DB::DistributedAsyncInsertDirectoryQueue::getStatus() @ 0x0000000012adf865 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.917470 [ 22000 ] <Fatal> BaseDaemon: 4. DB::StorageDistributed::getDirectoryQueueStatuses() const @ 0x0000000012649295 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.919854 [ 22000 ] <Fatal> BaseDaemon: 5. DB::StorageSystemDistributionQueue::fillData(std::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn>>>&, std::shared_ptr<DB::Context const>, DB::SelectQueryInfo const&) const @ 0x0000000010b84141 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.925928 [ 22000 ] <Fatal> BaseDaemon: 6. DB::IStorageSystemOneBlock<DB::StorageSystemDistributionQueue>::read(std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::SelectQueryInfo&, std::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x0000000010a2a661 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.926022 [ 22000 ] <Fatal> BaseDaemon: 7. DB::IStorage::read(DB::QueryPlan&, std::vector<String, std::allocator<String>> const&, std::shared_ptr<DB::StorageSnapshot> const&, DB::SelectQueryInfo&, std::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x00000000125d3fb9 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.926083 [ 22000 ] <Fatal> BaseDaemon: 8. DB::InterpreterSelectQuery::executeFetchColumns(DB::QueryProcessingStage::Enum, DB::QueryPlan&) @ 0x0000000011eed96b in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.928498 [ 22000 ] <Fatal> BaseDaemon: 9. DB::InterpreterSelectQuery::executeImpl(DB::QueryPlan&, std::optional<DB::Pipe>) @ 0x0000000011ed51c8 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.928705 [ 22000 ] <Fatal> BaseDaemon: 10. DB::InterpreterSelectQuery::buildQueryPlan(DB::QueryPlan&) @ 0x0000000011ed45f4 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.928771 [ 22000 ] <Fatal> BaseDaemon: 11. DB::InterpreterSelectWithUnionQuery::buildQueryPlan(DB::QueryPlan&) @ 0x0000000011f78996 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.928829 [ 22000 ] <Fatal> BaseDaemon: 12. DB::InterpreterSelectWithUnionQuery::execute() @ 0x0000000011f798a7 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.928890 [ 22000 ] <Fatal> BaseDaemon: 13. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000122bfe15 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.928934 [ 22000 ] <Fatal> BaseDaemon: 14. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x00000000122bb5b5 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.928989 [ 22000 ] <Fatal> BaseDaemon: 15. DB::TCPHandler::runImpl() @ 0x0000000013137519 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.929031 [ 22000 ] <Fatal> BaseDaemon: 16. DB::TCPHandler::run() @ 0x00000000131498f9 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.929085 [ 22000 ] <Fatal> BaseDaemon: 17. Poco::Net::TCPServerConnection::start() @ 0x0000000015b42834 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.929123 [ 22000 ] <Fatal> BaseDaemon: 18. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b43a31 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.929156 [ 22000 ] <Fatal> BaseDaemon: 19. Poco::PooledThread::run() @ 0x0000000015c7a667 in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.929234 [ 22000 ] <Fatal> BaseDaemon: 20. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c7893c in /usr/bin/clickhouse
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.929290 [ 22000 ] <Fatal> BaseDaemon: 21. ? @ 0x00007f1a42dda609 in ?
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:22.929336 [ 22000 ] <Fatal> BaseDaemon: 22. ? @ 0x00007f1a42cff353 in ?
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:23.197248 [ 22000 ] <Fatal> BaseDaemon: Integrity check of the executable successfully passed (checksum: 0F9D2E506AEB0B88DA19C06F13462FA9)
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:23.197584 [ 22000 ] <Fatal> BaseDaemon: You are using an Altinity Stable Build. Please log issues at https://url41.mailanyone.net/scanner?m=1tIpkO-000000007nD-2Snn&d=4%7Cmail%2F90%2F1733319600%2F1tIpkO-000000007nD-2Snn%7Cin41g%7C57e1b682%7C24575110%7C11599412%7C67505DC8AE07E70CF00CE99F6C5879D4&o=%2Fphti%3A%2FgtsmbthAco%2Fu.%2FnltltyCiieHicius%2Fkoseuss&s=CIdOKK0kBgJpqUfEug_ubBdAe9Y. Thank you!
[chi-clickhouse-clickhouse-6-0-0] 2024.12.04 13:46:23.197798 [ 22000 ] <Fatal> BaseDaemon: Changed settings: max_query_size = 16777216, connect_timeout_with_failover_ms = 5000, connect_timeout_with_failover_secure_ms = 5000, use_hedged_requests = false, distributed_directory_monitor_batch_inserts = true, distributed_directory_monitor_split_batch_on_failure = false, load_balancing = 'nearest_hostname', distributed_aggregation_memory_efficient = true, do_not_merge_across_partitions_select_final = true, os_thread_priority = 2, log_queries = true, joined_subquery_requires_alias = false, query_profiler_real_time_period_ns = 0, query_profiler_cpu_time_period_ns = 0, opentelemetry_start_trace_probability = 0., opentelemetry_trace_processors = false, timeout_before_checking_execution_speed = 0., memory_profiler_step = 0, prefer_localhost_replica = false, parallel_view_processing = true, cancel_http_readonly_queries_on_client_close = true, materialize_ttl_after_modify = false, cast_ipv4_ipv6_default_on_conversion_error = true, optimize_on_insert = false, background_pool_size = 64, background_fetches_pool_size = 512, background_schedule_pool_size = 512, background_distributed_schedule_pool_size = 64
↘ Progress: 795.13 thousand rows, 171.17 MB (196.53 thousand rows/s., 42.31 MB/s.)                                  (0.0 CPU, 29.71 MB RAM, 2.17 MB max/host)command terminated with exit code 137

The node was working in some strange state (may be reaching some limits, like file limits) when the problem happened.