[Bug]: CN panic under extreme testing conditions

[ouyuanning]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

main

Commit ID

main

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

测试场景：
1、分别启动log , tn, cn
2、进入cn，刷入tpcc 10仓数据
3、启动tpcc测试
4、不断kill tn, 重启tn, sleep 10秒... 循环
大概15分钟左右，cn panic，panic时最后几行日志如下：

{"level":"WARN","time":"2025/01/09 15:56:09.298875 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadaf"}
{"level":"WARN","time":"2025/01/09 15:56:09.299639 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb0"}
{"level":"WARN","time":"2025/01/09 15:56:09.300453 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb1"}
{"level":"INFO","time":"2025/01/09 15:56:09.300800 +0800","caller":"disttae/engine.go:370","msg":"FIND_TABLE loadDatabaseFromStorage","name":"mo_task","cacheTs":"1736409369250790000-0","txn":"70f70f4866a9b62a1818f63e2ecbadaf/Active/S:1736409333361697000-1"}
{"level":"WARN","time":"2025/01/09 15:56:09.301013 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb2"}
{"level":"WARN","time":"2025/01/09 15:56:09.301735 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb3"}
{"level":"INFO","time":"2025/01/09 15:56:09.301663 +0800","name":"cn-service.lockservice","caller":"lockservice/service.go:607","msg":"bind created","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","service":"1736408403282447000dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","remote":false,"bind":"0-2(2)-1736408403282447000dd1dccb4-4d3c-41f8-b482-5251dc7a41bf-1736409363515803000"}
{"level":"WARN","time":"2025/01/09 15:56:09.301890 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb4"}
{"level":"WARN","time":"2025/01/09 15:56:09.302264 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb5"}
{"level":"INFO","time":"2025/01/09 15:56:09.303225 +0800","caller":"disttae/logtail_consumer.go:521","msg":"[logtail-consumer] send subscribe tbl[db: 272401, tbl: 272410] request succeed"}
{"level":"WARN","time":"2025/01/09 15:56:09.303361 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb6"}
{"level":"INFO","time":"2025/01/09 15:56:09.303378 +0800","caller":"disttae/logtail_consumer.go:521","msg":"[logtail-consumer] send subscribe tbl[db: 1, tbl: 272416] request succeed"}
{"level":"WARN","time":"2025/01/09 15:56:09.303418 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb7"}
{"level":"WARN","time":"2025/01/09 15:56:09.303470 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb8"}
{"level":"INFO","time":"2025/01/09 15:56:09.303532 +0800","caller":"disttae/logtail_consumer.go:521","msg":"[logtail-consumer] send subscribe tbl[db: 272389, tbl: 272391] request succeed"}
{"level":"WARN","time":"2025/01/09 15:56:09.303619 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadb9"}
{"level":"WARN","time":"2025/01/09 15:56:09.303736 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadba"}
{"level":"WARN","time":"2025/01/09 15:56:09.303847 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadbc"}
{"level":"INFO","time":"2025/01/09 15:56:09.303904 +0800","name":"cn-service","caller":"client/client.go:605","msg":"txn client status changed to paused","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf"}
{"level":"INFO","time":"2025/01/09 15:56:09.303942 +0800","caller":"disttae/logtail_consumer.go:663","msg":"[logtail-consumer] dd1dccb4-4d3c-41f8-b482-5251dc7a41bf: logtail consumers stopped"}
{"level":"INFO","time":"2025/01/09 15:56:09.303975 +0800","caller":"disttae/logtail_consumer.go:869","msg":"[logtail-consumer] dd1dccb4-4d3c-41f8-b482-5251dc7a41bf: clean finished, start to reconnect to tn log tail service"}
{"level":"INFO","time":"2025/01/09 15:56:09.303985 +0800","caller":"disttae/logtail_consumer.go:1607","msg":"[logtail-consumer] try to get logtail service address"}
{"level":"INFO","time":"2025/01/09 15:56:09.303994 +0800","caller":"disttae/logtail_consumer.go:1610","msg":"[logtail-consumer] got logtail service address: 127.0.0.1:19001"}
{"level":"WARN","time":"2025/01/09 15:56:09.304016 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbada9"}
{"level":"WARN","time":"2025/01/09 15:56:09.304027 +0800","name":"cn-service","caller":"client/client.go:515","msg":"txn client is in pause state, wait for it to be ready","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbada9"}
{"level":"WARN","time":"2025/01/09 15:56:09.304047 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadbe"}
{"level":"WARN","time":"2025/01/09 15:56:09.304129 +0800","name":"cn-service","caller":"client/client.go:515","msg":"txn client is in pause state, wait for it to be ready","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadbe"}
{"level":"WARN","time":"2025/01/09 15:56:09.304158 +0800","name":"cn-service","caller":"client/client.go:520","msg":"txn client is in ready state","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadbd"}
{"level":"WARN","time":"2025/01/09 15:56:09.304169 +0800","name":"cn-service","caller":"client/client.go:515","msg":"txn client is in pause state, wait for it to be ready","service":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","uuid":"dd1dccb4-4d3c-41f8-b482-5251dc7a41bf","txn ID":"70f70f4866a9b62a1818f63e2ecbadbd"}
panic: send on closed channel

goroutine 610 [running]:
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*routineController).updateTimeFromT(0x1400dd17c80, {0x1818f71f19f2ab38, 0x0, 0x0, {}, {0x0, 0x0, 0x0}, 0x0}, {0x0?, ...})
	/Users/ouyuanning/workspace/go/src/matrixone/pkg/vm/engine/disttae/logtail_consumer.go:1824 +0x158
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.dispatchSubscribeResponse({0x107139840?, 0x1404147eb60?}, 0x14002a07b08, 0x1070f65f8?, {0x14006f24100, 0x4, 0x1400dd17f48?}, {0x2?, 0xc1d7fe264c2ed9c8?, 0x109b39b00?})
	/Users/ouyuanning/workspace/go/src/matrixone/pkg/vm/engine/disttae/logtail_consumer.go:1706 +0x270
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*PushClient).receiveOneLogtail(0x14002a07c50, {0x1071397d0?, 0x14000180910?}, 0x14002a07b08)
	/Users/ouyuanning/workspace/go/src/matrixone/pkg/vm/engine/disttae/logtail_consumer.go:593 +0x4d4
github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*PushClient).receiveLogtails(0x14002a07c50, {0x1071397d0, 0x14000180910}, 0x14002a07b08)
	/Users/ouyuanning/workspace/go/src/matrixone/pkg/vm/engine/disttae/logtail_consumer.go:644 +0x168
created by github.com/matrixorigin/matrixone/pkg/vm/engine/disttae.(*PushClient).run in goroutine 598
	/Users/ouyuanning/workspace/go/src/matrixone/pkg/vm/engine/disttae/logtail_consumer.go:685 +0x94

Expected Behavior

No response

Steps to Reproduce

类似上面描述

Additional information

No response

[volgariver6]

fixed