MatrixOne-v1.2.1

Release date: June 30, 2024
MatrixOne version: v1.2.1

Compared with the previous v1.2.0, v1.2.1 doesn't introduce new features but only focuses on bug fixes and minor optimizations.

Improvements

• Memory cache supports multiple memory allocators.
• Optimize the performance of show accounts.
• Optimize observability metrics for fileservice.
• Support incremental backup.
• Optimize TCP packet estimation.
• Refactor mologging.
• Support restoration of system tables.
• Support manual merge based on zonemap.

Bugfix

Fixed bugs related to snapshot reads, MySQL compatibility, lockservice, runtime filters, and system table upgrades. Check the What's Changed section for details.

What's Changed

• fix query result by @daviszhen in #16266
• fix restore partition table (#1.2-dev) by @YANGGMM in #16256
• pick #16245 to 1.2 : reduce agg context memory usage. by @m-schen in #16253
• sync to 1.2: optimize show accounts by @gouhongshen in #16241
• modify snapshot case syntax(1.2-dev) by @Ariznawlll in #16233
• fix visit tombstone, cherry-pick #16163 by @jiangxinmeng1 in #16189
• add debug info & profile interval by @daviszhen in #16223
• move some logic from compile to run (#15983) by @ouyuanning in #16079
• Fix mo cloud-issue-3298 (#16254) by @triump2020 in #16281
• [fix] use runtime filter for multi-row prepared insert stmt (cherry-pick to 1.2-dev) by @aunjgr in #16261
• refactor tn handler 1.2 by @aptend in #16136
• Support not deleting the checkpoint referenced by the snapshot 1.2 by @LeftHandCold in #16128
• test restore fk table by @YANGGMM in #16259
• Extended version upgrade package (#16049) -1.2-dev by @qingxinhome in #16271
• handle alter table add column issue (#16122) -1.2-dev by @qingxinhome in #16267
• inc ref of new bind in restart cn by @iamlinjunhong in #16294
• fix TestDedup2, cherry-pick #16301 by @jiangxinmeng1 in #16302
• syn to 1.2: collecting the mo_database, mo_tables, mo_columns and cluster table size for non-sys account. by @gouhongshen in #16287
• GC needs to be compatible with the earliest checkpoint for 1.2 by @LeftHandCold in #16305
• GC tombstone object (#16102) 1.2 by @LeftHandCold in #16238
• Refactor: change Chan of WaitRegister from *Batch to *Message(contains batch & error) by @ouyuanning in #16263
• change keep_user_target_list_in_result default to 1 by @YANGGMM in #16234
• Improve sql usage 1.2 dev by @w-zr in #16277
• fix 11250 on 1.2-dev by @w-zr in #16327
• pick 16239 and 16248 to 1.2-dev by @XuPeng-SH in #16257
• convert dup error if txn is orphan to 1.2 by @zhangxu19830126 in #16334
• fix TestRemovedCronTask panic on 1.2-dev by @w-zr in #16323
• fix restore view table (#1.2-dev) by @YANGGMM in #16330
• fix can not connect to remote backend when down cn by @iamlinjunhong in #16329
• fileservice: add i/o timeout as retryable error by @reusee in #16347
• add case for restore view(1.2-dev) by @Ariznawlll in #16335
• update save profile by @daviszhen in #16356
• fix show create table with fk in other database (#1.2-dev) by @YANGGMM in #16349
• add some recover logs by @volgariver6 in #16363
• fix rollback create table by @jiangxinmeng1 in #16369
• fix does not unlock deadlock txn by @iamlinjunhong in #16382
• fileservice: more metrics for Read by @reusee in #16390
• rm tag 16311 16306 16297(1.2-dev) by @Ariznawlll in #16396
• sync to 1.2: remove the old fast ranges and update metrics by @gouhongshen in #16359
• fix force flush timeout by @jiangxinmeng1 in #16381
• [bug] proxy: do not add to queue multi times when scaling CN by @volgariver6 in #16388
• add txn id by @daviszhen in #16395
• remove NullComputationWrapper by @daviszhen in #16290
• [cherry-pick-16195] : remove lock op for secondary index by @jensenojs in #16409
• Fix GetByteSlice out range for 1.2 by @LeftHandCold in #16418
• Optimize the estimation of tcp packet num send back to client in mo1.2 by @xzxiong in #16412
• fix panic during merge on 1.2-dev by @w-zr in #16416
• [opt] merge filters on composite primary keys in plan (cp to 1.2-dev) by @aunjgr in #16406
• fix 15848, cherry-pick #16435 by @jiangxinmeng1 in #16436
• sync to 1.2, re-pr: remove the old fast ranges and update metrics by @gouhongshen in #16444
• Fix fifocache evict for 1.2 by @LeftHandCold in #16451
• refactor mo logging for rawlog and statement_info in mo1.2 by @xzxiong in #16411
• [Cherry-pick] fix snapshot restore fk cross db by @ck89119 in #16454
• Optimize QueryResult decoupling from motrace in mo1.2 by @xzxiong in #16426
• cherry_pick-Supports Information_Schema FILES and is compatible with MySQL (#15… by @qingxinhome in #16457
• [cherry-pick-16445] : use ctx instead of proc.ctx for rangesOnePart by @jensenojs in #16465
• Fix-RW-Conflict by @triump2020 in #16477
• fix txnid by @daviszhen in #16458
• [cherry-pick-16424] : add log to track 16273 by @jensenojs in #16472
• chore: stable bvt/result_count.sql result check and enable load_data_LOCAL_infile case in mo1.2 by @xzxiong in #16478
• pick pr #16452 to 1.2-dev, fix a bug will cause hung if error happens. by @m-schen in #16453
• Support upgrading to skip Genesis version functionality-1.2-dev by @qingxinhome in #16486
• delay Ranges for partitioned table-1.2-dev by @qingxinhome in #16268
• fix stats misuse in some cases to 1.2-dev by @badboynt1 in #16500
• [CherryPick] : Fix failed BVT query in Secondary Index (#16429) by @arjunsk in #16499
• add test for restore view table and fk table (#1.2-dev) by @YANGGMM in #16432
• sync to 1.2: supporting reader linear search on unsorted fake pk col by @gouhongshen in #16461
• add hidden force option for objprune 1.2 by @aptend in #16514
• support restore system table (#1.2-dev) by @YANGGMM in #16462
• sync to 1.2: skip collecting mo_tables, mo_databases, mo_columns and cluster tables for non-sys. by @gouhongshen in #16515
• chore: make sure no aggr in bvt result_count in 1.2 by @xzxiong in #16536
• remove log print from automaxprocs to 1.2-dev by @badboynt1 in #16547
• malloc: fix log level by @reusee in #16509
• fix lock service ut by @iamlinjunhong in #16518
• support merge objects by table id on 1.2-dev by @w-zr in #16520
• sync to 1.2: making collect dirty blocks lazily by @gouhongshen in #16527
• chore: refactor purge_log calling objprune if target_date + 24h <= now (mo1.2) by @xzxiong in #16550
• sync to 1.2: filter optimization of the loa...

MatrixOne-v1.2.0

We are excited to announce MatrixOne 1.2.0 release!

MatrixOne is a hyper-converged cloud-native database. It is designed to provide a cloud-native, high-performance, highly scalable, MySQL-compatible HTAP database. MatrixOne enables users to handle mixed workloads such as transactions, analytics, time-series, and streaming processing through a one-stop data processing solution.

What's New in v1.2.0?

Snapshot Backup and Restore (Beta)

Database snapshot is an efficient technology for database backup and recovery, providing a read-only static copy of the database at a specific point in time. It assists database administrators and developers in performing various operations while ensuring the consistency and integrity of the data.

• Support for tenant-level snapshot backup and restore using the mo_br tool and sql statement.
• Support for uninterrupted database operation during the snapshot creation process.
• Support for rapidly restoring data to the state at the time of backup, effectively reducing the Recovery Time Objective (RTO).

Incremental Physical Backup(Enterprise Edition)

On the basis of full backup in the mo_backup tool, we support for incremental backup feature , which only backs up the newly changed data, significantly reducing the backup time and storage space requirements.

CTAS

Support for CTAS (Create Table As Select).CTAS is an SQL statement used to quickly create a new table based on existing data. It combines the functionality of table creation with a select query, providing an efficient way to create snapshots of tables, perform data transformations, or build data models for reporting and analysis.

BITMAP Fast Deduplication

BITMAP is a set of built-in functions for handling bitmaps, mainly used for rapid deduplication of large volumes of data.

• Support for rapid data deduplication using BITMAP functions.
• Support for determining the bucket number with the BITMAP_BUCKET_NUMBER() function.
• Support for returning the relative bit position within a bucket using the BITMAP_BIT_POSITION() function.
• Support for constructing bitmaps with the BITMAP_CONSTRUCT_AGG() function.

Vector Index

• Support for using vector indexing to accelerate KNN queries.

Other New Features

SQL Statements

• Support for INSERT IGNORE
• Support for CREATE TABLE ... LIKE
• Support for CREATE INDEX ... USING IVFFlat
• Support for ALTER TABLE ... ALTER REINDEX
• Support for LOAD DATA ... CHARACTER SET
• Support for CREATE SNAPSHOT
• Support for SHOW SNAPSHOTS
• Support for RESTORE ACCOUNT
• Support for DROP SNAPSHOT
• Optimized ALTER PUBLICATION
• Optimized SHOW PUBLICATIONS
• Optimized SHOW SUBSCRIPTIONS

Date Type

• Support for bit

Indexes and Constraints

• Support for Vector Index

Built-in Functions and Operators

• Added SYSDATE date function.
• Added TO_BASE64 and FROM_BASE64 encoding and decoding functions.
• Added MD5 and SHA1/SHA encryption functions.
• Added SUBVECTOR function for extracting subvectors.
• Added SERIAL_EXTRACT function for extracting sub-elements.
• Added CLUSTER_CENTERS cluster centers function.
• Support operations between vector and scalar.

System Parameters

• Added keep_user_target_list_in_result.
• Added foreign_key_checks.

MySQL Compatibility

• Refactored the CSV reader and CSV splitting to maintain compatibility with MySQL.

Known Issues

• Vector Index works only in l2_distance.
• Only support tenant-level snapshot backup and restore.
• Snapshot restore works as logical restore, requiring quite a lot of CPU and memory resources.
• Memory leak occasionally happens and may lead to an OOM error.
• Occasional system hung under high concurrency workload.

Full Changelog:v1.1.3...v1.2.0

MatrixOne-v1.1.3

Release date: April 16, 2024
MatrixOne version: v1.1.3

Compared with the previous v1.1.2, v1.1.3 doesn't introduce any new features but only focuses on bug fixes.

Bugfix

• Resolved data integrity issues caused by orphaned transactions #15452 .
• Fixed potential transaction leak issues #15505 .
• Addressed issues related to system tables #15331 #15332 .
• Rectified problems associated with the lockable #15402 .

What's Changed

• Solve cross tenant system view query error by @qingxinhome in #15264
• fix a bug that some filters don't push to block filter for 1.1-dev by @badboynt1 in #15290
• CI: upgrade docker compose to v2 for 1.1-dev by @guguducken in #15306
• fix json_extract(#1.1-dev) by @YANGGMM in #15305
• fix keep remote lock by @iamlinjunhong in #15297
• [Enhancement] add conn and req count metric by @ck89119 in #15284
• cherry-pick sys variables by @YANGGMM in #15330
• [Fix] remove trimspace in user authenticate by @ck89119 in #15259
• add alter table for enum type(1.1-dev) by @Ariznawlll in #15323
• bugfix: copy the map while SetLabelSelector by @xzxiong in #15283
• [enhancement] sql: add client host and proxy host in session status. by @volgariver6 in #15256
• fix sync lock table between cn and dn by @iamlinjunhong in #15350
• CI: migrate ci and remove some unuse file for 1.1-dev by @guguducken in #15353
• cherry pick Supports REFERENTIAL_CONSTRAINTS system view by @qingxinhome in #15333
• chore: record exec plan and cu stats while query failed in mo 1.1 by @xzxiong in #15361
• [Fix] table name overwritten when show table status by @ck89119 in #15362
• fix wait too long to 1.1 by @zhangxu19830126 in #15373
• [cherry-pick-1.1-dev] improve delete from table without where & limit (#13120) by @noorall in #15372
• non-sys tenant should not support cluster level backup(#1.1-dev) by @YANGGMM in #15383
• support push from_unixtime to block filters by @badboynt1 in #15387
• cherry-pick index related commits to 1.1-dev by @aunjgr in #15406
• sync to 1.1: a quick fix that avoids the failure of mo_table_size when the index table name is empty. by @gouhongshen in #15403
• fix handle lock table bind changed err by @iamlinjunhong in #15411
• support unknown flag by @zhangxu19830126 in #15397
• Revert #15348 by @zhangxu19830126 in #15421
• always try to push block filter for 1.1 by @badboynt1 in #15409
• Adjust mo_indexes Upgrade Order by @qingxinhome in #15424
• Handle system view Processlist Not Available issue by @qingxinhome in #15428
• fix validate lock service by @iamlinjunhong in #15437
• [Tech Request]: time and datetime type can not fold correctly (#14698) by @badboynt1 in #15439
• [cherry-pick-1.1-dev] fix load data stmt affect rows count calculation incorrectly by @noorall in #15435
• cherry-pick bug fixes to 1.1-dev by @aunjgr in #15425
• [enhancement] proxy: mark internal sql as non-user-sql by @volgariver6 in #15436
• add case for information_schema.referenctial_constraints(1.1-dev) by @Ariznawlll in #15392
• sync to 1.1: fix db not found when mo_table_size by @gouhongshen in #15444
• fix blocksel by @badboynt1 in #15446
• Dealing with tenant upgrade failure issues by @qingxinhome in #15449
• cherry pick "optimize plan for update with secondary index column (#14856)" to 1.1 by @badboynt1 in #15454
• fix date_add with param input (#1.1-dev) by @YANGGMM in #15450
• fix stats with limit by @badboynt1 in #15458
• pick 'chore: stable bvt case result abort statement query_type' in 1.1 by @xzxiong in #15470
• detect CN status before remove the lock on an orphan txn by @zhangxu19830126 in #15478
• modify defaultMaxLockRowCount to reduce deadlocks by @iamlinjunhong in #15472
• fix bvt/purge_log (#1.1-dev) by @YANGGMM in #15484
• fix panic too large fixed slice by @iamlinjunhong in #15487
• let go schedule cpu in large ap query (#15384) to 1.1 by @badboynt1 in #15483
• [bug] tae: fix hang issue and reserve some table IDs. by @volgariver6 in #15503
• Revert "detect CN status before remove the lock on an orphan txn" by @sukki37 in #15515
• split lock time consumption （1.1） by @Morranto in #15507
• Revert "Revert "detect CN status before remove the lock on an orphan txn"" by @sukki37 in #15534
• fix lockservice will hung in unstable network by @zhangxu19830126 in #15514
• fix stats for block filters by @badboynt1 in #15532
• Revert "cherry-pick sys variables" by @sukki37 in #15560

Full Changelog: v1.1.2...v1.1.3

MatrixOne-v1.1.2

Release date: April 02, 2024
MatrixOne version: v1.1.2

Compared with the previous v1.1.1, v1.1.2 doesn't introduce any new features but focuses on some improvements and bug fixes.

Improvements

• Query performance optimization #14819, #14519, #14719, #14835;
• Backup parallelization optimization #14543;
• Support for S3 retry #14992;
• Optimization of Proxy migration capabilities #14858, #14901, #15162;
• Optimized UUID performance #15187.

Bugfix

• Fixed data accuracy issues #14653;
• Fixed transaction commit WW and DUP issues #15034, #15123, #15145;
  For additional bug fixes, please refer to the "What's Changed" section for more details.

What's Changed

• support parallel config for backup (#1.1-dev) by @YANGGMM in #14539
• fix bug by @triump2020 in #14527
• cherry-pick index related commits to 1.1-dev by @aunjgr in #14519
• Add backup parallel by @LeftHandCold in #14543
• Move write-offset update from Ranges to Relation (#14434) by @ouyuanning in #14545
• Add more precise workspace size control (#14497) by @ouyuanning in #14546
• fix data race (cherry-pick #14429) by @jiangxinmeng1 in #14439
• sync to 1.1: increase the default val of the tn rpc worker by @gouhongshen in #14582
• fix scanner(1.1-dev) by @YANGGMM in #14567
• [Bug]: Incorrect explain in bvt with race enabled to 1.1-dev by @badboynt1 in #14596
• support isnull and isnotnull in block filter to 1.1-dev by @badboynt1 in #14595
• fileservice: add NoDefaultCredentialsForETL by @reusee in #14565
• fix writeoffset data race by @Morranto in #14606
• sync to 1.1: an interface for copy-on-write of table def by @gouhongshen in #14626
• fix bug: get incorrect result for "in" expr in some case (#14602) by @ouyuanning in #14631
• fix dump enum type error (#1.1-dev) by @YANGGMM in #14655
• fix scanner for '#' (#1.1-dev) by @YANGGMM in #14659
• sync to 1.1: adding detailed phased trace in the TN commit routine helps us catch the time killer when TN commits. by @gouhongshen in #14663
• 1.1: check nil vector before flushing blk & print w-w detail in pessimistic mode by @aptend in #14653
• sync to 1.1: clean all usage of the StartTxnWithLastestTs by @gouhongshen in #14684
• 1.1: add flush range lag by @aptend in #14690
• fix constant folding of literal vector with 1 item by @aunjgr in #14719
• 1.1: fix wrong Next call on memory deletes collecting by @aptend in #14759
• Fix TestAppendAndGC for 1.1 by @LeftHandCold in #14768
• sync to 1.1: restore the switched account id when mo_table_size and mo_table_rows. by @gouhongshen in #14741
• fix: fixed panic when python server cannot be connected (#14145) by @aylei in #14735
• Fix memory leak caused by fuzzyfilter by @jensenojs in #14777
• Added back to the release of compile and its referenced objects by @jensenojs in #14791
• sync to 1.1: fix the logic err in PrefixIn when vector unsorted by @gouhongshen in #14797
• [bug] BR: fix bug when restoring data by @volgariver6 in #14800
• fileservice: add ObjectStorageArguments.NoBucketValidation (#14510) by @reusee in #14808
• modify mo_task primary key type to bigint in 1.1 by @w-zr in #14809
• remove some stats to improve tp performance in 1.1-dev by @XuPeng-SH in #14819
• [fix] : fix #14521 by @jensenojs in #14816
• fix data race in ApplyCommit, cherry-pick 14709 by @jiangxinmeng1 in #14822
• change plan for insert without unique key column (#14713) by @ouyuanning in #14802
• cherry pick fix 14780 (#14782) to 1.1-dev by @XuPeng-SH in #14818
• Remove writes from the table by @nnsgmsone in #14830
• fix a bug that the runtime filter is not pushed for 1.1 by @badboynt1 in #14835
• [enhancement] proxy: transfer connection more proactively by @volgariver6 in #14858
• fix split command, cherry-pick 14855 by @jiangxinmeng1 in #14862
• [bug] fix ut case by @volgariver6 in #14853
• [1.1-dev] add GetPipelineInfo API to query-service by @aylei in #14847
• [1.1-dev] feat: add from proxy to status session by @aylei in #14848
• cherry-pick to 1.1dev: add character set by @daviszhen in #14874
• sync to 1.1: fix the storage usage obj double delete by @gouhongshen in #14870
• cherry pick "optimize sql statement with limit 0 (#14884)" to 1.1-dev by @badboynt1 in #14889
• cherry-pick to 1.1dev: add rollback last statement by @daviszhen in #14891
• [bug] proxy: fix lost connection issue in tpcc test. by @volgariver6 in #14896
• Add task schedule in 1.1 by @w-zr in #14857
• [enhancement] proxy: migrate prepare stmts when connection is transferred from one cn to another one. by @volgariver6 in #14901
• fix bug: can not show databases after drop database (#14899) by @ouyuanning in #14904
• cherry pick dup by @triump2020 in #14922
• Background query run in selected CN in 1.1 by @xzxiong in #14881
• cherry-pick to 1.1-dev: add foreign key self refer by @daviszhen in #14887
• sync to 1.1: adjust the db, table visibility in show accounts by @gouhongshen in #14924
• Estimate the tcp packet num response in 1.1 by @xzxiong in #14885
• fix insert into enum type with select clause (#1.1-dev) by @YANGGMM in #14921
• CU calculation & metric improvement in 1.1 by @xzxiong in #14948
• support insert into time/datetime/timestamp type with hint (#1.1-dev) by @YANGGMM in #14952
• cherry-pick to 1.1dev: update kinds of abnormal txn behavior by @daviszhen in #14942
• sync to 1.1: counting index table sizes and the original table size together when mo_table_size by @gouhongshen in #14954
• cherrypick to 1.1-dev: add statement latency metric by @EZ4BRUCE in #14966
• return allocate error in auto increment service to 1.1 by @zhangxu19830126 in #14868
• [Improvement] : For auto incr unique key, remove deduplication detection if there will be no conflict by @jensenojs in #14980
• [BugFix]: Upgrade Not getting applied to System Tenant by @arjunsk in #14990
• fix bug: get deleted db when autocommit=0 (#14957) by @ouyuanning in #14977
• fix lock not found or changed in new txn by @iamlinjunhong in #14984
• retry ww conflict in optimistic mode to 1.1 by @zhangxu19830126 in #14994
• cherry pick by @triump2020 in #14992
• bugfix 1.1: account access table in system_metrics db need redirect to sys by @xzxiong in #14998
• fix panic by @zhangxu19830126 in #15023
• improve filter cu by @Morranto in #15003
• [Cherry-pick] fix delete nil routine (#14951) by @ck89119 in #15026
• cherry pick for fix dup-ww-2 by @triump2020 in #15034
• fix json_extract (#1.1-dev) by @YANGGMM in #15035
• fix u...

MatrixOne-v1.1.1

Release date: February 04, 2024
MatrixOne version: v1.1.1

Compared with the previous v1.1.0, v1.1.1 doesn't introduce any new features but focuses on some improvements and bug fixes.

Improvements

• Improved stability by fixing bugs related to lock service and proxy.
• Enhanced forward compatibility.
• Optimized MySQL compatibility.
• Improved the accuracy of runtime resource consumption statistics.

What's Changed

• [cherry-pick] pr #13909 fix some unhandled error and redundant codes by @m-schen in #13910
• fix checkAddNameConflictLocked, cherry-pick #13916 by @jiangxinmeng1 in #13917
• fix mo_ctl by @daviszhen in #13926
• sync_to_1.1: adding a lock to cn storage usage cache to avoid data race by @gouhongshen in #13928
• fileservice: refactor aliyun credentials error handling (#13921) by @reusee in #13930
• CI: change account for sysbench for 1.1-dev by @guguducken in #13935
• cherry-pick 'modify the result of function year()' to 1.1-dev. by @Ariznawlll in #13945
• [postfix] proxy: fix the prepare wait condition from execute to close by @volgariver6 in #13896
• [cherry-pick] #13965 optimized the uuid-process map to avoid memory leaks. by @m-schen in #13966
• [cherry-pick] #13941 Remove an incorrectly timed dropout update that invalidates a live probe by @nnsgmsone in #13981
• fix duration metrics to 1.1 by @zhangxu19830126 in #13992
• fileservice: do not set default role arn from os env by @reusee in #13979
• fix non-reserverd keyword last by @iamlinjunhong in #14014
• fix cmd_field_list (#1.1-dev) by @YANGGMM in #14010
• [bug] disttae: fix a UT in the route package. by @volgariver6 in #14003
• cherry-pick update handle object insert (#14018) by @jiangxinmeng1 in #14022
• fix panic if allocating lock table from dn failed to 1.1 by @zhangxu19830126 in #14039
• sync_to_1.1: defer on time.since it does not work by @gouhongshen in #14029
• Fix bug: marshal query error (#14004) by @ouyuanning in #14036
• remove segdel batch and block meta without deltaloc in ckp, cherry-pick #13952 by @jiangxinmeng1 in #14023
• [cherry-pick] #14034 fix lots of unhandled errors. by @m-schen in #14035
• add more metrics for accepting connection duration by @ck89119 in #14045
• [cherry-pick] #14052 removed a context done error at the remote run stage. by @m-schen in #14053
• Cherry-pick #13268(1.1dev) by @nnsgmsone in #14051
• support cmd_set_option (#1.1-dev) by @YANGGMM in #14058
• fix a bug that shuffles on expression causing wrong result by @badboynt1 in #14060
• fix bug: prepare statement should throw an error when too many parameters… by @ouyuanning in #14037
• [fix] : fix moc2224 and add bvt test by @jensenojs in #14055
• update lazy-load to avoid unnecessary blocking -> 1.1-dev by @XuPeng-SH in #14089
• [enhancement] proxy: change default config value and sql stmt of use. by @volgariver6 in #14075
• Refactor cron tasks on 1.1 by @w-zr in #14103
• fix-cpu-stats by @Morranto in #14105
• fix-maxvalue-panic (#13938) by @Morranto in #14109
• fix bug related to "invalid file path" by @triump2020 in #14143
• fix bug: should throw err when preparing multi stmt (#14112) by @ouyuanning in #14124
• fix invalid deadlock error by @iamlinjunhong in #14153
• cherry pick [Bug]: [date 1.11]tke regression: cn restart, panic error to 1.1-dev by @badboynt1 in #14164
• disable auto create backend automatically in the background by default to 1.1 by @zhangxu19830126 in #14177
• fix lock table map to 1.1 by @zhangxu19830126 in #14202
• fix flush block error handling 1.1 by @aptend in #14215
• pick gc fix to 1.1 by @XuPeng-SH in #14218
• cp to 1.1: fix create account timeout by @daviszhen in #14182
• do not break when fetch cron tasks error by @w-zr in #14200
• fileservice: s3 fs backports by @reusee in #14184
• cherry pick #14221, load fewer CKP entries when replay by @jiangxinmeng1 in #14222
• Pick print more ckp details to 1.1 by @XuPeng-SH in #14188
• use mo_ctl task method to run the task by @w-zr in #14233
• fix race by @Morranto in #14196
• fix bind changed not handled to 1.1 by @zhangxu19830126 in #14243
• update trigger times before writing to async task by @w-zr in #14246
• sync to 1.1: adding RWMutex to avoid concurrent map write by @gouhongshen in #14238
• update table tail column names, cherry-pick #14227 by @jiangxinmeng1 in #14229
• fix mpool leak in LockMoDatabase (#14217) by @ouyuanning in #14249
• sync to 1.1: augment the hakeeperDefaultTimeout in ut to avoid TestHAKeeperCanBootstrapAndRepairShards failed by @gouhongshen in #14253
• fix mpool leak in txn.WriteBatch (#14220) by @ouyuanning in #14250
• TN should return an error, instead of panic! by @triump2020 in #14257
• sync_to_1.1: avoiding load object meta when object stats are valid in mo_table_size and metadata_scan by @gouhongshen in #13949
• fix bug: delete incorrect rows for hidden tables by @ouyuanning in #14261
• Fix mpool leak for remote run by @nnsgmsone in #14264
• remove reserved key word _binary(#1.1-dev) by @YANGGMM in #14262
• fix setvar in the active transaction by @daviszhen in #14258
• cherry-pick #14271 to 1.1-dev by @zengyan1 in #14272
• TableScan should merge a little batch to reduce operator call (#14171) by @ouyuanning in #14251
• Cherry-pick[#14067] Fix the mesa in workspace by @nnsgmsone in #14179
• [Cherry-pick] For small file loads, reduce their parallelism by @nnsgmsone in #14304
• fix can not connect to remote by @iamlinjunhong in #14294
• [Cherry-pick] Fix some bit/hex literal's bug by @ck89119 in #14290
• change gen uuid method to v7 (#1.1-de v) by @YANGGMM in #14318
• Date format improve (#14301) by @qingxinhome in #14323
• fix a bug that may cause wrong query result by @badboynt1 in #14333
• cherry-pick index related commits to 1.1-dev by @aunjgr in #14334
• Remove the dispatch in delete by @nnsgmsone in #14337
• fix access to system_metrics.metric under general account by @daviszhen in #14340
• fix a subscription bug by @jiangxinmeng1 in #14351
• [bug] ut: fix a ut case. by @volgariver6 in #14355
• fix keep remote lock failed to 1.1 by @zhangxu19830126 in #14357
• clear table cache if retry to 1.1 by @zhangxu19830126 in #14366
• fix bug by @triump2020 in #14368
• GC Adapt Object by @LeftHandCold in #14361
• fix a bug that cause funtion convert_tz bad performance by @badboynt...

MatrixOne-v1.1.0

We are excited to announce MatrixOne 1.1.0 release!

MatrixOne is a hyper-converged cloud-native database. It is designed to provide a cloud-native, high-performance, highly scalable, MySQL-compatible HTAP database. MatrixOne enables users to handle mixed workloads such as transactions, analytics, time-series, and streaming processing through a one-stop data processing solution.

What's New in v1.1.0?

Vector Data Type

These features enable users to quickly build AI applications, such as RAG applications based on large language models (LLMs). Unlike specialized vector databases, MatrixOne is a general database highly compatible with MySQL, enabling users to get started quickly without additional learning barriers. It also integrates structured and vector data processing for building AI applications.

• Implemented vecf32 (float32) and vecf64 (float64) type.
• Support for basic binary operators: +,-,*,/.
• Support for comparison operators: =, !=, >, >=, <, <=.
• Support for unary functions: sqrt,abs,cast.
• Support for vector functions: summation,l1_norm,l2_norm,vector_dims,inner_product,cosine_similarity.
• Support for aggregate function: count.

Time Series

• Support for streaming loading with LOAD INTO INLINE, surpassing INSERT INTO in performance.
• Support for time-series tables with timestamps as primary keys, and support for any dimension/metric columns.
• Support sliding window for downsampling queries over different time periods.
• Support for interpolation with various interpolation methods.

Kafka Connector(beta)

• Support for creating dynamic and append-only table with CREATE DYNAMIC TABLE.
• Support for configuring external data sources with CREATE SOURCE.
• Support for Kafka topic integration with JSON or protobuf format.

User Defined Functions(beta)

• Support for creating Python-based UDF

Other New Features

DDL Statements

• Support for insert on duplicate key ignore.
• Support for create or replace view.
• Support for alter sequence.
• Support for Key, hash partition pruning capabilities (beta).
• Support for List/List column, Range/Range Columns partition storage capabilities (beta).

Indexes and Constraints

• Full support of secondary indexes for dynamic query acceleration.

Built-in Functions

• Added SAMPLE sampling function.
• Added CONVERT_TZ time zone conversion function.
• Added SHA2 encryption function.
• Added Encode/Decode encoding and decoding functions.

Security

• Support for managing path permissions for select into through creating Stage.

Tools

• mo_dump tool (logical backup)
  • Starting from this iteration, modump is managed in a separate repo (https://github.com/matrixorigin/mo_dump).
  • Supports exporting DDLs separately.
  • Supports exporting multiple databases and tables.
• mo_backup tool (physical backup)
  • Supports file systems and object storage as storage media for backup and restoration.
• mo_ctl_standalone Tool
  • Supports automatic data backup.
  • Supports automatic log table data cleaning.
  • Supports converting data files from CSV format to insert or load data inline format.
  • Supports automatic docker image building.
  • Supports docker mode for standalone deployment.
• mo_ctl_distributed Tool
  • Supports one-click installation and uninstallation of distributed clusters.
  • Supports start/stop, upgrade/rollback operations for matrixone clusters.
  • Supports installing matrixone clusters in different k8s clusters.
• mo_operator tool
  • Supports configuring custom S3 certificates.
  • Supports backup and recovery, and management of backup data for matrixone clusters via API.
  • Supports setting optimized Go GC strategies automatically for matrixone clusters.
  • Supports enabling Python UDF for matrixone clusters.
  • Supports integration of matrixone on Kubernetes with Prometheus.

MySQL Compatibility

• Remove hundreds of MySQL-incompatible reserved keywords.

Known Issues

• Memory leak occasionally happens and may lead to an OOM error.
• Occasional system hung under high concurrency workload.
• Kafka connector works only in a standlone deployment.
• Secondary Index doesn't apply for IN queries.

New Contributors

• @aronchanisme made their first contribution in #11424
• @xmh1011 made their first contribution in #12112
• @orangekame3 made their first contribution in #12330
• @joker-star-l made their first contribution in #11098
• @lcxznpy made their first contribution in #13810

Full Changelog: v1.0.0...v1.1.0

MatrixOne-v1.0.2

Release date: December 23, 2023
MatrixOne version: v1.0.2
Compared with the previous v1.0.1, v1.0.2 introduces a few features/improvements but the focus was primarily on bug fixes.

Improvements

• Support transitivity of equalities: #13596

Bugfix

• Fix some reserved keywords bugs: #13561
• Fix PrefecthDelete bug: #13590
• Fix blob related bug: #13593
• Fix mo_ctl merge hung: #13601
• Fix the issue of incorrectly pushing down runtime filters during left join: #13661
• Fix a bug where the package partition never considers the constant vector: #13607
• Ignore invalid symlink in local fs list: #13675
• Fixed the issue of an abnormal interruption during flush: #13696, #13706, #13708
• Fixed the issue where executing SHOW SUBSCRIPTIONS; fails in certain special scenarios: #13656, #13599
• Fix the bug where DirtyBlocks removes the wrong block: #13647

Features

• Support lcase, ucase: #13686

MatrixOne-v1.0.1

Release date: December 16, 2023

MatrixOne version: v1.0.1

Compared with the previous v1.0.0, v1.0.1 introduces a few features but the focus was primarily on improvements and bug fixes.

Improvements

• Refactor the collection of account storage usage information #12468
• Improve lockservice p99 latency performance #12760
• Enhance system observability #12756, #12767, #12801, #13282, #13388
• Add some debug commands #12772, #12804, #12901
• Improve cu stats #12925

Bugfix

• Fixed some MySQL compatibility issues #12753, #13330
• Resolved an issue with incorrect calculation of FileService cache size #12729
• Fixed some system hung issues #12777, #13004, #13227
• Modified prefetch strategy to reduce memory usage #12827
• Corrected create/drop account issues #12841, #12863, #13130, #13343
• Resolved show processlist issue #12834
• Fixed the issue of a large number of SQL queries accumulating #12899, #12923
• Fixed some rpc bugs #12884
• Fixed go routine leak #13312
• Fixed some lock bugs #13082, #13346
• Fixed the continuous FATAL issue in CN #13047
• Fixed an error in the maximum data packet length calculation #13112
• Fixed incorrect routing issue #13103
• Fixed unexcepted ckp stopped bug #13308
• Resolved the issue of abnormal display of transaction DDL execution results #13440
• Fixed blob type encoding bug #13507
• Fixed non-reserved keyword collation #13501

Features

• Support lower(), upper(), locate() #13442

MatrixOne-v1.0.0

Release date: November 13, 2023

MatrixOne version: v1.0.0

Compared with the previous v1.0.0-RC2, v1.0.0 doesn't introduce any new features but focuses on some improvements and bug fixes.

Improvements

• Storage Optimization: #12465 #12676
• More metrics added: #12302 #12313 #12511
• Update Object Cache policy: #12699 #12661

Bugfix

• Fix aliyun sdk related bugs: #12296 #12695
• Fix some operator related bugs: #12360 #12611
• Fix flush panic: #12342
• Fix system table bugs: #12372 #12689
• Fix GC related bugs: #12487

MatrixOne-v1.0.0-RC2

Release date: October 24, 2023

MatrixOne version: v1.0.0-RC2

Compared with the previous v1.0.0-RC1, v1.0.0-RC2 doesn't introduce any new features but focuses on some improvements and bug fixes.

Improvements

• Improved stability #11384, #11416, #12134, #12145, #12217, #12289 and etc.
• Improved system observability #11607, #11240, #12237, #12276 and etc.
• Enhanced and improved Merge strategy and stability #11694, #12210.
• Optimized performance and stability of delete operations #11541, #11542, #11882, #11969.
• Optimized memory usage in some scenarios #11533, #12178 and etc.
• Enhanced performance and algorithms for some aggregate queries #11543, #11841, #11843, #11910 and etc.
• Optimized performance and stability of locks #11649, #12077 and etc.
• Optimized decimal data type operation #11617, #11606, #11820.
• Enhanced the functionality of prepared statements #11616.
• Enhanced the functionality of on duplicate key #11632, #11760.
• Improved recursive Common Table Expression (CTE) capabilities #11508, #11564, #11614.
• Optimized performance and resource consumption in metrics/log/trace collection #11540, #11648, #12061, and #12115.
• Optimized the filtering performance for composite primary keys #11626.
• Enhanced write performance for large datasets #11633.
• Added support for Aliyun OSS in fileservice #11660, #11715, #11880, #11942, #12037.
• Improved capabilities of mo_ctl #11680, #12247.
• Further enhancements to partition related features #11762.
• Enhanced and improved runtime filter #11868.
• Optimized the local testing framework #11904.
• Optimized performance for count/min/max operations #11959.
• Improved compatibility with MySQL DDL #11482, #11557, #11589 and etc.

Bugfix

• Fixed checkpoint related bugs #12014, #12124, #12172 and etc.
• Fixed alter table/db related bugs #11429, #11484, #12162 and etc.
• Fixed DDL related bugs #11842, #11770, #11789, #12188 and etc.
• Fixed GC-related issues #11689.
• Fixed taskservice related bugs #11544.
• Fixed window function-related bugs #11461.
• Fixed high-concurrency conflict issues #11759.
• Fixed show backend servers incorrect results #11414.
• Fixed insert related bugs #11495 and etc.
• Fixed deduplication-related bugs #11895 and etc.
• Fixed foreign key-related issues #11560, #11586 and etc.
• Fixed data race problems in the systems, #11571, #11588, and etc.
• Fixed bloomfilter bug #11914 and etc.
• Fixed various functional bugs #11668, #12270,#11504 and etc.
• Fixed startup issues in the system #11756, #12055, #12160 and etc.