Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [fix](index size) discard index size when meta size is invalid #46549 #46717

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

airborne12
Copy link
Member

cherry pick from #46549

…e#46549)

Issue Number: close apache#45817

Related PR: #xxx

Problem Summary:
Try to discard and get real inverted index size when rowset meta index
size is invalid.

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@airborne12
Copy link
Member Author

run buildall

@Thearas
Copy link
Contributor

Thearas commented Jan 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

TPC-H: Total hot run time: 41545 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6cd72d04c2ae8476430622e563d1e1c8e93aaaea, data reload: false

------ Round 1 ----------------------------------
q1	17974	7521	7408	7408
q2	2635	170	169	169
q3	11338	1180	1231	1180
q4	10600	760	747	747
q5	8363	3000	2872	2872
q6	242	154	154	154
q7	995	623	614	614
q8	9719	2004	2048	2004
q9	6703	6429	6447	6429
q10	6986	2280	2314	2280
q11	453	256	265	256
q12	413	217	213	213
q13	17789	2993	2993	2993
q14	238	205	203	203
q15	567	513	516	513
q16	670	600	620	600
q17	996	607	613	607
q18	7234	6723	6739	6723
q19	1401	1112	1071	1071
q20	471	207	207	207
q21	4012	3293	3353	3293
q22	1114	1018	1009	1009
Total cold run time: 110913 ms
Total hot run time: 41545 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7302	7238	7249	7238
q2	345	235	250	235
q3	3149	2988	2984	2984
q4	2042	1815	1870	1815
q5	5698	5759	5722	5722
q6	223	139	141	139
q7	2265	1824	1749	1749
q8	3362	3565	3521	3521
q9	8885	8957	8846	8846
q10	3613	3610	3546	3546
q11	606	491	513	491
q12	796	579	617	579
q13	6403	3198	3039	3039
q14	306	268	252	252
q15	564	503	495	495
q16	709	664	664	664
q17	1820	1577	1560	1560
q18	7899	7533	7473	7473
q19	1674	1516	1378	1378
q20	2058	1788	1795	1788
q21	5623	5138	5130	5130
q22	1108	1005	1017	1005
Total cold run time: 66450 ms
Total hot run time: 59649 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190919 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6cd72d04c2ae8476430622e563d1e1c8e93aaaea, data reload: false

query1	970	366	369	366
query2	6509	2128	2076	2076
query3	6719	212	217	212
query4	34085	23663	23435	23435
query5	4335	453	424	424
query6	275	169	179	169
query7	4622	311	308	308
query8	277	225	234	225
query9	9378	2674	2675	2674
query10	460	266	259	259
query11	18316	15087	15079	15079
query12	156	104	104	104
query13	1635	473	429	429
query14	8957	6691	6951	6691
query15	262	175	172	172
query16	8093	481	492	481
query17	1631	567	564	564
query18	2135	308	302	302
query19	244	157	150	150
query20	114	109	107	107
query21	214	101	103	101
query22	4477	4163	4363	4163
query23	34625	34223	34013	34013
query24	12018	2853	2840	2840
query25	765	405	411	405
query26	1892	174	172	172
query27	2967	353	341	341
query28	8088	2425	2423	2423
query29	1105	462	435	435
query30	342	164	168	164
query31	1043	797	809	797
query32	99	57	61	57
query33	799	310	307	307
query34	938	504	526	504
query35	861	718	722	718
query36	1108	966	943	943
query37	272	79	73	73
query38	3978	3870	3837	3837
query39	1498	1444	1431	1431
query40	293	102	101	101
query41	54	52	50	50
query42	110	99	102	99
query43	548	507	496	496
query44	1319	816	819	816
query45	190	170	181	170
query46	1123	737	714	714
query47	1955	1806	1820	1806
query48	464	376	427	376
query49	1302	412	392	392
query50	831	412	411	411
query51	7220	7021	6953	6953
query52	110	92	92	92
query53	263	184	187	184
query54	1343	467	470	467
query55	84	88	79	79
query56	290	255	249	249
query57	1229	1105	1112	1105
query58	251	224	213	213
query59	3148	2989	2965	2965
query60	303	255	242	242
query61	123	108	106	106
query62	881	681	750	681
query63	227	189	183	183
query64	5179	672	655	655
query65	3250	3210	3196	3196
query66	1423	322	313	313
query67	16059	15696	15444	15444
query68	4703	595	573	573
query69	444	274	278	274
query70	1206	1130	1101	1101
query71	412	258	262	258
query72	6329	4034	4007	4007
query73	769	346	348	346
query74	9823	9052	8924	8924
query75	3356	2604	2638	2604
query76	3012	1005	1101	1005
query77	381	278	271	271
query78	10538	9648	9607	9607
query79	1493	600	604	600
query80	1155	426	415	415
query81	548	240	236	236
query82	958	125	113	113
query83	226	152	143	143
query84	240	84	82	82
query85	1281	299	286	286
query86	374	300	302	300
query87	4374	4254	4299	4254
query88	3427	2393	2354	2354
query89	397	293	297	293
query90	1947	193	188	188
query91	190	151	155	151
query92	59	51	51	51
query93	1234	563	546	546
query94	898	299	265	265
query95	363	253	271	253
query96	608	286	282	282
query97	3372	3198	3132	3132
query98	210	194	195	194
query99	1535	1302	1302	1302
Total cold run time: 301696 ms
Total hot run time: 190919 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6cd72d04c2ae8476430622e563d1e1c8e93aaaea, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.63	0.10	0.10
query5	0.53	0.50	0.52
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.05	0.03
query9	0.56	0.49	0.49
query10	0.55	0.56	0.56
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.59	0.59
query14	2.93	3.02	2.93
query15	0.90	0.82	0.83
query16	0.37	0.41	0.39
query17	1.05	1.04	0.99
query18	0.23	0.22	0.21
query19	1.88	1.89	2.02
query20	0.01	0.01	0.01
query21	15.37	0.58	0.58
query22	2.77	2.84	1.63
query23	17.02	0.96	0.83
query24	2.69	1.36	0.99
query25	0.15	0.22	0.19
query26	0.46	0.15	0.14
query27	0.04	0.04	0.05
query28	10.68	1.09	1.06
query29	12.53	3.35	3.38
query30	0.25	0.06	0.06
query31	2.84	0.39	0.38
query32	3.25	0.46	0.47
query33	2.98	3.00	3.01
query34	16.95	4.48	4.46
query35	4.52	4.46	4.55
query36	0.65	0.51	0.49
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.68 s
Total hot run time: 33.02 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants