Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Optimize](Row store) allow to set row_store_page_size for tables, change default value to 16KB #37145

Merged
merged 26 commits into from
Jul 16, 2024

Conversation

lxr599
Copy link
Contributor

@lxr599 lxr599 commented Jul 2, 2024

Proposed changes

In current row storage format, under different page sizes, the compression effect of zstd on the hits data set (70 GB) is as follows.

no_compression zstd (default, page size=4KB) zstd ( page size=16KB) zstd ( page size=32KB) zstd ( page size=64KB)
105.76 GB 33.38 GB 22.07 GB 21.83 GB 18.29 GB

When the page size is 16KB, the optimization is the greatest (33.89%). When the page size is increased, the effect is not obvious.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon
Copy link
Member

run buildall

@eldenmoon eldenmoon changed the title change default value of row_column_page_size to 16KB Jul 2, 2024
eldenmoon
eldenmoon previously approved these changes Jul 2, 2024
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

github-actions bot commented Jul 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39399 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ec59d28e898d50455ab8fb919aead8d4cff9c869, data reload: false

------ Round 1 ----------------------------------
q1	17633	4345	4214	4214
q2	2007	184	180	180
q3	10472	1201	1076	1076
q4	10185	875	832	832
q5	7461	2628	2635	2628
q6	215	138	134	134
q7	944	573	588	573
q8	9221	2052	2031	2031
q9	9109	6477	6411	6411
q10	8979	3691	3715	3691
q11	468	226	231	226
q12	403	226	229	226
q13	17782	2939	2970	2939
q14	256	218	241	218
q15	526	502	481	481
q16	534	398	373	373
q17	958	609	713	609
q18	8031	7397	7338	7338
q19	7089	1472	1433	1433
q20	664	322	313	313
q21	4833	3146	3915	3146
q22	389	336	327	327
Total cold run time: 118159 ms
Total hot run time: 39399 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4381	4241	4233	4233
q2	364	267	267	267
q3	2997	2837	2891	2837
q4	1979	1693	1704	1693
q5	5618	5482	5451	5451
q6	222	137	136	136
q7	2232	1888	1880	1880
q8	3225	3399	3402	3399
q9	8677	8718	8769	8718
q10	4151	3731	3736	3731
q11	595	493	499	493
q12	818	649	660	649
q13	17065	3107	3230	3107
q14	292	278	262	262
q15	529	477	485	477
q16	492	419	440	419
q17	1812	1532	1502	1502
q18	8181	7822	7880	7822
q19	1823	1703	1624	1624
q20	2186	1889	1840	1840
q21	8592	4760	4684	4684
q22	626	546	541	541
Total cold run time: 76857 ms
Total hot run time: 55765 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 173949 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec59d28e898d50455ab8fb919aead8d4cff9c869, data reload: false

query1	904	378	373	373
query2	6438	2468	2411	2411
query3	6627	205	213	205
query4	21485	17512	17326	17326
query5	3730	466	467	466
query6	256	170	166	166
query7	4599	287	293	287
query8	315	289	285	285
query9	8377	2356	2334	2334
query10	585	326	276	276
query11	10644	9972	10178	9972
query12	117	90	82	82
query13	1646	358	352	352
query14	9663	7778	7801	7778
query15	243	180	187	180
query16	8022	268	277	268
query17	1891	543	529	529
query18	2091	281	275	275
query19	197	158	153	153
query20	90	82	83	82
query21	215	132	124	124
query22	4714	4189	4160	4160
query23	33735	33535	33442	33442
query24	11069	2849	2904	2849
query25	599	403	386	386
query26	703	157	156	156
query27	2224	322	331	322
query28	6124	2105	2088	2088
query29	912	649	644	644
query30	250	159	159	159
query31	985	787	757	757
query32	101	57	55	55
query33	772	303	311	303
query34	1022	478	481	478
query35	758	660	638	638
query36	1165	993	968	968
query37	140	82	80	80
query38	2984	2852	2807	2807
query39	887	811	804	804
query40	211	128	124	124
query41	58	52	55	52
query42	113	100	97	97
query43	599	552	570	552
query44	1192	731	711	711
query45	198	168	166	166
query46	1084	737	706	706
query47	1852	1758	1776	1758
query48	377	299	297	297
query49	851	505	404	404
query50	763	378	374	374
query51	6843	6754	6697	6697
query52	102	93	87	87
query53	356	288	289	288
query54	930	421	425	421
query55	71	82	70	70
query56	268	258	261	258
query57	1107	1088	1062	1062
query58	234	240	236	236
query59	3463	3154	3058	3058
query60	295	265	266	265
query61	111	87	87	87
query62	593	447	447	447
query63	320	292	291	291
query64	8491	2259	1708	1708
query65	3125	3193	3074	3074
query66	754	321	343	321
query67	15724	14941	14906	14906
query68	9013	540	535	535
query69	716	388	342	342
query70	1361	1154	1083	1083
query71	528	315	281	281
query72	9503	5390	5570	5390
query73	2224	320	317	317
query74	5823	5445	5447	5445
query75	6039	2616	2651	2616
query76	5261	901	870	870
query77	785	302	300	300
query78	10548	10005	9788	9788
query79	7824	508	504	504
query80	928	475	500	475
query81	549	221	228	221
query82	262	104	103	103
query83	323	169	162	162
query84	280	84	84	84
query85	888	267	268	267
query86	349	299	300	299
query87	3302	3080	3072	3072
query88	4386	2361	2344	2344
query89	492	411	374	374
query90	2074	187	178	178
query91	124	98	98	98
query92	67	48	47	47
query93	5332	501	493	493
query94	1332	183	183	183
query95	400	311	308	308
query96	616	262	257	257
query97	3163	3003	2966	2966
query98	207	223	197	197
query99	1244	846	813	813
Total cold run time: 293307 ms
Total hot run time: 173949 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ec59d28e898d50455ab8fb919aead8d4cff9c869, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.22	0.05	0.06
query4	1.69	0.08	0.08
query5	0.50	0.49	0.47
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.53	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.58	0.58	0.58
query14	0.78	0.76	0.79
query15	0.84	0.81	0.80
query16	0.35	0.35	0.36
query17	0.95	0.98	0.97
query18	0.20	0.27	0.26
query19	1.79	1.74	1.63
query20	0.01	0.01	0.01
query21	15.43	0.77	0.67
query22	4.61	6.37	2.08
query23	18.31	1.38	1.20
query24	2.17	0.23	0.23
query25	0.14	0.08	0.08
query26	0.28	0.18	0.18
query27	0.08	0.08	0.08
query28	13.22	1.01	0.99
query29	12.67	3.30	3.27
query30	0.25	0.06	0.06
query31	2.88	0.38	0.39
query32	3.29	0.48	0.46
query33	2.86	2.89	2.84
query34	16.78	4.41	4.42
query35	4.50	4.48	4.45
query36	0.63	0.45	0.46
query37	0.18	0.16	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.14	0.16
query41	0.08	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.44 s
Total hot run time: 30.49 s
@eldenmoon
Copy link
Member

change row_store_page_size to table property

Copy link
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@lxr599
Copy link
Contributor Author

lxr599 commented Jul 15, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40002 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5c543f73bd880e49534b2c7684695dee358ec1c1, data reload: false

------ Round 1 ----------------------------------
q1	17842	4547	4348	4348
q2	2017	193	187	187
q3	10548	1241	1052	1052
q4	10256	836	772	772
q5	7581	2733	2718	2718
q6	225	135	139	135
q7	962	608	597	597
q8	9233	2107	2118	2107
q9	8619	6564	6587	6564
q10	8823	3808	3778	3778
q11	454	235	239	235
q12	386	218	222	218
q13	18723	2994	2987	2987
q14	282	235	247	235
q15	523	496	487	487
q16	519	376	382	376
q17	983	653	679	653
q18	8190	7448	7364	7364
q19	6417	1429	1374	1374
q20	642	327	329	327
q21	4939	3203	3289	3203
q22	358	293	285	285
Total cold run time: 118522 ms
Total hot run time: 40002 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4543	4336	4394	4336
q2	387	271	278	271
q3	3119	2982	3055	2982
q4	2012	1774	1786	1774
q5	5702	5581	5458	5458
q6	237	132	143	132
q7	2253	1861	1830	1830
q8	3228	3473	3395	3395
q9	8862	8945	8828	8828
q10	4195	3705	3828	3705
q11	605	518	516	516
q12	805	652	601	601
q13	17184	3106	3145	3106
q14	328	276	289	276
q15	530	470	491	470
q16	483	430	451	430
q17	1779	1503	1495	1495
q18	8169	7950	7832	7832
q19	1784	1488	1635	1488
q20	2262	1878	1855	1855
q21	8368	4706	4650	4650
q22	623	497	515	497
Total cold run time: 77458 ms
Total hot run time: 55927 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 173520 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5c543f73bd880e49534b2c7684695dee358ec1c1, data reload: false

query1	921	371	366	366
query2	6446	1937	1836	1836
query3	6634	203	216	203
query4	28252	17530	17319	17319
query5	3616	494	488	488
query6	256	175	180	175
query7	4554	297	299	297
query8	249	195	194	194
query9	8276	2487	2472	2472
query10	423	299	274	274
query11	10790	10107	10070	10070
query12	116	88	85	85
query13	1610	377	376	376
query14	10347	7726	7686	7686
query15	222	165	167	165
query16	7563	326	310	310
query17	1810	539	527	527
query18	1783	280	270	270
query19	196	163	154	154
query20	96	82	86	82
query21	204	131	126	126
query22	4275	3978	4093	3978
query23	33834	33546	33539	33539
query24	9524	3003	3016	3003
query25	599	385	396	385
query26	699	153	150	150
query27	2306	276	294	276
query28	5903	2081	2092	2081
query29	857	662	623	623
query30	256	155	151	151
query31	999	765	728	728
query32	93	52	53	52
query33	674	304	289	289
query34	884	492	501	492
query35	684	576	589	576
query36	1106	1030	957	957
query37	150	97	81	81
query38	2958	2863	2846	2846
query39	914	819	802	802
query40	206	117	119	117
query41	50	46	44	44
query42	112	95	97	95
query43	489	478	452	452
query44	1089	718	725	718
query45	190	159	165	159
query46	1075	733	734	733
query47	1863	1766	1763	1763
query48	374	282	289	282
query49	821	405	443	405
query50	776	394	402	394
query51	6961	6730	6924	6730
query52	107	91	94	91
query53	364	294	298	294
query54	831	464	449	449
query55	75	73	73	73
query56	281	267	262	262
query57	1129	1036	1047	1036
query58	239	246	279	246
query59	2905	2784	2549	2549
query60	305	291	270	270
query61	95	92	91	91
query62	789	659	652	652
query63	313	292	287	287
query64	9131	2249	1648	1648
query65	3153	3098	3105	3098
query66	739	321	334	321
query67	15657	15010	15051	15010
query68	4416	532	543	532
query69	478	314	339	314
query70	1172	1146	1141	1141
query71	387	280	277	277
query72	6922	6177	5429	5429
query73	740	331	326	326
query74	6232	5660	5660	5660
query75	3394	2673	2668	2668
query76	2090	898	977	898
query77	436	304	310	304
query78	9825	9290	8993	8993
query79	2659	521	523	521
query80	2131	472	477	472
query81	588	217	217	217
query82	942	132	133	132
query83	275	171	171	171
query84	275	88	87	87
query85	1766	304	299	299
query86	461	328	303	303
query87	3324	3074	3076	3074
query88	4001	2466	2458	2458
query89	478	385	399	385
query90	1730	192	198	192
query91	127	100	97	97
query92	61	50	54	50
query93	3024	506	500	500
query94	1106	214	212	212
query95	404	320	313	313
query96	613	279	284	279
query97	3207	3107	3029	3029
query98	230	200	199	199
query99	1559	1235	1266	1235
Total cold run time: 276251 ms
Total hot run time: 173520 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 31.22 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5c543f73bd880e49534b2c7684695dee358ec1c1, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.67	0.10	0.10
query5	0.49	0.49	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.50	0.49
query10	0.54	0.53	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.74	0.80	0.76
query15	0.85	0.81	0.82
query16	0.37	0.37	0.38
query17	1.03	1.05	1.02
query18	0.22	0.21	0.22
query19	1.90	1.76	1.83
query20	0.01	0.01	0.01
query21	15.45	0.73	0.65
query22	4.44	6.38	2.55
query23	18.38	1.46	1.26
query24	2.07	0.23	0.22
query25	0.14	0.08	0.09
query26	0.28	0.21	0.21
query27	0.45	0.23	0.23
query28	13.34	1.00	0.97
query29	12.54	3.24	3.28
query30	0.26	0.06	0.06
query31	2.85	0.39	0.39
query32	3.28	0.48	0.47
query33	2.91	2.88	3.00
query34	17.26	4.32	4.41
query35	4.44	4.41	4.40
query36	0.66	0.46	0.49
query37	0.18	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.03	0.04
Total cold run time: 110.25 s
Total hot run time: 31.22 s
@lxr599
Copy link
Contributor Author

lxr599 commented Jul 16, 2024

run external

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 16, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@eldenmoon
Copy link
Member

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39905 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 617dafd5ec4a9b3ce1bdd6dffc65ad2c54189725, data reload: false

------ Round 1 ----------------------------------
q1	18780	8571	4392	4392
q2	2718	197	191	191
q3	11207	1216	1020	1020
q4	11057	859	804	804
q5	7574	2747	2700	2700
q6	234	143	138	138
q7	969	612	607	607
q8	9578	2062	2075	2062
q9	8780	6595	6573	6573
q10	8787	3776	3720	3720
q11	445	243	245	243
q12	396	223	224	223
q13	18984	2931	2981	2931
q14	270	236	228	228
q15	535	478	488	478
q16	479	394	384	384
q17	970	636	683	636
q18	8185	7554	7533	7533
q19	6891	1429	1273	1273
q20	676	338	332	332
q21	4903	3175	3154	3154
q22	356	286	283	283
Total cold run time: 122774 ms
Total hot run time: 39905 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4253	4237	4237
q2	366	264	272	264
q3	2991	2724	2791	2724
q4	1869	1626	1586	1586
q5	5332	5296	5323	5296
q6	220	129	129	129
q7	2125	1669	1758	1669
q8	3200	3328	3310	3310
q9	8413	8398	8413	8398
q10	3919	3713	3679	3679
q11	584	495	482	482
q12	766	598	608	598
q13	16653	3018	2952	2952
q14	327	271	274	271
q15	517	473	483	473
q16	465	411	437	411
q17	1767	1494	1498	1494
q18	7656	7469	7318	7318
q19	1693	1475	1379	1379
q20	2041	1785	1787	1785
q21	4910	4614	4525	4525
q22	600	472	489	472
Total cold run time: 70777 ms
Total hot run time: 53452 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 172990 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 617dafd5ec4a9b3ce1bdd6dffc65ad2c54189725, data reload: false

query1	921	362	367	362
query2	6493	1968	1856	1856
query3	6664	203	215	203
query4	28357	17578	17450	17450
query5	4197	500	481	481
query6	273	161	161	161
query7	4594	290	286	286
query8	237	190	184	184
query9	8550	2469	2445	2445
query10	431	290	267	267
query11	10924	10238	10227	10227
query12	139	79	80	79
query13	1627	378	365	365
query14	10123	7813	7669	7669
query15	216	168	167	167
query16	7548	324	319	319
query17	1787	548	540	540
query18	1781	271	266	266
query19	195	148	144	144
query20	92	81	87	81
query21	204	126	130	126
query22	4403	4074	4124	4074
query23	33786	33082	32888	32888
query24	12007	2883	2801	2801
query25	650	367	373	367
query26	1771	148	146	146
query27	2803	275	274	274
query28	7394	2023	2013	2013
query29	1046	658	612	612
query30	292	152	146	146
query31	954	733	765	733
query32	96	54	57	54
query33	782	311	319	311
query34	900	484	487	484
query35	656	596	585	585
query36	1067	926	976	926
query37	173	81	79	79
query38	2884	2743	2760	2743
query39	858	794	818	794
query40	281	120	116	116
query41	50	45	45	45
query42	114	95	99	95
query43	507	478	488	478
query44	1210	739	723	723
query45	189	163	162	162
query46	1079	710	725	710
query47	1870	1813	1757	1757
query48	377	296	297	296
query49	1192	412	409	409
query50	793	396	393	393
query51	6988	6795	6733	6733
query52	105	90	95	90
query53	353	287	284	284
query54	1008	445	492	445
query55	79	76	73	73
query56	280	253	260	253
query57	1158	1017	1059	1017
query58	251	251	243	243
query59	2955	2805	2719	2719
query60	309	275	282	275
query61	93	88	88	88
query62	837	653	668	653
query63	313	296	292	292
query64	10567	2222	1676	1676
query65	3175	3126	3106	3106
query66	1379	332	327	327
query67	15548	15040	15265	15040
query68	4599	537	540	537
query69	472	329	323	323
query70	1174	1095	1090	1090
query71	400	272	262	262
query72	6906	5408	5409	5408
query73	746	327	323	323
query74	6147	5680	5715	5680
query75	3414	2703	2657	2657
query76	2652	954	929	929
query77	444	309	300	300
query78	9575	9002	8962	8962
query79	2520	533	534	533
query80	1914	465	472	465
query81	597	219	231	219
query82	955	136	137	136
query83	287	161	176	161
query84	268	89	86	86
query85	1284	313	299	299
query86	405	332	302	302
query87	3287	3160	3168	3160
query88	3968	2483	2511	2483
query89	478	385	381	381
query90	1785	261	188	188
query91	127	102	100	100
query92	61	48	51	48
query93	1767	497	495	495
query94	1182	214	215	214
query95	397	315	313	313
query96	596	273	278	273
query97	3245	3036	3009	3009
query98	213	197	189	189
query99	1602	1259	1285	1259
Total cold run time: 284402 ms
Total hot run time: 172990 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 31.26 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 617dafd5ec4a9b3ce1bdd6dffc65ad2c54189725, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.10
query5	0.50	0.49	0.49
query6	1.22	0.72	0.74
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.48	0.49
query10	0.54	0.55	0.55
query11	0.17	0.12	0.12
query12	0.14	0.13	0.13
query13	0.60	0.58	0.58
query14	0.76	0.78	0.77
query15	0.84	0.80	0.81
query16	0.38	0.36	0.36
query17	0.96	1.02	1.06
query18	0.22	0.21	0.21
query19	1.90	1.83	1.79
query20	0.01	0.01	0.02
query21	15.39	0.75	0.67
query22	3.96	6.53	2.60
query23	18.30	1.39	1.25
query24	2.15	0.22	0.21
query25	0.17	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.24	0.22
query28	13.33	1.01	1.01
query29	12.62	3.28	3.25
query30	0.25	0.06	0.06
query31	2.90	0.38	0.39
query32	3.23	0.48	0.47
query33	2.87	2.94	2.84
query34	17.08	4.30	4.38
query35	4.43	4.45	4.39
query36	0.65	0.46	0.47
query37	0.19	0.15	0.16
query38	0.15	0.15	0.17
query39	0.04	0.03	0.03
query40	0.15	0.11	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.67 s
Total hot run time: 31.26 s
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang requested review from xiaokang and removed request for xiaokang July 16, 2024 06:16
@eldenmoon eldenmoon merged commit 5c34742 into apache:master Jul 16, 2024
28 of 31 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
…ange default value to 16KB (apache#37145)

In current row storage format, under different page sizes, the
compression effect of zstd on the hits data set (70 GB) is as follows.
| no_compression| zstd (default, page size=4KB) |zstd ( page size=16KB)
|zstd ( page size=32KB) |zstd ( page size=64KB) |
|:--------:| :---------:|:--------:|:--------:|:--------:|
| 105.76 GB |33.38 GB |22.07 GB|21.83 GB|18.29 GB|

When the page size is 16KB, the optimization is the greatest (33.89%).
When the page size is increased, the effect is not obvious.
eldenmoon pushed a commit that referenced this pull request Jul 17, 2024
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.1-merged doing meta-change reviewed
5 participants