Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](fe) Add ignore_bdbje_log_checksum_read for BDBEnvironment (#31247) #37614

Merged
merged 1 commit into from
Jul 10, 2024
Merged

[feat](fe) Add ignore_bdbje_log_checksum_read for BDBEnvironment (#31247) #37614

merged 1 commit into from
Jul 10, 2024

Conversation

SWJTU-ZhangLei
Copy link
Contributor

@SWJTU-ZhangLei SWJTU-ZhangLei commented Jul 10, 2024 •
edited by dataroaring
Loading

Proposed changes

pick #31247

Issue Number: close #xxx
@SWJTU-ZhangLei @dataroaring
[feat](fe) Add ignore_bdbje_log_checksum_read for BDBEnvironment (a…
6398c4f 
…pache#31247)

* https://forums.oracle.com/ords/apexds/post/je-log-checksumexception-2812

* When meeting disk damage or other reason described in the oracle forums
  and fe cannot start due to `com.sleepycat.je.log.ChecksumException`, we
  add a param `ignore_bdbje_log_checksum_read` to ignore the exception, but
  there is no guarantee of correctness for bdbje kv data

Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.
@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall
@doris-robot
Copy link

TPC-H: Total hot run time: 50020 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6398c4fde2a86e616d15fba845a0c8f6f059132c, data reload: false

------ Round 1 ----------------------------------
q1	17831	4422	4364	4364
q2	2033	158	146	146
q3	10270	1891	1948	1891
q4	10146	1299	1326	1299
q5	8458	3928	3919	3919
q6	235	126	126	126
q7	2044	1635	1601	1601
q8	9317	2755	2731	2731
q9	10979	10275	10332	10275
q10	8639	3547	3480	3480
q11	435	248	253	248
q12	471	308	309	308
q13	18349	3950	4033	3950
q14	348	338	338	338
q15	511	463	463	463
q16	680	581	579	579
q17	1143	982	973	973
q18	7309	6878	7023	6878
q19	1791	1651	1640	1640
q20	547	303	300	300
q21	4450	4139	4059	4059
q22	532	454	452	452
Total cold run time: 116518 ms
Total hot run time: 50020 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4349	4318	4316	4316
q2	323	227	220	220
q3	4220	4180	4148	4148
q4	2773	2759	2753	2753
q5	7149	7127	7086	7086
q6	238	122	123	122
q7	3245	2861	2825	2825
q8	4346	4512	4529	4512
q9	16889	16709	16810	16709
q10	4222	4268	4314	4268
q11	752	674	687	674
q12	1023	848	875	848
q13	7331	3774	3776	3774
q14	461	432	420	420
q15	517	461	456	456
q16	745	670	673	670
q17	3795	3886	3846	3846
q18	8832	8717	8760	8717
q19	1722	1734	1681	1681
q20	2403	2148	2141	2141
q21	8477	8546	8537	8537
q22	1062	973	1001	973
Total cold run time: 84874 ms
Total hot run time: 79696 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 204067 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6398c4fde2a86e616d15fba845a0c8f6f059132c, data reload: false

query1	933	417	380	380
query2	6539	2682	2902	2682
query3	6917	202	197	197
query4	21020	18048	17931	17931
query5	19740	6550	6547	6547
query6	299	220	238	220
query7	4155	298	309	298
query8	438	498	414	414
query9	3048	2649	2561	2561
query10	439	298	309	298
query11	11302	10744	10709	10709
query12	124	73	76	73
query13	5596	699	710	699
query14	18112	13502	13909	13502
query15	365	241	248	241
query16	6460	291	264	264
query17	1700	1449	882	882
query18	2299	417	408	408
query19	220	151	153	151
query20	77	77	78	77
query21	187	95	96	95
query22	5141	4996	4954	4954
query23	32554	32171	31874	31874
query24	6937	6496	6543	6496
query25	518	438	426	426
query26	529	159	160	159
query27	1877	302	299	299
query28	6099	2339	2292	2292
query29	2897	2727	2810	2727
query30	247	166	165	165
query31	915	727	733	727
query32	76	65	59	59
query33	400	256	267	256
query34	829	485	476	476
query35	1133	941	983	941
query36	1291	1354	1352	1352
query37	86	61	60	60
query38	3093	2966	2977	2966
query39	1353	1326	1317	1317
query40	199	98	98	98
query41	53	45	44	44
query42	80	82	78	78
query43	667	737	670	670
query44	1110	715	730	715
query45	250	239	236	236
query46	1228	969	970	969
query47	1856	1690	1954	1690
query48	1030	730	721	721
query49	613	380	377	377
query50	874	585	632	585
query51	4823	4657	4629	4629
query52	93	86	91	86
query53	449	330	334	330
query54	2677	2448	2470	2448
query55	84	80	76	76
query56	246	208	213	208
query57	1126	1107	1145	1107
query58	216	211	198	198
query59	4169	3811	3838	3811
query60	223	218	208	208
query61	100	96	97	96
query62	836	462	459	459
query63	501	345	350	345
query64	2495	1549	1407	1407
query65	3593	3546	3621	3546
query66	788	377	388	377
query67	17950	15804	15223	15223
query68	8935	649	667	649
query69	576	347	342	342
query70	1443	1441	1343	1343
query71	419	310	323	310
query72	6509	3508	3498	3498
query73	728	321	317	317
query74	6315	5975	5879	5879
query75	5325	3686	3670	3670
query76	5409	1160	1241	1160
query77	944	263	257	257
query78	12422	12390	11972	11972
query79	13143	635	653	635
query80	826	405	391	391
query81	494	233	236	233
query82	1174	100	105	100
query83	177	132	138	132
query84	255	69	71	69
query85	791	335	331	331
query86	322	301	290	290
query87	3213	3032	3021	3021
query88	6001	2320	2319	2319
query89	431	307	290	290
query90	2116	211	214	211
query91	176	142	140	140
query92	61	54	52	52
query93	6268	560	611	560
query94	859	210	208	208
query95	1098	1056	1051	1051
query96	655	332	327	327
query97	6505	6344	6443	6344
query98	184	172	173	172
query99	2975	882	1011	882
Total cold run time: 321023 ms
Total hot run time: 204067 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 30.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6398c4fde2a86e616d15fba845a0c8f6f059132c, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.80	0.10	0.07
query5	0.52	0.53	0.52
query6	1.26	0.62	0.61
query7	0.01	0.00	0.01
query8	0.04	0.02	0.02
query9	0.51	0.48	0.48
query10	0.55	0.53	0.54
query11	0.12	0.09	0.08
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.78	0.79
query15	0.78	0.76	0.76
query16	0.37	0.37	0.40
query17	1.01	1.02	1.01
query18	0.21	0.25	0.25
query19	1.88	1.83	1.83
query20	0.01	0.01	0.02
query21	15.47	0.55	0.53
query22	2.11	2.28	1.43
query23	17.38	1.02	0.91
query24	4.49	1.73	1.30
query25	0.37	0.08	0.06
query26	0.63	0.16	0.16
query27	0.04	0.05	0.04
query28	7.64	0.71	0.73
query29	12.62	2.24	2.05
query30	0.55	0.53	0.52
query31	2.81	0.38	0.37
query32	3.38	0.50	0.49
query33	3.07	3.03	3.13
query34	15.24	4.79	4.78
query35	4.85	4.83	4.82
query36	1.04	1.01	1.01
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.08	0.01	0.02
query42	0.02	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 103.02 s
Total hot run time: 30.56 s
@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6398c4fde2a86e616d15fba845a0c8f6f059132c with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s
@dataroaring dataroaring merged commit dfadf61 into apache:branch-2.0 Jul 10, 2024
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
3 participants
@SWJTU-ZhangLei @doris-robot @dataroaring