Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](readconsistency) avoid table not exist error #37593

Merged
merged 4 commits into from
Jul 11, 2024
Merged

[fix](readconsistency) avoid table not exist error #37593

merged 4 commits into from
Jul 11, 2024

Conversation

dataroaring
Copy link
Contributor

@dataroaring dataroaring commented Jul 10, 2024 •
edited
Loading

Query following createting table would throw table not exist error.

For example.
t1: client issue create table to master fe
t2: client issue query sql to observer fe, the query would fail due to not exist table in plan phase.
t3: observer fe receive editlog creating the table from the master fe

After the pr:
query at t2 would wait until latest edit log is received from master fe in the observer fe.
@dataroaring
[fix](readconsistency) avoid table not exist error
47ade42 
Query following createting table would throw table not exist error.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.
@dataroaring
Copy link
Contributor Author

run buildall
gavinchou
gavinchou previously approved these changes Jul 10, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.
@doris-robot
Copy link

TPC-H: Total hot run time: 40029 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47ade42db98ceb6311e243f141682f859f38baa7, data reload: false

------ Round 1 ----------------------------------
q1	17852	4480	4379	4379
q2	2668	200	231	200
q3	11546	1149	1127	1127
q4	11247	851	814	814
q5	7857	2712	2632	2632
q6	223	141	143	141
q7	957	599	622	599
q8	9235	2056	2103	2056
q9	8991	6502	6452	6452
q10	8841	3705	3697	3697
q11	457	240	247	240
q12	402	236	236	236
q13	19011	2987	3030	2987
q14	278	246	232	232
q15	535	487	480	480
q16	498	396	388	388
q17	964	685	684	684
q18	8068	7459	7433	7433
q19	1911	1609	1464	1464
q20	691	325	326	325
q21	4949	3118	3308	3118
q22	406	349	345	345
Total cold run time: 117587 ms
Total hot run time: 40029 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4350	4206	4265	4206
q2	386	275	275	275
q3	2975	2718	2742	2718
q4	1844	1626	1611	1611
q5	5241	5278	5285	5278
q6	219	133	131	131
q7	2155	1803	1720	1720
q8	3424	3328	3304	3304
q9	8221	8247	8235	8235
q10	3887	3662	3620	3620
q11	580	500	485	485
q12	808	635	594	594
q13	17409	3056	2975	2975
q14	300	273	267	267
q15	511	474	491	474
q16	477	419	441	419
q17	1776	1493	1476	1476
q18	7536	7601	7358	7358
q19	1692	1460	1455	1455
q20	1989	1786	1789	1786
q21	4997	4873	4851	4851
q22	629	541	551	541
Total cold run time: 71406 ms
Total hot run time: 53779 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 173431 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47ade42db98ceb6311e243f141682f859f38baa7, data reload: false

query1	906	376	366	366
query2	6467	2355	2457	2355
query3	6657	207	212	207
query4	26832	17208	17287	17208
query5	4157	513	506	506
query6	289	173	174	173
query7	4587	294	282	282
query8	325	303	299	299
query9	8446	2392	2375	2375
query10	433	287	275	275
query11	12271	9976	10060	9976
query12	136	85	82	82
query13	1636	377	367	367
query14	10151	7795	7875	7795
query15	236	189	193	189
query16	7207	308	305	305
query17	1807	562	553	553
query18	1349	318	275	275
query19	208	152	153	152
query20	97	83	91	83
query21	212	132	127	127
query22	4248	4168	3990	3990
query23	33848	33095	33172	33095
query24	11907	2864	2832	2832
query25	672	369	366	366
query26	1788	148	148	148
query27	2653	268	272	268
query28	7080	2060	2054	2054
query29	975	636	622	622
query30	288	149	148	148
query31	949	763	771	763
query32	95	53	56	53
query33	770	311	327	311
query34	910	507	495	495
query35	674	555	548	548
query36	1096	929	911	911
query37	237	79	76	76
query38	2824	2710	2760	2710
query39	875	831	802	802
query40	276	122	123	122
query41	61	52	51	51
query42	126	100	108	100
query43	592	546	536	536
query44	1156	741	731	731
query45	198	165	160	160
query46	1081	726	725	725
query47	1878	1755	1757	1755
query48	375	314	294	294
query49	1189	410	417	410
query50	787	408	408	408
query51	6914	6765	6763	6763
query52	134	92	95	92
query53	359	301	299	299
query54	1100	450	469	450
query55	78	75	77	75
query56	304	284	272	272
query57	1174	1026	1032	1026
query58	246	271	288	271
query59	3504	3326	3162	3162
query60	309	277	285	277
query61	97	92	93	92
query62	813	647	651	647
query63	336	303	304	303
query64	10633	2180	1622	1622
query65	3235	3074	3073	3073
query66	1287	361	339	339
query67	15520	14891	15064	14891
query68	4541	541	557	541
query69	478	335	333	333
query70	1188	1125	1068	1068
query71	402	300	294	294
query72	7498	5306	5865	5306
query73	744	327	326	326
query74	6080	5525	5511	5511
query75	3501	2701	2671	2671
query76	2811	980	907	907
query77	497	309	316	309
query78	9370	9159	8853	8853
query79	2167	529	523	523
query80	1324	496	490	490
query81	572	215	218	215
query82	735	135	133	133
query83	266	168	169	168
query84	237	86	90	86
query85	1812	317	305	305
query86	489	327	329	327
query87	3274	3152	3071	3071
query88	3922	2523	2428	2428
query89	479	388	394	388
query90	1856	196	192	192
query91	132	104	105	104
query92	66	52	51	51
query93	2445	519	513	513
query94	1260	218	213	213
query95	416	328	323	323
query96	602	274	271	271
query97	3215	3067	3017	3017
query98	223	202	196	196
query99	1643	1289	1210	1210
Total cold run time: 284305 ms
Total hot run time: 173431 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 30.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47ade42db98ceb6311e243f141682f859f38baa7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.06	0.06
query4	1.66	0.07	0.09
query5	0.50	0.51	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.56	0.50	0.49
query10	0.55	0.54	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.76	0.78	0.78
query15	0.85	0.82	0.81
query16	0.37	0.37	0.37
query17	1.04	1.03	1.03
query18	0.23	0.22	0.23
query19	1.93	1.86	1.89
query20	0.02	0.01	0.02
query21	15.53	0.77	0.66
query22	4.20	7.80	1.49
query23	18.26	1.32	1.23
query24	2.11	0.22	0.22
query25	0.15	0.08	0.09
query26	0.29	0.21	0.21
query27	0.45	0.23	0.23
query28	13.28	1.01	1.00
query29	12.63	3.63	3.65
query30	0.26	0.06	0.05
query31	2.86	0.38	0.38
query32	3.29	0.48	0.47
query33	2.87	2.91	2.88
query34	17.06	4.34	4.48
query35	4.41	4.45	4.40
query36	0.65	0.47	0.47
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.12	0.13
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.93 s
Total hot run time: 30.71 s
@dataroaring
Copy link
Contributor Author

run buildall
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 10, 2024
morrySnow
morrySnow previously approved these changes Jul 10, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 10, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 10, 2024
@dataroaring
Copy link
Contributor Author

run buildall
@doris-robot
Copy link

TPC-H: Total hot run time: 40247 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 95ead5941907bdd1d14a4d5d7f93c13dae9ece49, data reload: false

------ Round 1 ----------------------------------
q1	17696	5113	4311	4311
q2	2021	194	190	190
q3	10501	1240	1117	1117
q4	10191	805	779	779
q5	7587	2758	2712	2712
q6	216	134	134	134
q7	963	603	608	603
q8	9211	2118	2106	2106
q9	8758	6574	6579	6574
q10	8844	3807	3866	3807
q11	451	240	234	234
q12	424	233	231	231
q13	19165	2947	2948	2947
q14	286	230	240	230
q15	531	486	477	477
q16	503	373	371	371
q17	989	623	705	623
q18	8015	7499	7473	7473
q19	7398	1546	1474	1474
q20	668	324	326	324
q21	4965	3197	3250	3197
q22	379	333	333	333
Total cold run time: 119762 ms
Total hot run time: 40247 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4428	4353	4305	4305
q2	366	280	267	267
q3	3177	3007	2911	2911
q4	2109	1789	1674	1674
q5	5589	5529	5591	5529
q6	225	133	133	133
q7	2249	1890	1863	1863
q8	3306	3450	3438	3438
q9	8781	8983	8837	8837
q10	4065	3811	3853	3811
q11	600	522	495	495
q12	821	669	678	669
q13	17108	3171	3144	3144
q14	323	271	301	271
q15	530	490	489	489
q16	485	435	438	435
q17	1812	1529	1533	1529
q18	8124	8181	7869	7869
q19	1775	1604	1675	1604
q20	2242	1857	1830	1830
q21	5213	4759	4886	4759
q22	632	550	561	550
Total cold run time: 73960 ms
Total hot run time: 56412 ms
@doris-robot
Copy link

TPC-DS: Total hot run time: 174628 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 95ead5941907bdd1d14a4d5d7f93c13dae9ece49, data reload: false

query1	932	386	368	368
query2	6457	2512	2401	2401
query3	6637	212	220	212
query4	24626	17656	17227	17227
query5	3766	478	498	478
query6	284	180	169	169
query7	4577	283	292	283
query8	326	306	302	302
query9	8668	2433	2388	2388
query10	440	285	271	271
query11	10595	10043	10106	10043
query12	121	83	82	82
query13	1637	372	366	366
query14	10252	7759	7689	7689
query15	242	187	200	187
query16	7868	324	325	324
query17	1793	560	565	560
query18	1899	283	265	265
query19	187	148	148	148
query20	90	78	78	78
query21	200	127	123	123
query22	4466	4046	4020	4020
query23	34216	33446	33635	33446
query24	10922	2887	2867	2867
query25	592	405	397	397
query26	703	146	149	146
query27	2367	274	286	274
query28	6393	2179	2149	2149
query29	871	647	640	640
query30	252	150	148	148
query31	991	757	758	757
query32	90	57	54	54
query33	713	315	299	299
query34	921	490	492	490
query35	719	645	612	612
query36	1156	982	995	982
query37	142	82	82	82
query38	3024	2824	2771	2771
query39	838	800	788	788
query40	196	119	117	117
query41	47	45	43	43
query42	121	96	102	96
query43	626	564	551	551
query44	1114	721	752	721
query45	205	165	162	162
query46	1075	732	752	732
query47	1850	1769	1765	1765
query48	376	292	295	292
query49	818	412	412	412
query50	776	387	394	387
query51	6855	6726	6701	6701
query52	104	94	91	91
query53	361	289	292	289
query54	886	446	441	441
query55	74	73	73	73
query56	279	257	310	257
query57	1119	1065	1071	1065
query58	251	245	248	245
query59	3362	3386	3107	3107
query60	312	279	265	265
query61	91	92	91	91
query62	796	647	648	647
query63	316	284	294	284
query64	9177	2282	1638	1638
query65	3189	3116	3087	3087
query66	743	333	330	330
query67	15582	15183	15032	15032
query68	4511	532	543	532
query69	515	360	333	333
query70	1216	1160	1112	1112
query71	384	283	283	283
query72	7227	5636	5626	5626
query73	740	320	316	316
query74	5957	5588	5546	5546
query75	3379	2668	2734	2668
query76	2347	905	909	905
query77	483	307	300	300
query78	9721	9013	9050	9013
query79	2908	528	525	525
query80	2051	480	485	480
query81	595	222	218	218
query82	974	137	132	132
query83	337	167	171	167
query84	274	89	84	84
query85	1831	306	306	306
query86	449	311	310	310
query87	3309	3142	3100	3100
query88	4212	2371	2373	2371
query89	486	379	406	379
query90	1721	195	202	195
query91	135	102	105	102
query92	62	48	49	48
query93	5074	530	515	515
query94	1019	210	213	210
query95	405	315	312	312
query96	608	285	279	279
query97	3211	3043	2996	2996
query98	230	202	191	191
query99	1603	1279	1267	1267
Total cold run time: 279648 ms
Total hot run time: 174628 ms
@doris-robot
Copy link

ClickBench: Total hot run time: 31.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 95ead5941907bdd1d14a4d5d7f93c13dae9ece49, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.04
query4	1.68	0.06	0.06
query5	0.50	0.49	0.48
query6	1.15	0.71	0.74
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.56	0.50	0.48
query10	0.55	0.53	0.54
query11	0.14	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.76	0.78	0.80
query15	0.87	0.81	0.81
query16	0.36	0.37	0.36
query17	1.03	1.04	1.02
query18	0.23	0.22	0.21
query19	1.94	1.78	1.82
query20	0.01	0.02	0.01
query21	15.45	0.76	0.66
query22	4.18	6.42	2.45
query23	18.27	1.43	1.39
query24	2.14	0.23	0.21
query25	0.16	0.09	0.09
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.27	1.01	1.02
query29	12.64	3.33	3.33
query30	0.25	0.06	0.06
query31	2.87	0.38	0.39
query32	3.26	0.48	0.47
query33	2.92	2.94	2.92
query34	17.10	4.38	4.28
query35	4.41	4.40	4.40
query36	0.66	0.46	0.47
query37	0.20	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 110.01 s
Total hot run time: 31.33 s
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.
@dataroaring dataroaring merged commit a801172 into apache:master Jul 11, 2024
26 of 29 checks passed
dataroaring added a commit to dataroaring/incubator-doris that referenced this pull request Jul 11, 2024
@dataroaring
[fix](readconsistency) avoid table not exist error (apache#37593)
2307723 
Query following createting table would throw table not exist error.

For example.
t1: client issue create table to master fe
t2: client issue query sql to observer fe, the query would fail due to
not exist table in plan phase.
t3: observer fe receive editlog creating the table from the master fe

After the pr:
query at t2 would wait until latest edit log is received from master fe
in the observer fe.
@dataroaring dataroaring mentioned this pull request Jul 11, 2024
Merged
dataroaring added a commit to dataroaring/incubator-doris that referenced this pull request Jul 11, 2024
@dataroaring
[fix](readconsistency) avoid table not exist error (apache#37593)
7e1a6ad 
Query following createting table would throw table not exist error.

For example.
t1: client issue create table to master fe
t2: client issue query sql to observer fe, the query would fail due to
not exist table in plan phase.
t3: observer fe receive editlog creating the table from the master fe

After the pr:
query at t2 would wait until latest edit log is received from master fe
in the observer fe.
dataroaring added a commit that referenced this pull request Jul 11, 2024
@dataroaring
[fix](readconsistency) avoid table not exist error (#37593) (#37641)
fdf21ec 
Query following createting table would throw table not exist error.

For example.
t1: client issue create table to master fe
t2: client issue query sql to observer fe, the query would fail due to
not exist table in plan phase.
t3: observer fe receive editlog creating the table from the master fe

After the pr:
query at t2 would wait until latest edit log is received from master fe
in the observer fe.

pick #37593

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
dataroaring added a commit to dataroaring/incubator-doris that referenced this pull request Jul 12, 2024
@dataroaring
[fix](readconsistency) avoid table not exist error apache#37593
95484da 
pick apache#37593
@dataroaring dataroaring mentioned this pull request Jul 12, 2024
Merged
dataroaring added a commit that referenced this pull request Jul 13, 2024
@dataroaring
[fix](readconsistency) avoid table not exist error #37593 (#37709)
c7e492d 
pick #37593

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
dataroaring added a commit that referenced this pull request Jul 15, 2024
@dataroaring
[fix](readconsistency) avoid table not exist error (#37593)
c7f2021 
Query following createting table would throw table not exist error.

For example.
t1: client issue create table to master fe
t2: client issue query sql to observer fe, the query would fail due to
not exist table in plan phase.
t3: observer fe receive editlog creating the table from the master fe

After the pr:
query at t2 would wait until latest edit log is received from master fe
in the observer fe.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@dataroaring @seawinde
[fix](readconsistency) avoid table not exist error (apache#37593)
80a093c 
Query following createting table would throw table not exist error.

For example.
t1: client issue create table to master fe
t2: client issue query sql to observer fe, the query would fail due to
not exist table in plan phase.
t3: observer fe receive editlog creating the table from the master fe

After the pr:
query at t2 would wait until latest edit log is received from master fe
in the observer fe.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged reviewed
4 participants
@dataroaring @doris-robot @gavinchou @morrySnow