Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Invalid view name] [data quality] Invalid view name when use r #16286

Open
3 tasks done
jeanleen opened this issue Jul 8, 2024 · 1 comment
Open
3 tasks done

[Invalid view name] [data quality] Invalid view name when use r #16286

jeanleen opened this issue Jul 8, 2024 · 1 comment
Labels
bug Something isn't working Waiting for user feedback Waiting for feedback from issue/PR author

Comments

@jeanleen
Copy link

jeanleen commented Jul 8, 2024

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

I want to make sure that the data proves to be compliant when using the oralce database;then bug below appear
User class threw exception: org.apache.spark.sql.AnalysisException: Invalid view name: imc.oa.pdc_xxxx
at org.apache.spark.sql.errors.QueryCompilationErrors$.invalidViewNameError(QueryCompilationErrors.scala:2283)
at org.apache.spark.sql.Dataset.createTempViewCommand(Dataset.scala:3542)
at org.apache.spark.sql.Dataset.createOrReplaceTempView(Dataset.scala:3495)
at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.registerTempView(SparkBatchExecution.java:67)
at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.registerInputTempView(SparkBatchExecution.java:77)
at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.lambda$execute$0(SparkBatchExecution.java:48)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.dolphinscheduler.data.quality.execution.SparkBatchExecution.execute(SparkBatchExecution.java:48)
at org.apache.dolphinscheduler.data.quality.context.DataQualityContext.execute(DataQualityContext.java:62)
at org.apache.dolphinscheduler.data.quality.DataQualityApplication.main(DataQualityApplication.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:739)

What you expected to happen

`public static List getReaderConfigList(
Map<String, String> inputParameterValue,
DataQualityTaskExecutionContext dataQualityTaskExecutionContext) throws DataQualityException {

    List<BaseConfig> readerConfigList = new ArrayList<>();

    // all the rule need the source config
    if (StringUtils.isNotEmpty(dataQualityTaskExecutionContext.getSourceConnectorType())) {
        BaseConnectionParam sourceDataSource =
                (BaseConnectionParam) DataSourceUtils.buildConnectionParams(
                        DbType.of(dataQualityTaskExecutionContext.getSourceType()),
                        dataQualityTaskExecutionContext.getSourceConnectionParams());
        BaseConfig sourceBaseConfig = new BaseConfig();
        sourceBaseConfig.setType(dataQualityTaskExecutionContext.getSourceConnectorType());
        Map<String, Object> config = new HashMap<>();
        if (sourceDataSource != null) {
            config.put(DATABASE, inputParameterValue.get(SRC_DATABASE));
            config.put(TABLE, inputParameterValue.get(SRC_TABLE));
            config.put(URL, DataSourceUtils.getJdbcUrl(DbType.of(dataQualityTaskExecutionContext.getSourceType()),
                    sourceDataSource));
            config.put(USER, sourceDataSource.getUser());
            config.put(PASSWORD, ParserUtils.encode(sourceDataSource.getPassword()));
            config.put(DRIVER, DataSourceUtils
                    .getDatasourceDriver(DbType.of(dataQualityTaskExecutionContext.getSourceType())));
            **String outputTable = inputParameterValue.get(SRC_DATABASE) + "_" + inputParameterValue.get(SRC_TABLE);**
            config.put(OUTPUT_TABLE, outputTable);
            inputParameterValue.put(SRC_TABLE, outputTable);
        }
        sourceBaseConfig.setConfig(config);

        readerConfigList.add(sourceBaseConfig);
    }`

How to reproduce

image

Anything else

No response

Version

3.1.x

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@jeanleen jeanleen added bug Something isn't working Waiting for reply Waiting for reply labels Jul 8, 2024
@SbloodyS
Copy link
Member

SbloodyS commented Jul 8, 2024

Can you provide reproduce steps?

@SbloodyS SbloodyS added Waiting for user feedback Waiting for feedback from issue/PR author and removed Waiting for reply Waiting for reply labels Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Waiting for user feedback Waiting for feedback from issue/PR author
2 participants