Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Oozie] Updated oozie.sh to include rocky8 and rocky9 image versions #1178

Merged
merged 2 commits into from
Jul 6, 2024

Conversation

TheAkertu
Copy link
Contributor

No description provided.

@cjac
Copy link
Contributor

cjac commented Apr 26, 2024

/gcbrun

@kuldeepkk-dev
Copy link
Contributor

Tests are failing on 2.2-debian12 and 2.2-ubuntu22.
Please check what's happening on these two images.

@cjac
Copy link
Contributor

cjac commented Jul 5, 2024

2.2-debian12 with 3 masters comes up fine

Fri Jul  5 04:41:19 PM PDT 2024
+ gcloud dataproc clusters create cluster-1668020639 --master-boot-disk-type pd-ssd --worker-boot-disk-type pd-ssd --secondary-worker-boot-disk-type pd-ssd --master-boot-disk-size 100 --worker-boot-disk-size 100 --secondary-worker-boot-disk-size 100 --master-machine-type n1-standard-8 --worker-machine-type n1-standard-8 --num-masters=3 --num-workers=3 --num-secondary-workers=0 --region us-central1 --zone us-central1-a --subnet subnet-cluster-1668020639 --no-address --image-version 2.2 --service-account=sa-cluster-1668020639@cjac-2021-00.iam.gserviceaccount.com --tags=tag-cluster-1668020639 --bucket cjac-docker-on-yarn --enable-component-gateway --initialization-action-timeout=35m --max-idle=2d --no-shielded-secure-boot --metadata include-gpus=true --metadata gpu-driver-provider=NVIDIA --initialization-action-timeout=15m --initialization-actions gs://cjac-docker-on-yarn/dataproc-initialization-actions/oozie/oozie.sh --properties spark:spark.executor.resource.gpu.amount=1,spark:spark.task.resource.gpu.amount=1 --metadata init-actions-repo=gs://cjac-docker-on-yarn/dataproc-initialization-actions --metadata install-gpu-agent=true --scopes https://www.googleapis.com/auth/cloud-platform
Waiting on operation [projects/cjac-2021-00/regions/us-central1/operations/df50baa3-c563-3466-be85-c153cc5fde89].
Waiting for cluster creation operation...
WARNING: Consider using Auto Zone rather than selecting a zone manually. See https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone
Waiting for cluster creation operation...done.
Created [https://dataproc.googleapis.com/v1/projects/cjac-2021-00/regions/us-central1/clusters/cluster-1668020639] Cluster placed in zone [us-central1-a].

real    5m38.775s
user    0m2.884s
sys     0m0.175s
+ date
Fri Jul  5 04:46:58 PM PDT 2024
@cjac
Copy link
Contributor

cjac commented Jul 6, 2024

2.2-ubuntu22 also succeeds:

Fri Jul  5 04:49:55 PM PDT 2024
+ gcloud dataproc clusters create cluster-1668020639 --master-boot-disk-type pd-ssd --worker-boot-disk-type pd-ssd --secondary-worker-boot-disk-type pd-ssd --master-boot-disk-size 100 --worker-boot-disk-size 100 --secondary-worker-boot-disk-size 100 --master-machine-type n1-standard-8 --worker-machine-type n1-standard-8 --num-masters=3 --num-workers=3 --num-secondary-workers=0 --region us-central1 --zone us-central1-a --subnet subnet-cluster-1668020639 --no-address --image-version 2.2-ubuntu22 --service-account=sa-cluster-1668020639@cjac-2021-00.iam.gserviceaccount.com --tags=tag-cluster-1668020639 --bucket cjac-docker-on-yarn --enable-component-gateway --initialization-action-timeout=35m --max-idle=2d --no-shielded-secure-boot --metadata include-gpus=true --metadata gpu-driver-provider=NVIDIA --initialization-action-timeout=15m --initialization-actions gs://cjac-docker-on-yarn/dataproc-initialization-actions/oozie/oozie.sh --properties spark:spark.executor.resource.gpu.amount=1,spark:spark.task.resource.gpu.amount=1 --metadata init-actions-repo=gs://cjac-docker-on-yarn/dataproc-initialization-actions --metadata install-gpu-agent=true --scopes https://www.googleapis.com/auth/cloud-platform
Waiting on operation [projects/cjac-2021-00/regions/us-central1/operations/3d6a7259-c6ae-3a61-a3e8-42cbf24f9364].
Waiting for cluster creation operation...⠛
WARNING: Consider using Auto Zone rather than selecting a zone manually. See https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone
Waiting for cluster creation operation...done.
Created [https://dataproc.googleapis.com/v1/projects/cjac-2021-00/regions/us-central1/clusters/cluster-1668020639] Cluster placed in zone [us-central1-a].

real    5m38.564s
user    0m2.792s
sys     0m0.238s
+ date
Fri Jul  5 04:55:33 PM PDT 2024
Copy link
Contributor

@cjac cjac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cjac cjac merged commit a1fdbf3 into GoogleCloudDataproc:master Jul 6, 2024
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
3 participants