Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metrics master #1117

Merged
merged 7 commits into from
Mar 7, 2024
Merged

Metrics master #1117

merged 7 commits into from
Mar 7, 2024

Conversation

guljain
Copy link
Contributor

@guljain guljain commented Feb 15, 2024

This PR does 2 things:

  • Currently the instrumentation contained metrics that could be per GHFS instance based but now they are also at executor level. This means that there is one metric per executor also.

Adding GCS specific statistics :

  • gcs_client_side_error_count : Counts the occurrence of client side error status code
  • gcs_server_side_error_count: Counts the occurrence of server side error status code
  • gcs_client_rate_limit_count: Counts the occurrence of 429 status code
  • gcs_total_request_count: Counts the total number of gcs requests made
  • exception_count: Counts the number of exceptions encountered (Can be expanded in future)
arunkumarchacko
arunkumarchacko reviewed Feb 26, 2024
View reviewed changes
gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GhfsGlobalStorageStatistics.java Outdated
private static final GoogleLogger logger = GoogleLogger.forEnclosingClass();

/** {@value} The key that stores all the registered metrics */
public static final String NAME = "GhfsGlobalStorageStatistics";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has to be same as that in 2.2.x, correct?
gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GhfsGlobalStorageStatistics.java Outdated Show resolved Hide resolved
gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GhfsGlobalStorageStatistics.java Outdated Show resolved Hide resolved
gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopFSInputStream.java Outdated Show resolved Hide resolved
gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopFSInputStream.java Outdated Show resolved Hide resolved
gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopFileSystem.java Outdated Show resolved Hide resolved
gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopOutputStream.java Show resolved Hide resolved
gcs/src/test/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopOutputStreamTest.java Outdated
@Before
public void setUp() throws IOException {
ghfs = GoogleHadoopFileSystemTestHelper.createInMemoryGoogleHadoopFileSystem();
// gcsFsIHelper = GoogleCloudStorageFileSystemIntegrationHelper.create();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we are not really using gcsFsIHelper in the tests.
gcs/src/test/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopOutputStreamTest.java Outdated
@@ -53,14 +54,19 @@ public class GoogleHadoopOutputStreamTest {

private GoogleHadoopFileSystem ghfs;

// private static GoogleCloudStorageFileSystemIntegrationHelper gcsFsIHelper;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove?
@guljain
Copy link
Contributor Author

guljain commented Feb 27, 2024

/gcbrun
2 similar comments
@guljain
Copy link
Contributor Author

guljain commented Feb 27, 2024

/gcbrun
@guljain
Copy link
Contributor Author

guljain commented Feb 27, 2024

/gcbrun
@guljain guljain merged commit b0fff1c into GoogleCloudDataproc:master Mar 7, 2024
2 of 4 checks passed
guljain added a commit that referenced this pull request Jun 5, 2024
@guljain
Branch 3.0.x metrics (#1154)
d7c43cf 
* Metrics master (#1117)

* Instrumentation

* Updating comments

* Adding global statistics along with GCS Connector specific statistics

* testing

* Resolving comments

* Resolving comments

(cherry picked from commit b0fff1c)

* changes for global metrics

* Addressing comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants
@guljain @arunkumarchacko