Add java-storage implementation for copy objects method. #1087
Conversation
shilpi23pandey
force-pushed
the
move-to-java-storage-copy
branch
from
c1e7d4a
to
b5317e8
Compare
|
/gcbrun |
shilpi23pandey
force-pushed
the
move-to-java-storage-copy
branch
from
b5317e8
to
7199a77
Compare
|
/gcbrun |
| CopyWriter copyWriter = storage.copy(copyRequestBuilder.build()); | ||
| while (!copyWriter.isDone()) { | ||
| copyWriter.copyChunk(); | ||
| logger.atFiner().log( |
There was a problem hiding this comment.
let us make this finest?
| if (storageOptions.getMaxRewriteChunkSize() > 0) { | ||
| copyRequestBuilder.setMegabytesCopiedPerChunk( | ||
| // Convert raw byte size into Mib. | ||
| storageOptions.getMaxRewriteChunkSize() / (1024 * 1024)); |
There was a problem hiding this comment.
If this is less than 1MB, it will be 0? Is that ok?
There was a problem hiding this comment.
We do have checks which enforces that getMaxRewriteChunkSize should be multiple of MiB so I don't think it will ever be zero.
| sourceToDestinationObjectsMap.entrySet()) { | ||
| StorageResourceId srcObject = entry.getKey(); | ||
| StorageResourceId dstObject = entry.getValue(); | ||
| copyInternal( |
There was a problem hiding this comment.
Why are we not use Rewrite if copyWithRewrite config is enabled?
There was a problem hiding this comment.
Veneer does not expose separate methods for rewrite and copy, it internally always uses rewrite and decides number of calls based on the object size. The APIary copy also uses rewrite API but with a single RPC call https://cloud.google.com/storage/docs/json_api/v1/objects/copy
shilpi23pandey
force-pushed
the
move-to-java-storage-copy
branch
from
7199a77
to
3e9edef
Compare
|
/gcbrun |
shilpi23pandey
merged commit c208df4
into
GoogleCloudDataproc:master
java-storageclient internally uses gcs rewrite API for copy so we don't need the logic to decide weather to issue a copy or rewrite as opposed to the APIary implementation. https://cloud.google.com/java/docs/reference/google-cloud-storage/latest/com.google.cloud.storage.Storage#com_google_cloud_storage_Storage_copy_com_google_cloud_storage_Storage_CopyRequest_