Retrieving artifacts to a specific location on a volume

[crileroro]

I have a question regarding sharing volumes in a multi step workflow. My use case is the following: In a workflow I have an initial step that generates a json output with paths to an S3 bucket for example ["path/object1", "path/object2", "path/object3"] (the number of paths is variable). After I have this info, in the next task I need to download all these objects and put them in a directory, let's say /app/objects so its content will be /app/objects/object*. The last task will read from that directory and use the files for further processing. What I have for now is the following:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: my-files-s3
spec:
  entrypoint: init
  volumeClaimTemplates:
  - metadata:
      name: workdir
    spec:
      accessModes:
        - "ReadWriteMany"
      storageClassName: efs-workflows
      resources:
        requests:
          storage: 1Gi
  templates:
    - name: init
      dag:
        tasks:
          - name: generate-paths
            template: generate-paths
          - name: get-from-s3
            template: get-from-s3
            arguments:
              parameters:
                - name: path_objects
                  value: '{{item}}'
            dependencies:
              - generate-paths
            withParam: '{{tasks.generate-paths.outputs.result}}'
          - name: check-artifacts
            template: check-artifacts
            dependencies:
              - get-from-s3
    - name: generate-paths
      container:
        image: generate-paths:0.1.0
    - name: get-from-s3
      inputs:
        parameters:
          - name: path_objects
        artifacts:
          - name: path_objects
            path: /app/objects/{{=sprig.base(inputs.parameters.path_objects)}}
            s3:
              key: '{{inputs.parameters.path_objects}}'
      container:
        image: alpine:latest
        volumeMounts:
          - name: workdir
            mountPath: /app/objects
    - name: check-artifacts
      container:
        image: alpine:latest
        command: [sh, -c]
        args: ["ls -la /app/objects"]
        volumeMounts:
          - name: workdir
            mountPath: /app/objects

The result of the last task check-artifacts is what I expect:

/app/objects/object1
/app/objects/object2
...

It does what it needs to do, but here comes my question. How can I improve this workflow taking into account that I just need 2 main tasks: generate-paths and check-artifacts ? I feel that the middle task get-from-s3 creates overhead in the pipeline. They are just tasks that do nothing. They are simply used to spin up a pod that mounts a volume where the S3 files are going to be stored and used by the last task.

Thank you folks!

[agilgur5]

As far as I can tell, you don't need the middle task, as you suspected. I believe this would work fine:

    - name: check-artifacts
      inputs:
        parameters:
          - name: path_objects
        artifacts:
          - name: path_objects
            path: /app/objects/{{=sprig.base(inputs.parameters.path_objects)}}
            s3:
              key: '{{inputs.parameters.path_objects}}'
      container:
        image: alpine:latest
        command: [sh, -c]
        args: ["ls -la /app/objects"]
        volumeMounts:
          - name: workdir
            mountPath: /app/objects

Pretty much just merging them, nothing fancy. Argo's init container retrieves the Artifacts, and then your container runs as the main container after. Previously your middle get-from-s3's container wasn't doing anything, if I understood correctly, and you can just make it do whatever you need

[crileroro]

The problem with this approach is that withParam will now be used under check-artifacts which will cause Argo to spin up multiple check-artifacts tasks running in parallel and processing the same files, instead one single check-artifacts task that will process the data which is what I would require.

[agilgur5]

Ah yes, you're right; I might've missed the loop, thanks for correcting me. It also would be helpful to rename path_objects as something singular for consistency (as there is only one item per iteration)

What you can do instead is retrieve the directory of artifacts, such as in this example. I think that would end up as something like:

    - name: check-artifacts
      inputs:
        artifacts:
          - name: path_objects
            path: /app/objects/
            s3:
              key: '<my_s3_dir>'
      container:
        image: alpine:latest
        command: [sh, -c]
        args: ["ls -la /app/objects"]
        volumeMounts:
          - name: workdir
            mountPath: /app/objects

That might've been what I thought you were doing initially already, as opposed to a loop, but I don't remember anymore 😅