Do I have a cleaner way to assign a parametrized json string to a bash variable?

Question

I'd like to prepare a test for a REST petstore server by the mean of curl.

To test its add new pet api, I prepare a json string that holds the Pet to insert in a bash function that executes the curl:

#!/bin/bash

# Add new pet
add_petstore() {
  local rest_url="$1"

  local id_pet="$2"
  local id_category="$3"
  local category_name="$4"
  local pet_name="$5"
  local photo_url="$6"
  local tag_id="$7"
  local tag_name="$8"

  local pet_to_add="{ \
        \"id\": $id_pet, \
        \"category\": {  \
          \"id\": $id_category,  \
          \"name\": \"$category_name\"  \
        },  \
        \"name\": \"$pet_name\",  \
        \"photoUrls\": [  \
          \"$photo_url\"  \
        ],  \
        \"tags\": [  \
          {
            \"id\": $tag_id,  \
            \"name\": \"$tag_name\"  \
          }  \
        ]  \
      }"

  echo "$pet_to_add"

  curl -X 'POST' \
    "$rest_url" \
    -H 'accept: application/xml' \
    -H 'Content-Type: application/json' \
    -d "$pet_to_add"
}

add_petstore "http://localhost:8080/pet" "151" "12" "Dogs" "REX" "http://photosofrex/rex_rage.jpg" "1" "ferocious sales"

The echo of $pet_to_add looks the one I willing to have:

{         "id": 151,         "category": {            "id": 12,            "name": "Dogs"          },          "name": "REX",          "photoUrls": [            "http://photosofrex/rex_rage.jpg"          ],          "tags": [            {
            "id": 1,              "name": "ferocious sales"            }          ]        }

and add_petstore method allows me to prepare few pets easily.

---

But the local pet_to_add=... declaration is really dirty.
If someone (or me later) has to adapt this script, this local variable isn't welcoming.

I thought first that I could put its content in a file and read it with a local pet_to_add=$(cat myfile). But this wouldn't resolve it's variable parameters then.

Do I have a way to write that local pet_to_add declaration a cleaner way?

Answer 1

Using jq to create pieces of JSON for each sub-element of the top-level payload document and then putting everything together. This way, you ensure that jq has an opportunity to encode every string, and you avoid injecting shell variables into JSON (which may break the document; see, e.g., the $pet_name value used below).

First, you'd set up the shell variables with their values. You could use values read from the command line.

pet_id=bee001
pet_name='Eric "the half a bee" II'

category_id=bee
category_name='The bees'

# I elected to do the tags as an associative array.
declare -A tags
tags=(
        [beauty]=high
        [cost]=medium
        [social]=low
)

# I'm assuming there can by many photo URLs.
photo_URLs=(
        "url1"
        "url2"
        "url3"
)

We then create the JSON for the category part:

# Create category
category_json=$(
        jq -c -n \
                --arg id "$category_id" \
                --arg name "$category_name" \
                '$ARGS.named'
)

This would create a single JSON object with id and name keys. The values for the key are taken from the given shell variables. The $ARGS.named thing is an internal jq object containing all keys+values given on the command line using --arg or --argjson (for values that should not be, or already are, string encoded). Since the jq expression is single-quoted, the shell will not mistake $ARGS for a shell variable.

We then create the photo URL array for the photoUrls part:

# Create photo URL array
photoUrls_json=$(
        jq -c -n \
                '$ARGS.positional' \
                --args "${photo_URLs[@]}"
)

This uses jq with --args, which populates the internal $ARGS.positional array with the given values. Note that --args and its argument list must always occur last on the command line.

Next, we create the tags part from our associative array:

# Create tags
tags_json=$(
        for tag_id in "${!tags[@]}"; do
                jq -c -n \
                        --arg id "$tag_id" \
                        --arg name "${tags[$tag_id]}" \
                        '$ARGS.named'
        done |
        jq -c -s '.'
)

We create the array elements from the keys and values of the associative shell array in a loop and insert them in an array with jq -s reading from the loop.

If your tag IDs are numeric, do tags as an ordinary bash array:

tags=(                                                                                                                                                                    
        [1]=high                                                                                                                                                          
        [12]=medium                                                                                                                                                       
        [90]=low                                                                                                                                                          
)

... and then convert it into JSON using

# Create tags
tags_json=$(
        for tag_id in "${!tags[@]}"; do
                jq -c -n \
                        --argjson id "$tag_id" \
                        --arg name "${tags[$tag_id]}" \
                        '$ARGS.named'
        done |
        jq -c -s '.'
)

Notice --argjson id "$tag_id" in place of --arg id "$tag_id" to pass the number, which should not be converted into a string.

We may then finally put together the JSON payload from these parts:

# Create final JSON payload
payload_json=$(
        jq -c -n \
                --arg id "$pet_id" \
                --argjson category "$category_json" \
                --arg name "$pet_name" \
                --argjson photoUrls "$photoUrls_json" \
                --argjson tags "$tags_json" \
                '$ARGS.named'
)

# DEBUG:
printf 'payload is\n%s\n' "$payload_json"

The bash script above would output a JSON document equivalent to the following (but in a compact form):

{
  "id": "bee001",
  "category": {
    "id": "bee",
    "name": "The bees"
  },
  "name": "Eric \"the half a bee\" II",
  "photoUrls": [
    "url1",
    "url2",
    "url3"
  ],
  "tags": [
    {
      "id": "beauty",
      "name": "high"
    },
    {
      "id": "cost",
      "name": "medium"
    },
    {
      "id": "social",
      "name": "low"
    }
  ]
}

Answer 2

Yes, there is a way that's cleaner for many types of JSON documents (though not all types), a here document:

local pet_to_add=$(cat - <<END_DOC
  {
    "id": $id_pet,
    "category": {
      "id": $id_category,
      "name": "$category_name"
    },
    "name": "$pet_name",
    "photoUrls": [
      "$photo_url"
    ],
    "tags": [
      {
        "id": $tag_id,
        "name": "$tag_name"
      }
    ]
  }
END_DOC
)

For this kind of use case, I think of a here document as a form of double-quoted string because it expands shell/environment variables in the document, but it doesn't remove double quote characters because they aren't the string delimiters.

---

When your JSON document is small you can use the builtin printf command to insert values in the middle of the document and save the resulting JSON in a shell variable:

local example_json
printf -v example_json '{ "name": "%s", "id": "%s" }' "${pet_name}" "${id_pet}"

Answer 3

You're currently creating a string that's a mixture of format (names of fields, structure chars like { and [, etc.) and data (arguments to the function). I'd separate the data from the format and use a here document to populate that format using - in the here doc start delimiter to provide tab indenting (leading tabs in the code where you populate the string don't appear in the output) and wrapping the here doc delimiter in 's so the whole format is treated as a single-quoted string to protect it from any possible shell expansion (e.g. Cost $5 would not expand $5 as a positional parameter), and then just let "$@" expand to fill in the values of the placeholder %ss in the format which as a side benefit means you don't need all those other local variables:

$ cat tst.sh
#!/usr/bin/env bash

add_petstore() {
  local rest_url=$1 fmt pet_to_add bad_chars="\""
  shift

  for arg; do
     if [[ $arg =~ $bad_chars ]]; then
        printf 'Error: "%s" contains a char we cannot accept.\n' "$arg" > &2
        return 1
     fi
  done

  IFS= read -r -d '' fmt <<-'!'
        {
          "id": %s,
          "category": {
            "id": %s,
            "name": "%s"
          },
          "name": "%s",
          "photoUrls": [
            "%s"
          ],
          "tags": [
            {
              "id": %s,
              "name": "%s"
            }
          ]
        }
        !

  printf -v pet_to_add "$fmt" "$@"
  echo "$pet_to_add"
}

add_petstore "http://localhost:8080/pet" "151" "12" "Dogs" "REX" "http://photosofrex/rex_rage.jpg" "1" "ferocious sales"

$ ./tst.sh
{
  "id": 151,
  "category": {
    "id": 12,
    "name": "Dogs"
  },
  "name": "REX",
  "photoUrls": [
    "http://photosofrex/rex_rage.jpg"
  ],
  "tags": [
    {
      "id": 1,
      "name": "ferocious sales"
    }
  ]
}

The leading spaces within the here document are tabs so they'll be ignored courtesy of the - in <<-. If you actually want those tabs in your output then change <<- to just <<.

If you really don't want newlines in pet_to_add but do want to use them when populating the format string then just change this:

printf -v pet_to_add "$fmt" "${@}"

to this:

printf -v pet_to_add "${fmt//$'\n'}" "${@}"

and then you'll get this output:

$ ./tst.sh
{  "id": 151,  "category": {    "id": 12,    "name": "Dogs"  },  "name": "REX",  "photoUrls": [    "http://photosofrex/rex_rage.jpg"  ],  "tags": [    {      "id": 1,      "name": "ferocious sales"    }  ]}

and if you wanted to squeeze all chains of contiguous white space to single blank chars then this:

shopt -s extglob
printf -v pet_to_add "${fmt//+([[:space:]])/ }" "${@}"

would produce this output:

$ ./tst.sh
{ "id": 151, "category": { "id": 12, "name": "Dogs" }, "name": "REX", "photoUrls": [ "http://photosofrex/rex_rage.jpg" ], "tags": [ { "id": 1, "name": "ferocious sales" } ] }

The point being you can trivially massage the format however you like before populating pet_to_add without impacting the data (and vice-versa) because we've decoupled the data from the format.