API Academy

Sponsored by Layer7

A black and white photo of the ceiling in an indoor building.

DevOps: REST API Execution Through Bash Shell Scripting

By Thomas ChengPosted on

A Simple Framework: Core

This is the first part of a paper proposing a framework that enables DevOps teams to issue REST API calls via bash shell scripts. The second part of the paper discusses the extension beyond the core of the proposed framework. The third part shares some experiences learned from real world applications.

Introduction

JSON-based REST APIs have become a standard approach for programmatic integration with SaaS offerings, microservices, and even modernized legacy applications. While resources such as SoapUI (https://www.soapui.org/) and Swagger UI (https://swagger.io/) have proved themselves being useful tools that facilitate the discovery and consumption of Web Services and REST APIs, the primary audience for APIs are generally still developers with rather in-depth programming experience. Non-programmer system administrators often tend to stick to scripting using CLI commands, which are not always available.

Both Swagger UI and most common API developer portal solutions in today’s market only provide snippets of the equivalent curl commands and JSON examples to help non-programmer system administrators with the automation of the common administrative tasks. To allow better-structured automation scripts that utilize REST API directly, this paper is proposing a simple framework for administrators and experienced programmers alike. The goal is to provide an approach that improves the REST API client script readability and development efficiency.

Ideally, REST API publishers would provide a client SDK that is based on this sort of framework so that its users will not have to construct one themselves. You are encouraged to adopt this framework and adjust it to the particular API implementations you are exposing. This would allow the users to focus on building their own applications and help contributing to extensions that benefit us all.

Tools and Portability

We recommend using GNU Bash shell as the scripting language. The popularity of various Linux distributions and the git bundle for Windows at https://git-scm.com/download have simplified the portability issue that addresses both the Linux and Windows environments. While developing your own REST API bash shell script client program, we suggest sticking with the utilities that are available as part of the git bundle for Windows.

In addition, we use jq, available at https://stedolan.github.io/jq/, to process JSON data and curl to access the REST API endpoints.

Core Framework

Our core framework consists of the base utilities highlighted by curl, the authentication module for privileged operations, the common HTTP operations, the directory structure for multiple object types, and the templates to tame the possible complex JSON data.

curl, CRUD and other Basic Commands

In REST APIs, the commonly known CRUD (Create, Read, Update, Delete) operations are usually translated into POST, GET, PUT, and DELETE, respectively. The curl command allows us to address the API endpoints along with the desired HTTP method. The method can be specified in the curl command line in the -X <<HTTP Method>>. A common curl example is structured as follows:

curl -s -k -X <<COMMAND>> –header Content-Type: application/json’ \
        –header ‘Accept:application/json’  \
        <<AUTHENTICATION>>  <<OPTIONALDATA>> \
        <<RESTAPIENDPOINT>>

-s is the silent mode. It limits the output of the curl command execution.

-k is to allow self-signed certificate or “in-secured connection”. Depending on your security policy, you may not want to use it.

-X <<COMMAND>> specifies the HTTP operation to be used.

<<RESTAPIENDPOINT>> depends on your API design and deployment. Typically, it is characterized as https://<<HOST>>:<<PORT>>/<<BASEURL>>/<<OTHERSPECIFICS>>

<<OPTIONALDATA>> tends to be required for POST and PUT and can simply be -d @”$JSONFILE”, where $JSONFILE is an environment variable that points to the JSON data file.

–header ‘Content-Type: application/json’ indicates the data coming with the request are JSON formatted

–header ‘Accept:application/json’ tells the API endpoint that JSON output is expected

<AUTHENTICATION>> varies and depends on the particular API. Some typical examples are:

  • Basic Authentication: –header ‘Authorization: Basic <<Id:PasswordBase64EncodedValue>>’
  • Bearer Token: –header ‘Authorization: Bearer <<BearerToken>>’
  • Cookie: –-cookie CookieJarFileToReadFrom –cookiejar CookieJarFileToWriteTo

Authentication Module

For privileged API operations that require authentication/authorization, the security token is specified through the curl command as shown above, the diversity of authentication options and REST API deployment instances suggest having a separate shell script as a module for all other REST API calls. This authentication module would export out the HOST, PORT, and AUTHN environment variables.

#!/bin/bash
export HOST=…
export PORT=…
export AUTHN=…

To increase the maintainability of a Rest API client script, we suggest constructing the commonly used CRUD (Create, Read, Update, Delete) commands using more descriptive file names. For example, with the authentication module in place, a Read operation named as read.sh could look like:

#!/bin/bash
. ${HOMEDIR}/authn
curl -s -k -X GET –header Content-Type: application/json’ \
        –header ‘Accept: application/json’  \
        “${AUTHN}” \
        “https://${HOST}${PORT}”/<<BASEURL>>/<<ENDPOINTSPECIFIC>>”

In addition to the common CRUD operations, there may be other application specific operations implemented through the POST/GET/PUT/DELETE verbs. In those cases, please name the operation similar to read.sh to correctly reflect the nature of them.

Directory Structure and Objects

The API CRUD operations operate on objects, represented by their URIs, used in the applications. When there are more object types, a simple read.sh could become readobjtype1.sh, readobjtype2.sh, …, and so on. Alternatively, a hierarchical directory structure similar to the URI itself can be used to group operations that belong to the same object type. Then, we could have objtype1/read.sh, objtype1/create.sh, …, and so on.

For applications that use endpoints with multi-level structures, you can choose to adopt the same subdirectory structures, for example domains/realms/rules/create.sh or you may choose to flatten it up as rules/create.sh so long as doing so does not result in conflicts.

Templates and JSON Files.

The create operation of a REST API usually takes a JSON input to hold the attributes of the object. Instead of attempting to use a program that accepts parameters to construct the JSON data on the fly, it is actually more appropriate to set aside commonly used templates to save the users from having to specify all those attributes. These templates could include both simple objects and more complicated relationships.

A template could simply be a shell script that takes parameters and uses the echo command to help deliver the desired JSON data. A minimal example could look like:

#!/bin/bash
Name=$1
echo <<EOF
{
    “Name”: “$Name”
}
EOF

You may also want to set aside fixed JSON files that you use repeatedly.

Beyond the Core

The core of your API shell scripts need only to include the operations that address the REST API endpoints directly. We will cover the other bells and whistles in the “extension†part of this paper. Stay tuned for more!

To explore the various aspects of the modern APIs development and management, please visit https://e3z.729.myftpupload.com/category/books-and-guides/.