API Academy

Sponsored by Layer7

A black and white photo of the ceiling in an indoor building.

DevOps: REST API Execution Through Bash Shell Scripting III

By Thomas ChengPosted on

A Simple Framework (Experiences)

This is the third and last part of the paper proposing a framework that enables DevOps teams to issue REST API calls via bash shell scripts. This part shares some experiences/lessons learned from real world applications.

Experiences

It would be nice if API publishers simply provided an API client SDK in shell scripts. While this is not yet common, we’re doing our part to get the word out and promote this need.

Agile Friendliness

While working on our REST API client program using this framework for a couple of applications, we found out that we could focus on a limited set of objects and operations we needed at the time and only expanded to other objects as required. This allowed us to plan our work using the recent popular continuous integration and deployment approach.

Attribute-Level Operations

The development of higher-level operations similar to the ones mentioned before usually meant to save the users from having to deal with the JSON structure directly. We saw ourselves begin writing operations that used add/rm to add/remove multi-value attributes and set/unset to set/unset single value attributes and/or relationship.

Performance Friendly Operations

To minimize performance impacts our high-level operations that may bring to the overall solution, we started constructing JSON manipulation operations to allow accumulation of a series of operations before submitting to the create/update operations that actual address the API endpoints. The set/unset/add/rm mentioned above then has their own jset/junset/jadd/jrm counter parts. Likewise, save.sh can actually have a jsave.sh version where it only gives us the JSON formatted data while the save.sh continues to invoke the create.sh and becomes:

!/bin/bash
 MYPATH=$(dirname "$0")
 NAME=$1
 JSON=$$.json
 bash "${MYPATH}/jsave.sh" "$NAME" > "$JSON" || exit $?
 bash "${MYPATH}/create.sh" "$JSON"

Miscellaneous Lessons

From our own experiences, we also identified the following list of interesting subjects we had to work through:

  • The JSON output from the base API endpoints could use wrappers that needed to be removed before further processing.
  • The existence of anonymous objects could be relatively difficult to locate and manipulate.
  • The implementation of the API update operation would sometimes impact what to include in the JSON data submitted to it. This would further impact all high-level operations that use update to carry out their operations. Depending on the implementation, the JSON data could either include all attributes for the target object to receive or only the attributes that are to be modified. For example, the copy.sh mentioned previously would actually need to include additional instructions to have the attributes that do not exist in the source object removed.
  • Inconsistent API implementation could disrupt an otherwise graceful framework

Putting Them Together

To demonstrate the result of our work, the following is a snippet of a bash shell script we now use to automate the task of enabling the access of a Layer7 SiteMinder Access Gateway Web UI. Within the script, we use:

  • Directory structure to contains objects like SiteMinder Agent and Domain to group operations for each object type
  • The exist.sh to demonstrate the checking of the existence of an object before the attempt to create it
  • Template file dir.temp to encapsulate all the details of creating a SiteMinder user directory details
  • Environment variables to naturally link operations from different objects together
  • The add-type operations to add multi-value attributes to an object
  • The set-type operation to set single-value attribute to an object
  • JSON data accumulation operations like jaddvar and jsetexp to minimize the frequency of using direct API calls and improve the overall performance 
!/bin/bash
 MYPWD=$(pwd)
 cd ${HOMEDIR}
 check if the SiteMinder Access Gateway is actually valid
 SMAGENT=sps-01
 if ! EXIST=$(bash SmAgents/exist.sh "$SMAGENT"); then
     >2 echo Access Gateway agent name $SMAGENT may be incorrect
     exit 1
 fi
 SMDOMAIN=DOMAIN-SPSADMINUI-${SMAGENT}
 if ! EXIST=$(bash SmDomains/exist.sh "$SMDOMAIN"); then
     2> echo "$SMDOMAIN does not exist,"
     2> echo "Access Gateway configuration may not have been run"
     exit 1
 fi
 echo "$EXIST"
 create user directory object if not yet exists
 SMDIR="CADirectory"
 SMLDAPHOST=$(hostname)
 SMLDAPPORT=20589
 SMDIRTEMP=${MYPWD}/dir.temp
 JSON=$$.json
 if EXIST=$(bash SmUserDirectories/exist.sh "$SMDIR"); then
     echo "$EXIST"
 else
     bash "$SMDIRTEMP" "$SMDIR" "$SMLDAPHOST" "$SMLDAPPORT" >"$JSON"
     bash SmUserDirectories/create.sh "$JSON"
 fi
 …
 add User Directory to Domain
 bash SmDomains/adduser.sh "$SMDOMAIN" "$SMDIR"
 add Variables to Domain
 SMVARNAME1=DeptName1
 SMATTRNAME=departmentNumber
 JSON=$$.json
 if EXIST=$(bash SmVariables/exist.sh "$SMDOMAIN" "$SMVARNAME1"); then
     echo "$EXIST"
 else
     bash SmVariables/temp/attrs255.temp "$SMVARNAME1" "$SMATTRNAME" > "$JSON"
     bash SmVariables/create.sh "$SMDOMAIN" "$JSON"
 fi
 … 
 SMPOLICY=POLICY-SPSADMINUI-${SMAGENT}
 … 
 update the policy to allow only users have the privileges
 bash SmPolicies/read.sh  "$SMDOMAIN" "$SMPOLICY" | \
     bash SmPolicies/jaddvar.sh "$SMDOMAIN" "$SMVARNAME1" |
     bash SmPolicies/jaddvar.sh "$SMDOMAIN" "$SMVARNAME2" |
     bash SmPolicies/jsetexp.sh "($SMVARNAME1==$SMVARNAME2)" > "$JSON"
 bash SmPolicies/update.sh "$SMDOMAIN" "$JSON"

Last but not least, we want to point out that Swagger UI has successfully demonstrated the capability to translate a REST API specification into curl command examples. Therefore, it is perceivable that a general-purpose utility can also take the same specification to generate the base API client SDK using this proposed framework, at least as a starting point. We eagerly await additional progress on this attempt to democratize API access to include sysadmins and others.