Importing Endpoint CertificatesOrganizations can have multiple environments like Development, Testing, QA, Staging, Production with own instance of API Managers. These environments often have different configurations for APIs.

WSO2 API Manager supports creating fully automated CI/CD pipelines for APIs. It supports a variety of features including maintaining the lifecycle of API, changing environment-specific configurations throughout the CI/CD process. This allows seamlessly migrating APIs across different environments reducing human intervene on the migration process.

Overview

CI/CD for API Manager relies on a Version Control system which acts as a Single Source of Truth for the pipeline. After APIs are exported from one environment, promoting it to the other environment is done via the CLI tool for API Manager known as apimcli. It is capable of handling environment-related configurations and can promote the API seamlessly to other environments via a single command.

Preparing Environments

Running the CLI tool

Step 1 - Deploy the API import/export tool
Step 2 - Optionally, set the proxy environment variables for CLI
Step 3 - Run the CLI tool

Step 1 - Deploy the API import/export tool

Download the latest version of WSO2 API Manager from http://wso2.com/products/api-manager/.
Start WSO2 API Manager.
Download the latest WSO2 API import/export tool (api-import-export-2.6.0-v17.war) from here.
- Note that the import/export tool attached is specific to this version of WSO2 API Manager.
- Make sure to delete all previous versions of the web app (e.g., api-import-export-2.6.0-v17.war), both the compressed and the extracted files, before copying over the new web app.
- This version of the API Import and Export web application is compatible with the WSO2 WUM updates from July 8, 2019, onwards. For more information on updating WSO2 API Manager, see Updating WSO2 API Manager.
- The previous version of the web application which is compatible with WUM updates that are prior to the above-mentioned date can be download from here.
- Note that API update support in the targeted environment is available only with the api-import-export-2.6.0-v14.war version onwards.
Copy the downloaded api-import-export-2.6.0-v17.war file to the <API-M_HOME>/repository/deployment/server/webapps folder.
The file is automatically deployed as hot deployment is enabled.

Step 2 - Optionally, set the proxy environment variables for CLI

You can set proxy related HTTP_PROXY, HTTPS_PROXY, http_proxy, and https_proxy standard environment variables, with or without basic authentication as shown below to send the requests initiated from CLI via a proxy server. After one of the following environment variables is set in your environment where CLI is used, all the requests will go through the proxy server specified.

export HTTP_PROXY="http://<host-name>:<port>"
export HTTPS_PROXY="https://<host-name>:<port>"
export http_proxy="http://<host-name>:<port>"
export https_proxy="https://<host-name>:<port>"

export HTTP_PROXY="http://<username>:<password>@<host-name>:<port>"
export HTTPS_PROXY="https://<username>:<password>@<host-name>:<port>"
export http_proxy="http://<username>:<password>@<host-name>:<port>"
export https_proxy="https://<username>:<password>@<host-name>:<port>"

export HTTP_PROXY="http://localhost:3128"
export HTTPS_PROXY="https://localhost:3128"
export http_proxy="http://localhost:3128"
export https_proxy="https://localhost:3128"

export HTTP_PROXY="http://testuser:[email protected]:3128"
export HTTPS_PROXY="https://testuser:[email protected]:3128"
export http_proxy="http://testuser:[email protected]:3128"
export https_proxy="https://testuser:[email protected]:3128"

In conjunction with the above proxy configurations, calls from the CLI to particular hosts that need to bypass proxies can be configured via either the NO_PROXY or no_proxy environment variables.

export NO_PROXY="https://<host-name>:<port>"
export no_proxy="https://<host-name>:<port>"

export NO_PROXY="https://localhost:3128"
export no_proxy="https://localhost:3128"

Step 3 - Run the CLI tool

Download API Controller based on your preferred platform (i.e., Mac, Windows, Linux).
Extract the downloaded archive containing the CLI Tool to a preferred location and navigate to it from the terminal (using cd ).
Navigate to the working directory where the executable CLI Tool resides.
Execute the following command to start the CLI tool.
```
./apimcli
```
Add the location of the extracted folder to your system's $PATH variable to be able to access the executable from anywhere.
For further instructions execute the following command.
apimcli --help

Global flags for CLI tool

The following are some global flags that you can use with the CLI tool.

      --verbose
           Enable verbose logs (Provides more information on execution)
      --insecure, -k
          Allow connections to SSL sites without certs
      --help, -h
          Display information and example usage of a command

Adding an Environment

Then you need to install apimcli tool from here. After downloading the correct version for your OS, add the CLI to the $PATH of the system to access CLI tool from anywhere.

Once the tool installed you can introduce API Manager environments using add-env command.
For example, you can configure an API Manager running on port 9443 as your development(dev) environment by issuing the following command.

apimcli add-env -n dev \
                      --registration https://localhost:9443/client-registration/v0.14/register \
                      --apim https://localhost:9443 \
                      --token https://localhost:8243/token \
                      --import-export https://localhost:9443/api-import-export-{version} \
                      --admin https://localhost:9443/api/am/admin/v0.14 \
                      --api_list https://localhost:9443/api/am/publisher/v0.14/apis \
                      --app_list https://localhost:9443/api/am/store/v0.14/applications

Let’s add another environment for production purposes

apimcli add-env -n prod \
                      --registration https://localhost:9444/client-registration/v0.14/register \
                      --apim https://localhost:9444 \
                      --token https://localhost:8244/token \
                      --import-export https://localhost:9444/api-import-export-{version} \
                      --admin https://localhost:9444/api/am/admin/v0.14 \
                      --api_list https://localhost:9444/api/am/publisher/v0.14/apis \
                      --app_list https://localhost:9444/api/am/store/v0.14/applications

Exporting APIs with CLI

Now we have to add two different environments. Our end goal is to automate the API migration between dev and prod environments. Publish an API in dev environment using Publisher UI of API Manager. If you don’t know how to follow Quick Start Guide to learn how to deploy an API in the API Manager.

For this example, we are gonna use the Swagger Petstore API.

Start creating it by using “Add New API” and selecting an existing API option. Then create the API and set its name as SwaggerPetstore and Version as 1.0.0

2. Next, put backend URLs for Production and Sandbox and next page Publish the API.

For this example, we can use

Production: http://dev.wso2.com

Sandbox: http://dev.sandbox.wso2.com

Login to API Manager from CLI

Now before exporting this API as a project, we need to login to the API Manager from the CLI.

Sample:

apimcli login <Envionment> -u <Username> -p <Password>

Example:

apimcli login dev -u admin -p admin

Export an API

After successfully logged in we can export the API via CLI using a simple command,

Sample:

apimcli export-api -e <Env>  -n <APINAME> -v <VERSION> -r <PROVIDER> -k

Example:

apimcli export-api -e dev  -n SwaggerPetstore -v 1.0.0 -r admin -k

After providing name, version and provider you can see the API is downloaded as an archive to your machine.

We are using insecure(-k) flag because we are using self-signed certificates.

Preparing API Project for CI/CD

After exporting the API as an archive, extract the content. You will find a directory called SwaggerPetstore-1.0.0
Copy this directory into your Version Control Repository. Rename it to SwaggerPetstore to reference it easily.
Next, when we promoting API to the production environment we want to have different backend URLs for Production and Sandbox. For easily do this CLI tool supports a special file called api_params

Adding environment-specific parameters for APIs

Create a file called api_params.yaml inside your SwaggerPetstore directory. Open this api_params.yaml file in your favourite text editor.
Adding environment-specific parameters for APIs is done via this file. We are going to change backends in the production environment to point for production endpoints
```
environments: 
  - name: dev 
    endpoints: 
        production: 
            url: http://dev.wso2.com
        sandbox:
            url: http://dev.sandbox.wso2.com
  - name: prod
    endpoints:
        production:
            url: http://prod.wso2.com
        sandbox:
            url: http://prod.sandbox.wso2.com
```
As you can see we have defined prod.wso2.com and prod.sandbox.wso2.com as backend URLs for the production environment.
Setup Multiple Endpoints
Setup Load Balancing Endpoints
To setup load balancing endpoints, you need to specify the field loadBalanceEndpoints instead of the usual endpoints field in the api_params.yaml file. Under the loadBalanceEndpoints, you can specify the following fields.
- production: An array which consists the multiple production endpoints
- sandbox: An array which consists the multiple sandbox endpoints
- sessionManagement: Values can be "none", "transport", "soap", "simpleClientSession" and if not specified the default value would be "transport"
- sessionTimeout: The number of milliseconds after which the session would time out
Example api_params.yaml for load balancing
environments: - name: dev endpoints: production: url: http://dev.wso2.com sandbox: url: http://dev.sandbox.wso2.com - name: prod loadBalanceEndpoints: production: - url: http://prod.wso2.com - url: http://prod.next.wso2.com - url: http://prod.later.wso2.com sandbox: - url: http://prod.sandbox.wso2.com - url: http://prod.next.sandbox.wso2.com
Setup Failover Endpoints
To setup failover endpoints, you need to specify the field failoverEndpoints instead of the usual endpoints field in the api_params.yaml file. Under the failoverEndpoints, you can specify the following fields.
- production: The primary production endpoint (not an array)
- sandbox: The primary sandbox endpoint (not an array)
- productionFailovers: An array which consists of failover production endpoints
- sandboxFailovers: An array which consists failover sandbox endpoints
Example api_params.yaml for failover
environments: - name: dev endpoints: production: sandbox: - name: prod failoverEndpoints: production: url: http://prod.wso2.com productionFailovers: - url: http://prod.next.wso2.com - url: http://prod.later.wso2.com sandbox: url: http://prod.sandbox.wso2.com sandboxFailovers: - url: http://prod.sandbox.next.wso2.com
Advanced Endpoint Configurations
Setting Up Advanced Endpoint Configurations
You can set up advanced configurations related to endpoints by specifying the field config for the corresponding endpoint URL as shown with the below fields. (Note that, the only supported values for actionSelect is "discard" and "fault")
Example for setting up advanced endpoint configurations in the api_params.yaml
environments: - name: prod endpoints: production: url: http://prod.wso2.com config: factor: 3 retryTimeOut: 13000 retryDelay: 23000 retryErroCode: 101504 actionSelect: discard actionDuration: 75000 suspendErrorCode: 101504 suspendMaxDuration: 25000 suspendDuration: 45000 sandbox: url: http://prod.sandbox.wso2.com
Import Certificates
Importing Endpoint Certificates
You can import endpoint certificates using the api_params.yaml file by specifying the field certs. This field is an array where each item contains the following fields.
- host: The endpoint URL which the certificate belongs to
- alias: Alias for the certificate
- path: File path to locate the certificate file
Example api_params.yaml to import endpoint certificates
environments: - name: dev endpoints: production: sandbox: - name: prod endpoints: production: url: http://prod1.wso2.com sandbox: url: http://prod2.wso2.com certs: - host: https://prod1.wso2.com alias: Prod1 path: ~/.certs/prod1.pem - host: https://prod2.wso2.com alias: Prod2 path: ~/.certs/prod2.pem
Importing Mutual SSL Certificates
You can import Mutual SSL certificates using the api_params.yaml file by specifying the field mutualSslCerts. This field is an array where each item contains the following fields.
- tierName: Name of the tier of the certificate
- alias: Alias for the certificate
- path: File path to locate the certificate file
Example api_params.yaml to import mutual ssl certificates
environments: - name: dev endpoints: production: sandbox: - name: prod endpoints: production: url: http://prod1.wso2.com sandbox: url: http://prod2.wso2.com mutualSslCerts: - tierName: Unlimited alias: Prod1 path: ~/.certs/prod1.pem - tierName: Gold alias: Prod2 path: ~/.certs/prod2.pem
Endpoint Security
Setting Up Endpoint Security
You can set up endpoint security by defining the field security. Under the security field, you can specify the following fields.
- enabled: If this is set to true you need to specify the below attributes as well. If this is false, then non of the security parameters will be set. If the enabled attribute is not set (blank), then the security parameters in api.yaml file will be considered.
- type: Endpoint authentication type (can be either only basic or digest)
- username: Endpoint username
- password: Endpoint password
Example api_params.yaml to set up endpoint security
environments: - name: prod endpoints: production: url: http://prod1.wso2.com sandbox: url: http://prod2.wso2.com security: enabled: true type: basic username: admin password: admin
Subscription Policies/Tiers
Setting Up Subscription Policies for an API
You can set up subscription policies of an API by defining the field policies which is an array where you can list down the policy names that are needed to be enforced to the API. (Make sure that the policy/policies should be available in the environment. Otherwise, you will get an error.)
Example api_params.yaml to set up subscription policies of an API
environments: - name: prod endpoints: production: url: http://prod1.wso2.com sandbox: url: http://prod2.wso2.com policies: - Gold - Silver
Now we can commit this project to a version control system.

Importing API into a different environment

Now we can try out importing this API into the production environment and test if the API works. To do this issue
```
apimcli import-api -f ./SwaggerPetstore -e prod -k
```
(You need to login to prod environment before issuing the command)
With -f flag we point to the archive or project directory and with -e flag we provide target environment for the import process.

2. Now the tool will automatically detect the target environment and prepare a new artifact containing environment-related details.

3. You can investigate that the API has been imported with correct environment-specific details you defined and you can also see the API is in the published state too.

If you try again importing the same API to prod environment it will fail as API Manager does not allow updating existing APIs by default, it is to ensure that any accidental modification not happen.

But don’t worry, you can issue the same command with an additional flag called --update to update an existing API

apimcli import-api -f ./SwaggerPetstore -e prod --update -k

Now if you try again you can see it successfully imports the API

Automating API promotion with CI/CD Pipeline

Now you know building blocks of API import/export. By using the above commands you can create an automated pipeline for API promotion between environments.

After installing the CLI tool on the CI/CD server you can configure environments using add-env as we did earlier. As the next step, you can create a pipeline with your automation server.
For a reference, we are providing you with a Jenkins pipeline

Jenkinsfile
pipeline {
    agent any
    environment {
        CI = 'true'
        API = './SwaggerPetstore'
    }
    stages {
        stage('Deploy to Test') {
            environment{
                ENV = 'test'
            }
            steps {
                echo 'Deploying to Test'
                sh 'apimcli import-api -f $API -e $ENV -k --preserve-provider=false --update --verbose'
            }
        }
        stage('Deploy to Production') {
            environment{
                ENV = 'prod' 
            }
            steps {
                echo 'Deploying to Production'
                sh 'apimcli import-api -f $API -e $ENV -k --preserve-provider=false --update --verbose'
            }
        }
    }
}

You can see we have used an additional flag called --preserveProvider in the Jenkinsfile. It is used because if the User exported API does not exist in the target environment provider will be set to the current user who executes the import command.

Using environment variables

There are situations where hiding certain information is required in parameters file or reuse values across the script. You can define them as environment variables in the parameters file in usual notation

For example:

  - name: test
    endpoints:
      production:
        url: $TEST_BACKEND

When importing the CLI tool will pick these values from your shell environment and replace them.

An example project containing all these can be found here.

Building CICD Pipeline

Overview

Preparing Environments

Running the CLI tool

Step 1 - Deploy the API import/export tool

Step 2 - Optionally, set the proxy environment variables for CLI

Step 3 - Run the CLI tool

Global flags for CLI tool

Adding an Environment

Exporting APIs with CLI

Login to API Manager from CLI

Export an API

Preparing API Project for CI/CD

Adding environment-specific parameters for APIs

Setup Load Balancing Endpoints

Setup Failover Endpoints

Setting Up Advanced Endpoint Configurations

Importing Endpoint Certificates

Importing Mutual SSL Certificates

Setting Up Endpoint Security

Setting Up Subscription Policies for an API

Importing API into a different environment

Automating API promotion with CI/CD Pipeline

Using environment variables