Importing Endpoint CertificatesOrganizations can have multiple environments like Development, Testing, QA, Staging, Production with own instance of API Managers. These environments often have different configurations for APIs.
WSO2 API Manager supports creating fully automated CI/CD pipelines for APIs. It supports a variety of features including maintaining the lifecycle of API, changing environment-specific configurations throughout the CI/CD process. This allows seamlessly migrating APIs across different environments reducing human intervene on the migration process.
Overview
CI/CD for API Manager relies on a Version Control system which acts as a Single Source of Truth for the pipeline. After APIs are exported from one environment, promoting it to the other environment is done via the CLI tool for API Manager known as apimcli. It is capable of handling environment-related configurations and can promote the API seamlessly to other environments via a single command.
Preparing Environments
Download the latest version of WSO2 API Manager from http://wso2.com/products/api-manager/. Download the latest WSO2 API import/export tool ( Note that the import/export tool attached is specific to this version of WSO2 API Manager. Make sure to delete all previous versions of the web app (e.g., Copy the downloaded You can set proxy related In conjunction with the above proxy configurations, calls from the CLI to particular hosts that need to bypass proxies can be configured via either the Download API Controller based on your preferred platform (i.e., Mac, Windows, Linux). Execute the following command to start the CLI tool. Add the location of the extracted folder to your system's For further instructions execute the following command. The following are some global flags that you can use with the CLI tool.Running the CLI tool
Step 1 - Deploy the API import/export tool
api-import-export-2.6.0-v17.war
) from here.api-import-export-2.6.0-v17.war
), both the compressed and the extracted files, before copying over the new web app.api-import-export-2.6.0-v14.war
version onwards.api-import-export-2.6.0-v17.war
file to the <API-M_HOME>/repository/deployment/server/webapps
folder.
The file is automatically deployed as hot deployment is enabled.Step 2 - Optionally, set the proxy environment variables for CLI
HTTP_PROXY
, HTTPS_PROXY
, http_proxy
, and https_proxy
standard environment variables, with or without basic authentication as shown below to send the requests initiated from CLI via a proxy server. After one of the following environment variables is set in your environment where CLI is used, all the requests will go through the proxy server specified.export HTTP_PROXY="http://<host-name>:<port>"
export HTTPS_PROXY="https://<host-name>:<port>"
export http_proxy="http://<host-name>:<port>"
export https_proxy="https://<host-name>:<port>"
export HTTP_PROXY="http://<username>:<password>@<host-name>:<port>"
export HTTPS_PROXY="https://<username>:<password>@<host-name>:<port>"
export http_proxy="http://<username>:<password>@<host-name>:<port>"
export https_proxy="https://<username>:<password>@<host-name>:<port>"
export HTTP_PROXY="http://localhost:3128"
export HTTPS_PROXY="https://localhost:3128"
export http_proxy="http://localhost:3128"
export https_proxy="https://localhost:3128"
export HTTP_PROXY="http://testuser:[email protected]:3128"
export HTTPS_PROXY="https://testuser:[email protected]:3128"
export http_proxy="http://testuser:[email protected]:3128"
export https_proxy="https://testuser:[email protected]:3128"
NO_PROXY
or no_proxy
environment variables.export NO_PROXY="https://<host-name>:<port>"
export no_proxy="https://<host-name>:<port>"
export NO_PROXY="https://localhost:3128"
export no_proxy="https://localhost:3128"
Step 3 - Run the CLI tool
cd
)../apimcli
$PATH
variable to be able to access the executable from anywhere.apimcli --help
Global flags for CLI tool
--verbose
Enable verbose logs (Provides more information on execution)
--insecure, -k
Allow connections to SSL sites without certs
--help, -h
Display information and example usage of a command
Adding an Environment
- Then you need to install apimcli tool from here. After downloading the correct version for your OS, add the CLI to the $PATH of the system to access CLI tool from anywhere.
Once the tool installed you can introduce API Manager environments using add-env command.
For example, you can configure an API Manager running on port 9443 as your development(dev) environment by issuing the following command.apimcli add-env -n dev \ --registration https://localhost:9443/client-registration/v0.14/register \ --apim https://localhost:9443 \ --token https://localhost:8243/token \ --import-export https://localhost:9443/api-import-export-{version} \ --admin https://localhost:9443/api/am/admin/v0.14 \ --api_list https://localhost:9443/api/am/publisher/v0.14/apis \ --app_list https://localhost:9443/api/am/store/v0.14/applications
Let’s add another environment for production purposes
apimcli add-env -n prod \ --registration https://localhost:9444/client-registration/v0.14/register \ --apim https://localhost:9444 \ --token https://localhost:8244/token \ --import-export https://localhost:9444/api-import-export-{version} \ --admin https://localhost:9444/api/am/admin/v0.14 \ --api_list https://localhost:9444/api/am/publisher/v0.14/apis \ --app_list https://localhost:9444/api/am/store/v0.14/applications
Exporting APIs with CLI
Now we have to add two different environments. Our end goal is to automate the API migration between dev and prod environments. Publish an API in dev environment using Publisher UI of API Manager. If you don’t know how to follow Quick Start Guide to learn how to deploy an API in the API Manager.
For this example, we are gonna use the Swagger Petstore API.
Start creating it by using “Add New API” and selecting an existing API option. Then create the API and set its name as SwaggerPetstore and Version as 1.0.0
2. Next, put backend URLs for Production and Sandbox and next page Publish the API.
For this example, we can use
Production: http://dev.wso2.com
Sandbox: http://dev.sandbox.wso2.com
Login to API Manager from CLI
Now before exporting this API as a project, we need to login to the API Manager from the CLI.
Sample:
apimcli login <Envionment> -u <Username> -p <Password>
Example:
apimcli login dev -u admin -p admin
Export an API
After successfully logged in we can export the API via CLI using a simple command,
Sample:
apimcli export-api -e <Env> -n <APINAME> -v <VERSION> -r <PROVIDER> -k
Example:
apimcli export-api -e dev -n SwaggerPetstore -v 1.0.0 -r admin -k
After providing name, version and provider you can see the API is downloaded as an archive to your machine.
We are using insecure(-k) flag because we are using self-signed certificates.
Preparing API Project for CI/CD
After exporting the API as an archive, extract the content. You will find a directory called SwaggerPetstore-1.0.0
Copy this directory into your Version Control Repository. Rename it to SwaggerPetstore to reference it easily.
Next, when we promoting API to the production environment we want to have different backend URLs for Production and Sandbox. For easily do this CLI tool supports a special file called api_params
Adding environment-specific parameters for APIs
Create a file called api_params.yaml inside your SwaggerPetstore directory. Open this api_params.yaml file in your favourite text editor.
Adding environment-specific parameters for APIs is done via this file. We are going to change backends in the production environment to point for production endpoints
environments: - name: dev endpoints: production: url: http://dev.wso2.com sandbox: url: http://dev.sandbox.wso2.com - name: prod endpoints: production: url: http://prod.wso2.com sandbox: url: http://prod.sandbox.wso2.com
As you can see we have defined prod.wso2.com and prod.sandbox.wso2.com as backend URLs for the production environment.
Setup Multiple Endpoints
Setup Load Balancing Endpoints
To setup load balancing endpoints, you need to specify the field loadBalanceEndpoints instead of the usual endpoints field in the api_params.yaml file. Under the loadBalanceEndpoints, you can specify the following fields.
- production: An array which consists the multiple production endpoints
- sandbox: An array which consists the multiple sandbox endpoints
- sessionManagement: Values can be "none", "transport", "soap", "simpleClientSession" and if not specified the default value would be "transport"
- sessionTimeout: The number of milliseconds after which the session would time out
Example api_params.yaml for load balancingenvironments: - name: dev endpoints: production: url: http://dev.wso2.com sandbox: url: http://dev.sandbox.wso2.com - name: prod loadBalanceEndpoints: production: - url: http://prod.wso2.com - url: http://prod.next.wso2.com - url: http://prod.later.wso2.com sandbox: - url: http://prod.sandbox.wso2.com - url: http://prod.next.sandbox.wso2.com
Setup Failover Endpoints
To setup failover endpoints, you need to specify the field failoverEndpoints instead of the usual endpoints field in the api_params.yaml file. Under the failoverEndpoints, you can specify the following fields.
- production: The primary production endpoint (not an array)
- sandbox: The primary sandbox endpoint (not an array)
- productionFailovers: An array which consists of failover production endpoints
- sandboxFailovers: An array which consists failover sandbox endpoints
Example api_params.yaml for failoverenvironments: - name: dev endpoints: production: sandbox: - name: prod failoverEndpoints: production: url: http://prod.wso2.com productionFailovers: - url: http://prod.next.wso2.com - url: http://prod.later.wso2.com sandbox: url: http://prod.sandbox.wso2.com sandboxFailovers: - url: http://prod.sandbox.next.wso2.com
Advanced Endpoint Configurations
Setting Up Advanced Endpoint Configurations
You can set up advanced configurations related to endpoints by specifying the field config for the corresponding endpoint URL as shown with the below fields. (Note that, the only supported values for actionSelect is "discard" and "fault")
Example for setting up advanced endpoint configurations in the api_params.yamlenvironments: - name: prod endpoints: production: url: http://prod.wso2.com config: factor: 3 retryTimeOut: 13000 retryDelay: 23000 retryErroCode: 101504 actionSelect: discard actionDuration: 75000 suspendErrorCode: 101504 suspendMaxDuration: 25000 suspendDuration: 45000 sandbox: url: http://prod.sandbox.wso2.com
Import Certificates
Importing Endpoint Certificates
You can import endpoint certificates using the api_params.yaml file by specifying the field certs. This field is an array where each item contains the following fields.
- host: The endpoint URL which the certificate belongs to
- alias: Alias for the certificate
- path: File path to locate the certificate file
Example api_params.yaml to import endpoint certificatesenvironments: - name: dev endpoints: production: sandbox: - name: prod endpoints: production: url: http://prod1.wso2.com sandbox: url: http://prod2.wso2.com certs: - host: https://prod1.wso2.com alias: Prod1 path: ~/.certs/prod1.pem - host: https://prod2.wso2.com alias: Prod2 path: ~/.certs/prod2.pem
Importing Mutual SSL Certificates
You can import Mutual SSL certificates using the api_params.yaml file by specifying the field mutualSslCerts. This field is an array where each item contains the following fields.
- tierName: Name of the tier of the certificate
- alias: Alias for the certificate
- path: File path to locate the certificate file
Example api_params.yaml to import mutual ssl certificatesenvironments: - name: dev endpoints: production: sandbox: - name: prod endpoints: production: url: http://prod1.wso2.com sandbox: url: http://prod2.wso2.com mutualSslCerts: - tierName: Unlimited alias: Prod1 path: ~/.certs/prod1.pem - tierName: Gold alias: Prod2 path: ~/.certs/prod2.pem
Endpoint Security
Setting Up Endpoint Security
You can set up endpoint security by defining the field security. Under the security field, you can specify the following fields.
- enabled: If this is set to true you need to specify the below attributes as well. If this is false, then non of the security parameters will be set. If the enabled attribute is not set (blank), then the security parameters in api.yaml file will be considered.
- type: Endpoint authentication type (can be either only basic or digest)
- username: Endpoint username
- password: Endpoint password
Example api_params.yaml to set up endpoint securityenvironments: - name: prod endpoints: production: url: http://prod1.wso2.com sandbox: url: http://prod2.wso2.com security: enabled: true type: basic username: admin password: admin
Subscription Policies/Tiers
Setting Up Subscription Policies for an API
You can set up subscription policies of an API by defining the field policies which is an array where you can list down the policy names that are needed to be enforced to the API. (Make sure that the policy/policies should be available in the environment. Otherwise, you will get an error.)
Example api_params.yaml to set up subscription policies of an APIenvironments: - name: prod endpoints: production: url: http://prod1.wso2.com sandbox: url: http://prod2.wso2.com policies: - Gold - Silver
Now we can commit this project to a version control system.
Importing API into a different environment
Now we can try out importing this API into the production environment and test if the API works. To do this issue
apimcli import-api -f ./SwaggerPetstore -e prod -k
(You need to login to prod environment before issuing the command)
With -f flag we point to the archive or project directory and with -e flag we provide target environment for the import process.
2. Now the tool will automatically detect the target environment and prepare a new artifact containing environment-related details.
3. You can investigate that the API has been imported with correct environment-specific details you defined and you can also see the API is in the published state too.
If you try again importing the same API to prod environment it will fail as API Manager does not allow updating existing APIs by default, it is to ensure that any accidental modification not happen.
But don’t worry, you can issue the same command with an additional flag called --update to update an existing API
apimcli import-api -f ./SwaggerPetstore -e prod --update -k
Now if you try again you can see it successfully imports the API
Automating API promotion with CI/CD Pipeline
Now you know building blocks of API import/export. By using the above commands you can create an automated pipeline for API promotion between environments.
After installing the CLI tool on the CI/CD server you can configure environments using add-env as we did earlier. As the next step, you can create a pipeline with your automation server.
For a reference, we are providing you with a Jenkins pipelineJenkinsfile pipeline { agent any environment { CI = 'true' API = './SwaggerPetstore' } stages { stage('Deploy to Test') { environment{ ENV = 'test' } steps { echo 'Deploying to Test' sh 'apimcli import-api -f $API -e $ENV -k --preserve-provider=false --update --verbose' } } stage('Deploy to Production') { environment{ ENV = 'prod' } steps { echo 'Deploying to Production' sh 'apimcli import-api -f $API -e $ENV -k --preserve-provider=false --update --verbose' } } } }
You can see we have used an additional flag called --preserveProvider in the Jenkinsfile. It is used because if the User exported API does not exist in the target environment provider will be set to the current user who executes the import command.
Using environment variables
There are situations where hiding certain information is required in parameters file or reuse values across the script. You can define them as environment variables in the parameters file in usual notation
For example:
- name: test endpoints: production: url: $TEST_BACKEND
When importing the CLI tool will pick these values from your shell environment and replace them.
An example project containing all these can be found here.