The WSO2 API Manager when in use, will store metadata and runtime data in its connected databases. For e.g., APIs, applications, subscriptions, and tokens created by users will be stored. Metadata related to applications and APIs are not been written to the databases frequently. However, since runtime data depends on different attributes such as the number of users, number of connected applications, and usage patterns, having a considerable load on the system will result in runtime data accumulating slowly over time. This will result in high data growth of the tables and in return negatively impact the system's performance.
Invalid access tokens, revoked access tokens, registry transaction-related logs, authorization codes, and user sessions are some of the runtime data that gets stored in these databases. A careful analysis of this data shows us that we do not always need to keep this data, other than for audit purposes. Hence, this data could be cleaned up periodically.
WSO2 API Manager provides two methods to do the cleanup.
|Table of Contents|
Note to writers:
This page has been duplicated in the WSO2 IS docs. If there are any changes to the content, please make the changes in the latest version of the IS docs as well or inform the IS writer of the change in content.
This cleanup is done within the product. It cleans up unused token related data during the runtime. This is an event-based cleaning where specific entries based on specific user actions are cleaned. For e.g., when an access token is revoked, this revoked token is taken from the access token table and put into the IDN_OAUTH2_ACCESS_TOKEN table. In addition to revoked tokens, inactive and expired tokens also accumulate in this table. This table is not used by the WSO2 API-M. These tokens are kept in the database for logging and audit purposes, but they can have a negative impact on the server's performance over time. Therefore, it is recommended to clean them.
From 2.6.0 onwards, WSO2 API Manager is configured by default to trigger token clean up during token generation, token refreshing, and token revocation. Therefore, when the state of the token (`TOKEN_STATE`) is changed during any of the latter mentioned processes for tokens that were in the `ACTIVE` state before, by default, such tokens will be removed from the IDN_OAUTH2_ACCESS_TOKEN table and stored in an audit table. Thus you don't need to manually clean up the unused tokens as guided below from API-M 2.6.0 onwards.
Configuring API Manager for token cleanup
WSO2 API Manager triggers token cleanup during the following instances.
To enable token cleanup, open the
<API-M_HOME>/repository/conf/identity/identity.xml file and do the following changes.
<!-- token cleanup feature config--> <TokenCleanup> <!-- old access token cleaning feature --> <EnableTokenCleanup>true</EnableTokenCleanup> <!-- old access token will be retain in audit table --> <RetainOldAccessToken>true</RetainOldAccessToken> </TokenCleanup>
Set this property to true to enable token cleanup.
Set it to false to disable token cleanup.
Set this property to true to move the old, invalid tokens to the Audit table when token cleaning is enabled.
Set it to false if you do not wish to store old tokens in the Audit table.
In this cleaning method, all the remaining token data, session data, and registry data can be cleaned up using separate stored procedures for each. The unused data is periodically analyzed and removed through a stored procedure that runs against the database. In a deep cleaning, each and every record is checked for the validity of the data. If unused or old data is detected, the stored procedure will clean them. There are three stored procedures provided that could be used to do the following three cleanups.
1. Token cleanup
2. Session cleanup
3. Registry cleanup
While the regular cleanup is good for regular housekeeping a hybrid approach is recommended for a production environment that removes all unused token, session, and registry data. While the regular cleanup will slow down unused token growth, deep cleaning will take care of the leftover unused data and prevent the tables from continuously growing, impacting performance.
Enable deep cleaning (token, session, and registry cleanup)
This will remove the old and invalid tokens, sessions and auth codes, which cannot be cleaned by the products inbuilt cleanup process.
Tip: It is safe to run these steps in read-only mode or during a time when traffic on the server is low, but that is not mandatory.
Take a backup of the running database.
Set up the database dump in a test environment and test it for any issues.
Tip: We recommend you to test the database dump before the cleanup task as the cleanup can take some time.
Depending on your database, select the appropriate token cleanup script from here and run it on the database dump. This takes a backup of the necessary tables, turns off SQL updates, and cleans the database of unused tokens.
Select the `token-cleanup` script to clean up the tokens, the `sessiondata-cleanup` script to cleanup the session data and the `registry-cleanup` script to clean up the registry unused data.
Once the cleanup is over, start the API Manager pointing to the cleaned-up database dump and test thoroughly for any issues.
You can also schedule a cleanup task that will automatically run after a given period of time. Here's an example:
Localtab Group Localtab active true title MySQL Code Block
USE 'WSO2AM_DB'; DROP EVENT IF EXISTS 'cleanup_tokens_event'; CREATE EVENT 'cleanup_tokens_event' ON SCHEDULE EVERY 1 WEEK STARTS '2018-01-01 00:00.00' DO CALL 'WSO2AM_DB'.'cleanup_tokens'(); -- 'Turn on the event_scheduler' SET GLOBAL event_scheduler = ON;
Localtab title SQL Server Code Block
USE WSO2AM_DB ; GO -- Creates a schedule named CleanupTask. -- Jobs that use this schedule execute every day when the time on the server is 01:00. EXEC sp_add_schedule @schedule_name = N'CleanupTask' , @freq_type = 4, @freq_interval = 1, @active_start_time = 010000 ; GO -- attaches the schedule to the job BackupDatabase EXEC sp_attach_schedule @job_name = N'BackupDatabase', @schedule_name = N'CleanupTask' ; GO
WSO2AM_DBwith the name of your API Manager database in the above script.