Keycloak and Kong Analysis

Does Kong go to the DB for every jwt token validation 

No, One consumer can have multiple keys and it's going to DB only once per key and later use the values from cache. 

Kong tables

Consumers table 

Jwt secrets table

Cached jwtauth_credential_key

Cached consumer_key

Request authentication in kong

JWT

• Once the request reaches to JWT plugin, it reads jwt token sent with a request and decodes it to get the key.
• If the key is valid, then it gets secret mapped to key from the cache.
• If the key and secret mapping don't exist in the cache and it read it from DB and cache it for next use. 
• based on the secret it verifies the JWT signature and if it does then it Verify the JWT registered claims ( e.g expiry of the token)
•  Finally, it checks if the consumer available for the key in the cache.
• If the entry doesn't exist in the cache and it's read if from DB and cache it. 

Does Kong use a master key to create to the secret for every device (mobile device) so that it is able to just cache this master key and validate the tokens for each device

-> No, Kong does not hold any master key but it's upon the API request that used to register and get the secret key for mobile device consumer. Kong creates a secret key against each unique key and sends it back to the requester. If the key is already registered then it just returns the secret key mapped to it.

curl -X POST \
https://dev.open-sunbird.org/api/api-manager/v1/consumer/mobile_device_openrap/credential/register \
-H 'Authorization: bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiI3Y2MzZDA2MDcyN2Q0Y2FmOTFmNTM4ODAzNjM4YzZmOSJ9.QVEpbyeV2AEXhNgA1xBIlxK75VGSN-w4eyEPiVZu64o' \
-H 'Content-Type: application/json' \
-d '{
"request": {
"key": "test-0.2.1"
}
}'

Response

{"id":"ekstep.api.am.adminutil.consumer.create","ver":"1.0","ets":1543319108287,"params":{"status":"successful","err":null,"errmsg":null,"msgid":"","resmsgid":"69ff1a9f-5216-4cd3-83bb-36b419593529"},"result":{"key":"test-0.2.1","secret":"5afdd8e574f144c2adba9923008394a4"}}

How can we purge old records of rate limits from Kong

I think we can directly truncate table(`TRUNCATE TABLE ratelimiting_metrics`). We have already changed rate-limiting config policy to local for all the API's so truncating this table has no effect anywhere else.

Does keycloak go to the DB for every session validation

@Todo Need in-depth analysis- But I think no, Keycloak cache the users and sessions data locally on every node.

Ref
- https://www.keycloak.org/docs/3.0/server_development/topics/user-storage/cache.html
- https://www.keycloak.org/docs/3.0/server_installation/topics/cache.html

How can we purge keycloak sessions that have expired for DB housekeeping

- @Todo - need in-depth analysis for this but I think record from session table automatically getting purged after user/client logout.

Database growth rate - you can get the data from dev & prod from devops team to understand this
List of key tables in keycloak and kong and their current row counts

- Kong DB- https://docs.google.com/spreadsheets/d/1mvk2L-uXlfxgJuFg_VgSBaC7NPe3w8FPr67qZMZh6bw/edit#gid=1513347828
- Keycloak DB- https://docs.google.com/spreadsheets/d/1EsjGKAF1NXd34lAsWVbqpUg4JWv5QjEzXu0au94ePfU/edit#gid=193179170

How do we enable time limits for jwt tokens to expire and how will the auth work in that scenario. Will Kong return 401 in that case?

We can patch an existing jwt plugin: This adds verification for both nbf and exp claims:

curl -X PATCH http://kong:8001/plugins/{jwt plugin id} \
--data "config.claims_to_verify=exp,nbf"

Migrations

After adding this claim, old issued jwt token will not work and it will throw 403 error. We will need to re-issue the new jwt tokens.

Kong cache

kong.cache is backed by kong.mlcache, which uses resty.lrucache for Lua VM cache and ngx.shared.DICT for shm cache. 

Tuning the kong_cache size

The Kong cache size can be tuned via the mem_cache_size configuration property. (See how the template injects it here).

Can we assign unlimited memory to kong?

The shared memory zones allocated by Nginx are not bounded by this limit, and we are free to assign very large areas of memory! More details on the ngx_lua documentation for lua_shared_dict. 

Cache for rate limiting plugin

As we are using the rate-liming to config policy as local and this plugin configured per consumer so it needs more cached memory.

How to estimate the amount of cache memory needed?

• There exist tools to monitor shared memory usage for OpenResty that we can leverage: https://github.com/openresty/stapxx#ngx-lua-shdict-info. 

Note: As of now there is no inbuilt cache memory analysis tool available for  Kong open source/community. 

For the enterprise, there is bundled charts and information on cache size/ + hits/misses, for open source/community there is none provided by Kong or created by 3rd party community members.

Ref:  https://discuss.konghq.com/t/how-to-do-cache-analysis/2409/2

What if cache memory full?

When it fails to allocate memory for the current key-value item, then set will try removing existing items in the storage according to the Least-Recently Used (LRU) algorithm. Note that, LRU takes priority over expiration time here. If upto tens of existing items have been removed and the storage left is still insufficient (either due to the total capacity limit specified by lua_shared_dict or memory segmentation), then the err return value will be no memory and success will be false.

Conf file: local/share/lua/5.1/kong/templates/nginx_kong.lua

lua_shared_dict kong 4m;
lua_shared_dict cache ${{MEM_CACHE_SIZE}};

lua_shared_dict- https://github.com/openresty/lua-nginx-module#lua_shared_dict

Ref: https://github.com/openresty/lua-nginx-module#ngxshareddictset

Ref- https://discuss.konghq.com/t/how-to-tune-the-kong-cache-size-and-the-proxy-cache-size/1939

The master key to sign different JWT tokens

→ Haven't found the way. @Todo

Steps for Diksha consumer cleanup

• As of now, there are 10 consumers exist in Diksha Prod.
  • If we want then we can delete them by removing their entries from the Kong Diksha prod inventory
  • And after that run the Consumer onboard job

Does keycloak go to the DB for every session validation

→ No,  Keycloak cache using the cached data for this.

→ Ref

• https://www.keycloak.org/docs/3.0/server_development/topics/user-storage/cache.html
• https://www.keycloak.org/docs/3.0/server_installation/topics/cache.html

How can we purge keycloak sessions that have expired for DB housekeeping

→ Keycloak dose it owns. It stores only offline session data in DB and purges it when the session expires. 

→ Ref - https://www.keycloak.org/docs/3.0/server_installation/topics/cache/eviction.html