Monitoring performance using JMX statistics

What is Cúram JMX?

Cúram JMX Kubernetes Limitations

• Access by using the SPM Administration Application is not supported because:
  • Not all pods have the requisite SPM client installed, which means JMS performance is unavailable using this access mechanism.
  • The reverse proxy nature of Kubernetes means that even for the pods with clients installed, identifying the correct target server would be difficult.

Cúram JMX timer-based statistics on Kubernetes

Cúram JMX statistics persisteance

-Dcuram.jmx.output_statistics_timer_enabled=true
-Dcuram.jmx.output_statistics_timer_folder=/tmp/jmx/
-Dcuram.jmx.output_statistics_timer_period=60000
Copy to clipboard

Considerations

• The default of 1 minute for the timer period might be too short because over an extended time, depending upon the amount of application activity, the persistent storage could be exhausted.

  • Consider also the length of time a pod will run; if the timer period is too short, data since the last timer period is lost.

• No timer-based statistics are generated until the first login or JMS message is processed.

• Turn off statistics for running pods by editing the configmaps where -Dcuram.jmx.output_statistics_timer_enabled=true is set, changing the value to false, and restarting the pods. For example:

  • Use the kubectl get configmaps command to list all the configmaps.
  • The configmaps you need to change are those named with *-jvm-override-* and the number and names of the configmaps vary depending on your configuration.
  • For each configmap, use the kubectl edit configmap command to change the curam.jmx.output_statistics_timer_enabled setting to false.
  • Use the kubectl delete pod command for each of the application producer and consumer pods that have a corresponding application name to the configmap.
  • On restart the new pod no longer generates statistics files.

• By default, statistics are accumulated over the life of a pod; therefore, the last file generated is a superset of all the other files for that pod, therefore and you should only need the latest file.

  • There is a property to change this behavior so each period resets the statistics, but then you would have to address how to amalgamate the data and make sense of it; so,changing this is not recommended.

• In support of EJB timers, IBM WebSphere Liberty dynamically creates three tables each time a pod starts; a PART, PROP, and TASK table.

  • The amount of data in these tables is small; but, over time could result in a large number of these tables in your database.
  • You should monitor these tables and drop those for pods that no longer exist.
  • These tables are prefixed with EJBTIMER_, followed by the pod name. For example, for the bugsbunny-apps-curam-consumer deployment, the following three tables were created for a single pod:

    EJBTIMER_BUGSBUNNY_APPS_CURAM_CONSUMER_786586FD5F_BGMGWPART
    EJBTIMER_BUGSBUNNY_APPS_CURAM_CONSUMER_786586FD5F_BGMGWPROP
    EJBTIMER_BUGSBUNNY_APPS_CURAM_CONSUMER_786586FD5F_BGMGWTASK
  Copy to clipboard

  In the context of Kubernetes pods, if the JMX and Liberty metrics are being made available from the pods by the JMX Exporter to be scraped by Prometheus, the following rules file may be employed to ensure that the metrics are surfaced and labelled appropriately.

  Example jmxExporter rules file