Logging and Monitoring

• View the ACD logs by configuring a logging dashboard
• View pod status and logs
• Log in to a pod to investigate its status
• Enabling ACD prometheus metrics

Configuring a logging dashboard

A note about tenant and correlation identifiers in ACD logs

• correlationId: a UUID used to correlate all log entries for an ACD invocation across all annotators. This can be helpful in performing root cause analysis when problems occur.
• tenantId: The unique identifier for a specific tenant if ACD is being utilized in a multi-tenant manner. In a single tenant environment it will always be “defaultTenant”.

Using the OpenShift cluster logging operator

• To filter out logs for automated verification testing that occurs during pod startup, add NOT "\"correlationId\"\:\"junit-*" to the query string.
• If your cluster contains multiple deployments of ACD in different namespaces, add AND kubernetes.namespace_name:"<namespace>" to view the logs for only one deployment.
• To view logs filtered by correlationId, include "\"correlationId\":\"<correlation_id>\"".
• In a multi-tenant ACD deployment, add "\"tenantId\":\"<tenant_id>\"" to see only log entries related to a specific tenant.

Enabling JSON logging for OpenShift Container Platform

Prerequisites

1. Access to Red Hat OpenShift Container Platform
2. In your OpenShift project, make sure that you install below operators: a. Red Hat OpenShift logging operator b. OpenShift Elasticsearch operator

• You need to ensure that the OpenShift Logging Operator can parse the JSON data correctly. JSON parsing is possible as of version 5.1 of this operator. You only need to deploy a custom ClusterLogForwarder resource. This will overwrite the Fluentd pods and provide the configuration needed to parse JSON logs. Log in to your OpenShift platform to create cluster log forwarder as shown below:

• As shown in the above image, once you choose to create Cluster Log Forwarder, select the yaml view radio button and paste the below configuration:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputDefaults:
    elasticsearch:
      structuredTypeKey: kubernetes.labels.app_kubernetes_io/part-of
Copy to clipboardCopied!

• structuredTypeKey (string, optional) is the name of a message field. The value of that field, if present, is used to construct the index name.
• The value of structuredTypeKey prefixes with “kubernetes.labels.key”. In this case, the value of “key” is “app_kubernetes_io/part-of”.
• In the above snippet of code, we are making use of structuredTypeKey to create index in Kibana. The new index will be created as “app-{app_kubernetes_io/part-of}“.
• In the above case, the value of “app_kubernetes_io/part-of” is “merative-acd”. The index will be created as “app-merative-acd”.
• Once the new index is created using the Custom Log Forwarder, log in to Kibana and create the index pattern with the name matching as “app-merative-acd-*” as shown below:
• Once you browse to the discover screen, select the index pattern you created above and you will be able to find the logs inside message fields coverted to JSON prefixed as “structured” fields as shown in below:
• As the logs are now converted to JSON, you can use the fields in the visualizations/dashboards as per the requirement.
• Here is the Custom Dashboard that can be useful to analyze your data:

[
  {
    "_id": "1bc00b00-72f4-11ec-8b80-f979ac279214",
    "_type": "dashboard",
    "_source": {
      "title": "ACD CE Dashboard",
      "hits": 0,
      "description": "",
      "panelsJSON": "[{\"gridData\":{\"x\":0,\"y\":0,\"w\":24,\"h\":15,\"i\":\"1\"},\"version\":\"6.8.1\",\"panelIndex\":\"1\",\"type\":\"visualization\",\"id\":\"41c2c050-5782-11ec-b7f6-83b6c3cdab1d\",\"embeddableConfig\":{}},{\"gridData\":{\"x\":24,\"y\":0,\"w\":24,\"h\":15,\"i\":\"2\"},\"version\":\"6.8.1\",\"panelIndex\":\"2\",\"type\":\"visualization\",\"id\":\"4273e080-5785-11ec-b7f6-83b6c3cdab1d\",\"embeddableConfig\":{}},{\"gridData\":{\"x\":0,\"y\":15,\"w\":24,\"h\":15,\"i\":\"3\"},\"version\":\"6.8.1\",\"panelIndex\":\"3\",\"type\":\"visualization\",\"id\":\"3197dbc0-5787-11ec-b7f6-83b6c3cdab1d\",\"embeddableConfig\":{}},{\"gridData\":{\"x\":24,\"y\":15,\"w\":24,\"h\":15,\"i\":\"4\"},\"version\":\"6.8.1\",\"panelIndex\":\"4\",\"type\":\"visualization\",\"id\":\"a735b160-578a-11ec-b7f6-83b6c3cdab1d\",\"embeddableConfig\":{}},{\"gridData\":{\"x\":0,\"y\":30,\"w\":24,\"h\":15,\"i\":\"5\"},\"version\":\"6.8.1\",\"panelIndex\":\"5\",\"type\":\"visualization\",\"id\":\"050ed340-5784-11ec-b7f6-83b6c3cdab1d\",\"embeddableConfig\":{}}]",
Copy to clipboardCopied!

Using IBM Log Analysis on a Red Hat OpenShift on IBM Cloud Cluster (ROKS)

• To filter out logs for automated verification testing that occurs during pod startup, add -mdc.correlationId:junit to the query string.
• If your cluster contains multiple deployments of ACD in different namespaces, add namespace:<namespace> to view the logs for only one deployment.
• To view logs filtered by correlationId, include mdc.correlationId:<correlation_id>.
• In a multi-tenant ACD deployment, add mdc.tenantId:<tenant_id> to see only log entries related to a specific tenant.

Other logging solutions

View pod status and logs

Log in to a pod

Enabling and Configuring ACD prometheus metrics

• Read OpenShift monitoring overview

  https://docs.openshift.com/container-platform/4.12/monitoring/monitoring-overview.html

• Enable OpenShift user-defined monitoring in the ACD namespace

  https://docs.openshift.com/container-platform/4.12/monitoring/enabling-monitoring-for-user-defined-projects.html

• The promethus configuration for an ACD instance can be modified by editing the PodMonitor resource in the ACD namespace. The polling interval is the most likely parameter to be changed. Prometheus metrics gathering of a specific ACD instance can also be disabled by deleting the PodMonitor resource in that namespace.
• NOTE: You must change the prometheus.createPodMonitor parameter in the ACD operator yaml instance to false before the PodMonitor object can be modified or deleted. This will not delete the PodMonitor resource if it already exists.
• Example prometheus config section in the Acd resource instance yaml:

  "prometheus": {
    "createPodMonitor": false,
    "scrape": true
  },
  Copy to clipboardCopied!

• The ACD PodMonitor resource can be edited from the OpenShift UI by searching for the PodMonitor resource in the namespace where ACD is installed.
• Example default ACD PodMonitor configuration

  apiVersion: monitoring.coreos.com/v1
  kind: PodMonitor
  metadata:
    name: merative-acd-prometheus-monitor
    namespace: <acd namespace>
    labels:
      app.kubernetes.io/instance: merative-acd-prometheus-monitor-acd-instance
      app.kubernetes.io/name: merative-acd-prometheus-monitor
      app.kubernetes.io/part-of: merative-acd
  Copy to clipboardCopied!

ACD Metrics

Example prometheus ACD queries

• Request rate by pod (requests per second, 5 minute sample)

  sum by(pod)(rate(clinical_data_annotator_api_calls_count_total[5m]))
  Copy to clipboardCopied!

• Request rate by pod with namespace filter. Use this filter if you have multiple instances of ACD installed.

  sum by (pod)(rate(clinical_data_annotator_api_calls_count_total{namespace="merative-acd-operator-system"}[5m]))
  Copy to clipboardCopied!

• Total request rate

  sum(rate(clinical_data_annotator_api_calls_count_total[5m]))
  Copy to clipboardCopied!

• Average request size

  avg(clinical_data_annotator_api_request_size_bytes)
  Copy to clipboardCopied!

• Total request size

  sum(clinical_data_annotator_api_request_size_bytes)
  Copy to clipboardCopied!

• Concurrent requests by pod

  sum by(pod)(clinical_data_annotator_api_concurrency_count)
  Copy to clipboardCopied!

• Total concurrent requests

  sum(clinical_data_annotator_api_concurrency_count)
  Copy to clipboardCopied!

• Response count by return code

  sum by (acd_api_rc)(clinical_data_annotator_api_calls_count_total)
  Copy to clipboardCopied!

• Total response count with 5xx return codes

  sum by (acd_api_rc)(clinical_data_annotator_api_calls_count_total{acd_api_rc=~"5.."})
  Copy to clipboardCopied!

• Average response time by uri

  avg by (acd_api_resource)(clinical_data_annotator_api_time_seconds)
  Copy to clipboardCopied!