Monitoring XML servers

Introduction

• The XML server statistics, described in Monitoring XML server statistics
• The JVM that the XML server runs on, described in Monitoring the XML server JVM
• Health checks, described in XML server health checks

Monitoring XML server statistics

Sample implementation

• The Prometheus client_java is used for implementing the Prometheus collector and exporter, hence it is coded in Java.
• A sidecar container is used to run the Prometheus collector and exporter as this allows for:
  • Avoiding making any Kubernetes or Prometheus-specific code changes to the XML server code base. This is achieved by the use of persistent storage, which is described in more detail below
  • Avoiding additional processes in the xmlserver pod, which could impact on the main xmlserver process.

• Implementing a Prometheus collector and exporter to make the /opt/ibm/Curam/xmlserver/stats/ThreadPoolWorker-* file contents available to Prometheus
• Creating a Docker image for the sidecar container that contains the Prometheus collector and exporter
• Defining a sidecar container in the xmlserver deployment
• Defining the necessary Prometheus configuration for the Kubernetes cluster. The sample deployment provided utilizes the OpenShift PodMonitor.

The role of persistent storage in monitoring XML server statistics

• The xmlserver-metrics sidecar container has access to the same persistent file system as the xmlserver container.
• As the XML server writes statistics to the files in its ./stats folder those same statistics records are available to the xmlserver-metrics sidecar.
• No changes are required to the XML server code base or the xmlserver container.

• For the xmlserver-metrics container dockerfiles/Liberty/xmlserver-metrics/XMLServer-metrics.Dockerfile creates the /opt/ibm/Curam/xmlserver/ folder and the ./tmp and ./stats subfolders in its local file system.
• In helm-charts/xmlserver/templates/deployment.yaml the xmlserver-metrics container defines the same volumeMounts: as the xmlserver container and they both reference the deployment’s persistentVolumeClaim:.
• In the xmlserver-metrics container the start-prometheus_collector_exporter.sh startup shell script ensures the persistent mountpoint is available and creates symbolic links from the container’s file system to the persistent file system.

Prometheus collector and exporter

Choosing XML server statistics

• Success - reported as success or failure
• Job type - PDF, HTML, TEXT, and RTF
• Elapsed connection
• Job preview data length

Coding the Prometheus metric variables

  /** Prometheus counter - of xmlserver jobs. */
  private static final Counter jobTotals = Counter.build()
      .namespace("curam_xmlserver")
      .name("jobs_total")
      .help("Job counts with labels for status=success|fail ; type=PDF|HTML|TEXT|RTF")
      .labelNames("status", "type")
      .register();
Copy to clipboard

  /** Prometheus histogram - job connect time. */
  static final Histogram jobConnectTime = Histogram.build()
      .namespace("curam_xmlserver")
      .name("job_connect_seconds")
      .help("Job connect time")
      .buckets(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9,
               1.0, 1.5, 2.0, 2.5, 3.0, 4.0, 5.0) 
      .register();
Copy to clipboard

  /** Prometheus histogram - job data length. */
  static final Histogram jobDataLength = Histogram.build()
      .namespace("curam_xmlserver")
      .name("job_data_length_bytes")
      .help("Job preview data length")
      .buckets(5000, 7000, 9000, 10000, 12000, 14000, 15000, 
               20000, 25000, 30000, 35000, 50000)
      .register();
Copy to clipboard

Java code for the Prometheus collector and exporter

• Process optional arguments
  • -httpPort <port> - Port that Prometheus will listen on. Defaults to: 8080
  • -checkInterval <seconds> - Number of seconds between checks for data in the ./stats/ folder to check for termination. Defaults to: 10
• Start the collector thread
• Start the exporter thread

• Identify all ThreadPoolWorker-* files in the ./stats folder
• While the thread is not interrupted
  • Iterate over each file
    • If there is content
      • If the content has not been previously read
        • Create a java.io.RandomAccessFile in read mode
        • Bypass previously read content
        • Process each new line in the file
          • Call the io.prometheus.client.Counter.inc() method to increment status (success|fail) and type (PDF|HTML|TEXT|RTF)
          • Call the io.prometheus.client.Histogram.observe(double amt) method for elapsed time
          • Call the io.prometheus.client.Histogram.observe(double amt) method for bytes processed
        • Record the new number of bytes read from the file
    • Sleep for the specified interval

• Using the specified port create a new io.prometheus.client.exporter.HTTPServer.HTTPServer(int port)

Creating the Docker image for the sidecar container

• The .Dockerfile that builds the image: XMLServer-metrics.Dockerfile
• The sample proof-of-concept implementation executable, xmlserver_prometheus.jar, and source, xmlserver_prometheus-sources.jar
• The start shell script, ./xmlserver-metrics/start-prometheus_collector_exporter.sh, invoked by the xmlserver-metrics deployment’s cmd: structure and when invoked:
  • Ensures the persistent storage mount point is available and the expected directories exist.
  • Starts the Prometheus collector/exporter Java program in the background.
  • Waits for the Java program to terminate and logs the end of the process.
• The stop shell script, ./xmlserver-metrics/stop-prometheus_collector_exporter.sh, invoked by the xmlserver-metrics deployment’s lifecycle: structure and when invoked:
  • Cleans up the .pid file created when the Java process was started.

Building the Docker image

• Using RedHat Registry
• Unauthenticated / Local image
• Using RedHat Registry (Windows PS)
• Unauthenticated / Local image (Windows PS)

cd $SPM_HOME/dockerfiles/Liberty/
docker build \
     --tag $DOCKER_REGISTRY/$PROJECT/xmlserver-metrics:latest \
     --build-arg "BASE_REGISTRY=registry.connect.redhat.com/" \
     --file xmlserver-metrics/XMLServer-metrics.Dockerfile .
Copy to clipboard

cd $SPM_HOME/dockerfiles/Liberty/
docker build \
     --tag $DOCKER_REGISTRY/$PROJECT/xmlserver-metrics:latest \
     --file xmlserver-metrics/XMLServer-metrics.Dockerfile .
Copy to clipboard

cd $env:SPM_HOME\dockerfiles\Liberty\
docker build `
     --tag $env:DOCKER_REGISTRY/$env:PROJECT/xmlserver-metrics:latest `
     --build-arg "BASE_REGISTRY=registry.redhat.io/" `
     --file xmlserver-metrics\XMLServer-metrics.Dockerfile .
Copy to clipboard

cd $env:SPM_HOME\dockerfiles\Liberty\
docker build `
     --tag $env:DOCKER_REGISTRY/$env:PROJECT/xmlserver-metrics:latest `
     --file xmlserver-metrics\XMLServer-metrics.Dockerfile .
Copy to clipboard

Deploying the sidecar container and configuring for Prometheus

• global.apps.common.persistence.enabled: true
• global.useBetaFeatures: true
• xmlserver.metrics.enabled: true

• ./templates/deployment.yaml - defines the xmlserver-metrics sidecar
• ./templates/monitor-xmlserver.yaml - defines a PodMonitor

Confirming the status of the xmlserver-metrics container

Interacting with the XML server statistics

• Using the curl command in a pod’s shell you can get the current metrics (remember that with two containers your Kubernetes exec command must use -c xmlserver-metrics to enter the correct shell), for example:
  • curl -s http://localhost:8080/metrics - displays all the Prometheus data and help text
  • curl -s http://localhost:8080/metrics | grep "curam_xmlserver_jobs_total.status=.success" - displays the count of successful XML server jobs
  • curl -s http://localhost:8080/metrics | grep "curam_xmlserver_jobs_total.status" - displays the count of successful and failed jobs

Monitoring the XML server JVM

• global.useBetaFeatures: true
• xmlserver.jvmStats.enabled: true

• xmlserver.jvmStats.port: 8083
• xmlserver.jvmStats.configYaml: ''

• See the Prometheus JMX Exporter configuration documentation
• See this sample Grafana JVM dashboard

XML server health checks

readinessProbe

livenessProbe

[xmlserver] [INFO] [XMLConnectionHandler] Mon Apr 03 12:28:02 BST 2023 Connection received.
[xmlserver] [INFO] [XMLConnectionHandler] Awaiting job configuration.
[xmlserver] [INFO] [XMLConnectionHandler] Probe check received.
Copy to clipboard