Understanding your GHES graphs part 2 - Application server, background jobs, network, storage, and pre-receive hooks

[djdefi]

In part 1 of our 'Understanding your graphs' mini-series, we talked about GitHub Enterprise System, Process, and Authentication graphs. In part 2, we're going to talk about GitHub Enterprise Application server, background jobs, network, storage, and pre-receive graphs.

App servers

The application servers section provides insight into the activity of GitHub Enterprise services which provide data to users, or integrations.

Sessions

• Profile of active sessions connected to GitHub Enterprise backend services. This graph provides a summary of the volume and type of activity from users.
• Web unicorns sessions are often the largest portion of this graph, as users interact via the Web UI and API.

Errors

• High error rates may indicate a problem with a service, or potential saturation due to request volume.
• Please reach out to GitHub Enterprise Support if you regularly encounter errors on this graph.

Active Workers

• Service workers which are currently serving a request.
• User and integration daily activity trends are very visible in this graph.
• Plateaus for extended periods in this graph indicate worker saturation, and should be investigated for any request queueing.
• Worker counts automatically scale with system memory size at boot.

Queued Requests

• Values in this graph indicate that requests were required to wait for a worker process to become available before it was able to process and serve the request.
• If requests are constantly queuing, users will notice delays in responsiveness, as well as encounter errors or timeouts more frequently.
• Queued requests occurring regularly is a major indicator of an undersized appliance for the amount of incoming requests.

App request/response

The Application request / response section looks at the rate of requests, how quickly those requests are responded to, and with what status they returned.

Throughput

• Per minute request counts, broken down by type.
• API is typically the highest on systems with many integrations or active CI and project management tools.

Response time

• Reflects the speed of web requests at the 90th percentile in milliseconds.
• Times of over a few seconds can indicate a poor user experience due to long browser load times, or slow API responses.

CPU Time

• Time spent in Ruby garbage collection within the GitHub Enterprise web application.
• Plateaus for extended periods of GC time may indicate a problem with the GitHub Enterprise application itself.

I/O Time

• Time spent accessing disk IO by data services which GitHub Enterprise depends on.
• Plateaus for extended periods of time may indicate system resource saturation.

Response Code

• The number of responses per HTTP status code.
• 2xx successful status codes will normally be the largest.
• 401 Unauthorized codes will also be present in environments where API and Git over HTTP traffic is present, as initial requests from clients may not provide authentication headers.
• 500 statuses indicate a potential issue with the GitHub Enterprise application, and should be investigated with support.

Errors

• Represents the number of application exceptions generated per minute.
• High rates of errors may indicate an issue impacting the GitHub Enterprise application.

Background jobs

• Number of tasks queued for background processing on the GitHub Enterprise appliance.

Aqueduct

• Many user and application actions trigger jobs which run asynchronously on GitHub Enterprise, and are queued to be processed by Aqueduct.
• Workers which process the maint_git-serv queues are paused during GitHub Enterprise Backup Utilities snapshot runs. It is normal to see the number for this queue increase while a snapshot is in progress. The queue should then drain rather quickly once the snapshot run is complete.
• As there are a finite number of Aqueduct worker processes, queues which never drain to 0 may indicate resource saturation or in some cases jobs which have gotten stuck, requiring manual intervention to clear.
• Many queues simultaneously having hundreds or thousands of jobs pending can indicate resource saturation. Queue length can also be inspected from the SSH admin console by running ghe-aqueduct-info.

E-mail

• When E-mail for notifications is enabled, this graph displays the length of the onboard postfix mail queues.
• High numbers of deferred E-mail messages may indicate a problem with the configured SMTP server, or failures in mail delivery to specific user E-mail addresses.

Network

The network interface graphs can be useful in profiling user activity, and throughput of traffic in and out of the the GitHub Enterprise appliance.

Clients

• Breaks down the number of clients per TCP port, which is useful for examining how users are interacting with GitHub Enterprise.

Sockets

• Further details on the TCP connection state, which can be useful in troubleshooting network or Load Balancer issues in some cases.

Interface Throughput

• The amount of data transferred inbound and outbound from the GitHub Enterprise appliance.
• TX (outbound) traffic is most commonly higher than RX (inbound), especially when many systems are "polling" the API or Git repositories for changes.
• Plateaus in this graph can be an indication of link saturation or reaching the maximum possible link throughput.

Interface Errors

• The presence of any errors may indicate a problem with the physical or virtual network card, or cables connected to the Hypervisor host system.

Replication Throughput

• The amount of data sent to, and received by replica instances over the internal OpenVPN interface.

Replication Interface Errors

• Errors may occur here due to saturation or MTU problems on the physical link; however, these are generally not critical errors.

Storage

GitHub Enterprise repository performance is very dependent on the underlying storage system. Low latency, local SSD disks provide the highest performance. For more information on the GitHub Enterprise storage architecture, please see the System Overview guide on our documentation site.

Disk usage (Root Device)

• Disk space in bytes available for root volume storage.
• Growth on this volume is generally due to logging, which is on a 24 hour rotation schedule.
• The root volume reaching 100% usage can cause a system outage, or indicate a service issue which is causing extreme log growth.

Disk usage (Data Device dm-0)

• Disk space in bytes available for the user data volume.
• All user profile data, pull request and issue metadata, repositories, and release assets are stored on this device.
• The data volume reaching 85% usage will cause problems with the built in search functionality of GitHub Enterprise. It is recommended to increase storage capacity of the data volume prior to reaching 85% usage.

Disk latency (Root Device & Data Device dm-0)

• For best IO performance, average latency values below 10ms are recommended.
• Large spikes may be an indication of storage system saturation.

Disk operations (Root Device)

• Abnormally large amounts of time spent in root IO suddenly appearing may indicate a logging issue, or a general storage problem.

Disk operations (Data Device dm-0)

• Abnormally large amounts of time spent in data volume IO suddenly appearing may indicate a repository maintenance issue, or a general storage problem.
• Graph for reads trends generally follows the pattern of Git fetch or clone traffic on the system.

Disk pending operations (root Device)

• Pending disk operations on the root device may indicate storage system saturation for the root volume.

Disk pending operations (Data Device dm-0)

• Pending disk operations on the data device may indicate storage system saturation for the data volume.

Disk traffic (Root Device)

• Write traffic on the root volume is mostly due to logging and collectd graph data collection.
• Read traffic on the root volume is typically very low; However, support bundle generation may cause temporary spikes.

Disk traffic (Data Device dm-0)

• Read and write trends depend on user and integration activity.
• Plateaus in this graph may indicate storage system saturation.

System services

System services graphs contain data related to the major databases on GitHub Enterprise. These are MySQL, and Elasticseach persistent databases, as well as Redis and Memcached which contain ephemeral data.

Memcached

Memcached provides a layer of in-memory caching for web and API operations. Memcached helps to provide quicker response times for users and integrations interacting with the system.

Memcached usage

• Displays the remaining free memory that Memcached can consume if required.
• On systems with over 64GB RAM, it is possible for this to reach 0, indicating that the daemon has cached the maximum amount of data for the instance.

Memcached operations

• Abnormal spikes or plateaus in operations per second could indicate polling or other busy activity periods.
• API and Web UI requests will often have portions of the response cached in memcache to speed up future requests for the same data.

MySQL

MySQL is the primary database in GitHub Enterprise. User, issue, and other non-git or search related metadata is stored within MySQL.

MySQL usage

• Reflects memory InnoDB buffer pool memory used.
• Memory used by the buffer pool is typically very constant after the system has been warmed up and running with a few hours of normal usage.

MySQL threads

• Running and cached threads may fluctuate based on activity.
• Connected will reflect a constant value in most cases.

MySQL operations

• Select activity is most commonly the highest value on this graph, and reflects user and API integration activity most closely as data is read from the database.
• Insert and update trends will be influenced by API POST activity, or other operations which write to the database, such as issue comments.

MySQL rows

• The rows read from the database is most commonly the highest value on this graph.

Redis

The Redis database mainly contains background job queue, as well as session state information.

Redis usage

• Memory used by Redis is typically constant after the system has warmed up and is running with normal usage patterns.

Redis operations

• Large spikes in Redis operations may indicate an issue with background job processing.

Elasticsearch

Elasticsearch powers the built in search features in GitHub Enterprise.

Elasticsearch index

• The number of documents and bytes may increase over time, as data is indexed through normal operation of the GitHub Enterprise appliance.

Elasticsearch operations

• Fetch and query operation trends will fluctuate based on usage of GitHub Enterprise Search, and the Search API.

Elasticsearch memory

• Non-heap typically remains constant, while the heap memory fluctuates as background garbage collection and reindexing occurs.

Elasticsearch garbage collection

• Elasticsearch is a Java based service, and Java is a garbage-collected language. It is normal for these operations to regularly occur on a GitHub Enterprise system.
• Abnormal trends of garbage collection could indicate a performance problem with the Elasticsearch service.

Custom hooks

Graphs related to pre-receive hook execution.

Pre-Receive Hooks

• Execution time of pre-receive hooks, in milliseconds.
• Pre-recieve hooks have a non-configurable 5 second timeout. Longer or more indepth checks should be performed via CI and reported as a required Status Check to the relevant Pull Request instead.

Git fetch caching

GitHub Enterprise will attempt to cache intensive operations, such as git pack-objects, when multiple identical requests arrive in quick succession.

Cached Requests

• Git client requests which GitHub Enterprise cached.
• High sustained rates of git requests being cached can be a result of clients polling for changes.

Served Requests

• Git client requests that GitHub Enterprise was able to serve from cache.
• Indicates a detected "Thundering Herd" of requests.

Ignored Requests

• Requests ignored by the caching system, as they were not good candidates for caching.
• High sustained rates of ignored requests may also indicate polling, however the built in caching was unable to provide any benefit. These requests may have lower performance overall.

Cluster

Graphs related to GitHub Enterprise High Availability or Clustering.

Cluster ping

• High ping response times between HA and Cluster nodes may impact replication performance.
• In Geo-replication environments ping times between replicas may reach upwards of several hundred milliseconds. Overall Git push speeds in Geo-replication environments will also be impacted by this latency.

[yju01]

good

[BTC2YOU]

,♥️