Documentation > Administration > Grafana Integration Setup

Grafana Integration Setup

Complete guide to setting up Grafana integration with SysManage for OpenTelemetry metrics visualization.

Overview

SysManage integrates with Grafana to provide powerful visualization and monitoring capabilities for OpenTelemetry metrics collected from your managed hosts. This guide walks you through the complete setup process, from installing Grafana Alloy to verifying metrics flow.

Prerequisites

  • A running Grafana instance (version 9.0 or later recommended)
  • Administrative access to the Grafana server
  • Network connectivity between SysManage, Grafana, and managed hosts
  • Appropriate permissions in SysManage (DEPLOY_OPENTELEMETRY role)

Installing Grafana Alloy

Grafana Alloy is the OpenTelemetry collector that receives OTLP metrics from managed hosts and forwards them to Prometheus for storage and visualization.

Debian/Ubuntu Installation

# Add Grafana GPG key and repository
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

# Update and install Alloy
sudo apt-get update
sudo apt-get install -y alloy

RHEL/CentOS/Fedora Installation

# Add Grafana repository
cat <

Other Platforms

For macOS, Windows, or other Linux distributions, refer to the official Grafana Alloy installation documentation.

Configuring Grafana Alloy for OTLP

Configure Alloy to receive OTLP metrics on port 4317 and forward them to Prometheus.

Create Configuration File

Create or edit the Alloy configuration file at /etc/alloy/config.alloy:

// OTLP receiver on port 4317
otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }

  output {
    metrics = [otelcol.processor.batch.default.input]
  }
}

// Batch processor for efficiency
otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.prometheus.default.input]
  }
}

// Export to Prometheus
otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.default.receiver]
}

// Prometheus remote write (local instance)
prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9090/api/v1/write"
  }
}

Enable and Start Alloy

# Enable Alloy to start on boot
sudo systemctl enable alloy

# Start the Alloy service
sudo systemctl start alloy

# Verify it's running
sudo systemctl status alloy

Verify Port 4317 is Listening

# Check that Alloy is listening on port 4317
sudo ss -tlnp | grep 4317

# Expected output:
# LISTEN 0 4096 0.0.0.0:4317 0.0.0.0:* users:(("alloy",pid=XXXX,fd=X))

Configuring Prometheus for Remote Write

Prometheus must be configured to accept remote write requests from Grafana Alloy. This requires enabling the remote write receiver feature.

Enable Remote Write Receiver

Edit the Prometheus systemd service file to add the --web.enable-remote-write-receiver flag:

# Edit the Prometheus service file
sudo systemctl edit --full prometheus

# Find the ExecStart line and add the flag:
ExecStart=/usr/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.enable-remote-write-receiver

# Save and exit the editor

Apply Configuration Changes

# Reload systemd to pick up the service file changes
sudo systemctl daemon-reload

# Restart Prometheus
sudo systemctl restart prometheus

# Verify Prometheus is running with the flag
ps aux | grep prometheus | grep enable-remote-write-receiver
Warning: Without the --web.enable-remote-write-receiver flag, Alloy will receive "404 Not Found" errors when attempting to write metrics to Prometheus. This is a required configuration step.

Restart Alloy After Prometheus Configuration

If Alloy was already running and encountering errors before Prometheus was configured, restart it to clear any retry backoff state:

sudo systemctl restart alloy

Configuring SysManage Integration

Enable Grafana integration in SysManage to allow OpenTelemetry deployment to hosts.

Enable Grafana Integration

  1. Navigate to Settings in the SysManage web interface
  2. Find the Grafana Integration card
  3. Toggle Enabled to Yes
  4. Enter your Grafana server URL (e.g., http://grafana.example.com:4317)
  5. Click Save Changes
Note: The URL should point to the Grafana Alloy OTLP endpoint (port 4317), not the Grafana web interface (typically port 3000).

Deploying OpenTelemetry to Hosts

Once Grafana integration is enabled, you can deploy OpenTelemetry collectors to individual hosts.

Deployment Requirements

  • Host must have a privileged SysManage agent running (with root/administrator access)
  • Grafana integration must be enabled in SysManage settings
  • User must have the DEPLOY_OPENTELEMETRY permission

Deployment Steps

  1. Navigate to the host detail page for the target host
  2. Scroll to the OpenTelemetry Status card
  3. Click Deploy OpenTelemetry
  4. Wait for deployment to complete (typically 30-60 seconds)
  5. Verify the status shows "Deployed: Yes" and "Service Status: running"
Auto-refresh: The OpenTelemetry Status card refreshes automatically every 30 seconds to reflect deployment status changes.

Verifying Metrics Flow

After deployment, verify that metrics are flowing from hosts through Alloy to Prometheus and Grafana.

Check OpenTelemetry Service Status

On the managed host, verify the OpenTelemetry collector is running:

# Check service status
sudo systemctl status otelcol-contrib

# View recent logs
sudo journalctl -u otelcol-contrib -n 50

Check Grafana Alloy Logs

On the Grafana server, verify Alloy is receiving metrics:

# View Alloy logs
sudo journalctl -u alloy -f

# Look for messages indicating OTLP metric reception

Query Prometheus for Metrics

Access Prometheus (typically at http://your-grafana-server:9090) and run a query:

# Query for any OpenTelemetry metrics
{job="otelcol"}

# Query for specific system metrics
system_cpu_utilization
system_memory_usage

Create Grafana Dashboard

  1. Log in to Grafana web interface
  2. Create a new dashboard
  3. Add panels with Prometheus queries for OpenTelemetry metrics
  4. Save and share the dashboard with your team

Troubleshooting

No Metrics Appearing in Prometheus

  • Verify OpenTelemetry collector is running on the host: systemctl status otelcol-contrib
  • Check collector configuration file at /etc/otelcol-contrib/config.yaml
  • Verify network connectivity from host to Grafana server on port 4317
  • Check firewall rules allowing traffic on port 4317

Deployment Fails

  • Ensure SysManage agent is running in privileged mode
  • Check agent logs for error messages: ~/dev/sysmanage-agent/logs/agent.log
  • Verify user has DEPLOY_OPENTELEMETRY permission
  • Confirm package manager (apt/yum) is working correctly on the target host

Connection Errors in Logs

  • Verify Grafana URL in SysManage settings points to port 4317 (not 3000)
  • Test connectivity: nc -zv grafana-server 4317
  • Ensure Grafana Alloy is running: systemctl status alloy
  • Review Alloy configuration for correct OTLP receiver setup

"Frame Too Large" Errors

  • This indicates the collector is trying to send OTLP to Grafana's web interface (port 3000) instead of Alloy (port 4317)
  • Update the Grafana URL in SysManage settings to use port 4317
  • Redeploy OpenTelemetry to the affected hosts to update their configuration

Management Operations

Connect to Grafana

If a host was deployed before Grafana integration was enabled, you can connect it:

  • Enable Grafana integration in Settings
  • Navigate to the host detail page
  • Click Connect to Grafana in the OpenTelemetry Status card

Disconnect from Grafana

To stop sending metrics while keeping OpenTelemetry installed:

  • Navigate to the host detail page
  • Click Disconnect from Grafana
  • The collector will remain installed but stop exporting metrics

Restart Service

To restart the OpenTelemetry collector service:

  • Navigate to the host detail page
  • Click Restart in the OpenTelemetry Status card

Remove OpenTelemetry

To completely uninstall OpenTelemetry from a host:

  • Navigate to the host detail page
  • Click Remove OpenTelemetry
  • The collector package will be completely removed, including configuration files

Security Best Practices

Use TLS Encryption

  • Configure Alloy to use TLS for OTLP receiver
  • Use valid SSL certificates from a trusted CA
  • Update SysManage Grafana URL to use https://

Network Security

  • Restrict port 4317 access to only SysManage-managed hosts
  • Use network segmentation or VPN for metrics traffic
  • Implement firewall rules to limit exposure

Authentication

  • Configure Alloy with authentication for OTLP endpoint
  • Use API keys or mTLS for additional security
  • Rotate credentials regularly

Monitor Access

  • Review Alloy logs for unauthorized access attempts
  • Set up alerts for connection anomalies
  • Audit OpenTelemetry deployment history in SysManage

Performance Considerations

Collection Intervals

Adjust metric collection intervals based on your monitoring needs and system resources. More frequent collection provides better granularity but increases resource usage.

Data Retention

Configure Prometheus retention policies to balance historical data availability with storage requirements. Consider using remote storage for long-term metrics.

Batch Processing

Alloy's batch processor groups metrics before sending to Prometheus, reducing network overhead. Adjust batch size and timeout based on metric volume.

Scaling Considerations

For large deployments, consider running multiple Alloy instances behind a load balancer, or using Prometheus federation for distributed metrics collection.

Quick Navigation

← Monitoring & Alerts Backup & Recovery →