Skip to content

SERVICE_GUIDE.md

Latest commit

 

History

History
547 lines (410 loc) · 11.7 KB

SERVICE_GUIDE.md

File metadata and controls

547 lines (410 loc) · 11.7 KB

MeshC2 Service Management Guide

Table of Contents

Service Status & Health

Check Service Status

# Quick status check
sudo systemctl status meshc2

# Detailed status with recent logs
sudo systemctl status meshc2 -l --no-pager

# Check if service is active
sudo systemctl is-active meshc2

# Check if service is enabled (auto-start on boot)
sudo systemctl is-enabled meshc2

Monitor Service Health

# Watch service status in real-time
watch -n 2 'sudo systemctl status meshc2'

# Check service resource usage
sudo systemctl show meshc2 --property=MemoryCurrent,CPUUsageNSec,TasksCurrent

# Check service uptime
sudo systemctl show meshc2 --property=ActiveEnterTimestamp

Local Messaging

MeshC2 allows local scripts and applications to send messages through the mesh network. This is useful for system notifications, alerts, and automation.

Basic Message Sending

# Send a simple message
meshc2 --message "Task completed"

# Send message with timestamp
meshc2 --message "Backup finished" --timestamp

# Send to specific channel
meshc2 --message "System alert" --channel "alerts"

Message Features

  • Socket Communication: Uses running service via Unix socket (/var/lib/meshc2/meshc2.sock)
  • No Device Conflicts: Messages sent through running service, avoiding USB device competition
  • Automatic Message Splitting: Messages longer than 230 characters are automatically split
  • Channel Validation: Invalid channels show available options
  • Retry Logic: Failed sends are retried up to 3 times
  • Connection Management: Waits up to 30 seconds for device connection
  • Comprehensive Logging: All messages logged to service log
  • Concurrent Access: Multiple scripts can send messages simultaneously

Integration Examples

Backup Script Integration:

#!/bin/bash
# Backup with mesh notifications

meshc2 --message "Starting backup on $(hostname)" --timestamp

if backup_command; then
    meshc2 --message "Backup completed successfully" --timestamp
else
    meshc2 --message "BACKUP FAILED on $(hostname)" --timestamp --channel "critical"
fi

System Monitoring:

# Check system health and notify
if [ $(df -h / | awk 'NR==2{print $5}' | sed 's/%//') -gt 80 ]; then
    meshc2 --message "Disk usage warning: $(df -h / | awk 'NR==2{print $5}') on $(hostname)" --timestamp
fi

Service Integration:

# In systemd service files
ExecStartPost=/usr/local/bin/meshc2 --message "Service %i started" --timestamp
ExecStopPost=/usr/local/bin/meshc2 --message "Service %i stopped" --timestamp

Troubleshooting Messages

If message sending fails:

  1. Check Service Status

    sudo systemctl status meshc2

  2. Verify Device Connection

    meshc2 --list-devices

  3. Test Channel Access

    meshc2 --real-device --list-channels

  4. Check Logs for Errors

    sudo tail -f /var/log/meshc2/meshc2.log

Starting & Stopping

Basic Service Control

# Start the service
sudo systemctl start meshc2

# Stop the service
sudo systemctl stop meshc2

# Restart the service (stop then start)
sudo systemctl restart meshc2

# Reload configuration without stopping (if supported)
sudo systemctl reload meshc2

# Enable auto-start on boot
sudo systemctl enable meshc2

# Disable auto-start on boot
sudo systemctl disable meshc2

Emergency Controls

# Force kill if service is hung
sudo systemctl kill -s KILL meshc2

# Reset failed state (after fixing issues)
sudo systemctl reset-failed meshc2

# Mask service (prevent it from being started)
sudo systemctl mask meshc2

# Unmask service
sudo systemctl unmask meshc2

Debugging & Troubleshooting

View Logs

Real-time Logs

# Follow service logs in real-time
sudo journalctl -u meshc2 -f

# Follow with timestamps
sudo journalctl -u meshc2 -f --output=short-precise

# Follow only errors
sudo journalctl -u meshc2 -f -p err

Historical Logs

# View last 100 lines
sudo journalctl -u meshc2 -n 100

# View logs from last hour
sudo journalctl -u meshc2 --since "1 hour ago"

# View logs from specific time range
sudo journalctl -u meshc2 --since "2025-08-01 16:00" --until "2025-08-01 17:00"

# View logs from last boot
sudo journalctl -u meshc2 -b

# Export logs to file
sudo journalctl -u meshc2 > /tmp/meshc2-logs.txt

Application Log File

# View application log directly
sudo tail -f /var/log/meshc2/meshc2.log

# View with less
sudo less /var/log/meshc2/meshc2.log

# Search for errors
sudo grep -i error /var/log/meshc2/meshc2.log

# Monitor message activity
/opt/meshc2/monitor_messages.sh

Manual Testing

Run Service Manually

# Stop the systemd service first
sudo systemctl stop meshc2

# Run as service user with real device
sudo -u meshc2 /opt/meshc2/venv/bin/python -m src.meshtastic_troubleshoot.main --real-device

# Run with debug output
sudo -u meshc2 PYTHONPATH=/opt/meshc2 /opt/meshc2/venv/bin/python -m src.meshtastic_troubleshoot.main --real-device --debug

# Run with mock device (for testing without hardware)
sudo -u meshc2 PYTHONPATH=/opt/meshc2 /opt/meshc2/venv/bin/python -m src.meshtastic_troubleshoot.main

Test Specific Components

# Test device detection
/opt/meshc2/meshc2 --list-devices

# List channels on device
/opt/meshc2/meshc2 --real-device --list-channels

# Validate configuration
/opt/meshc2/meshc2 --validate-config

# Check service status via CLI
/opt/meshc2/meshc2 --status

Debug Mode

Enable Debug Logging

  1. Edit configuration:
sudo nano /etc/meshc2/config.yml
  1. Change log level:
logging:
  level: "DEBUG"  # Change from INFO to DEBUG
  1. Restart service:
sudo systemctl restart meshc2

Strace Debugging

# Trace system calls
sudo strace -p $(pgrep -f meshtastic_troubleshoot)

# Trace with timestamps
sudo strace -t -p $(pgrep -f meshtastic_troubleshoot)

# Trace file operations
sudo strace -e file -p $(pgrep -f meshtastic_troubleshoot)

Configuration Management

View Current Configuration

# View main config
sudo cat /etc/meshc2/config.yml

# View service definition
sudo systemctl cat meshc2

# Check environment variables
sudo systemctl show-environment

Edit Configuration

# Edit main config
sudo nano /etc/meshc2/config.yml

# Edit service file (not recommended, use override instead)
sudo systemctl edit meshc2

# Create override file
sudo systemctl edit meshc2 --full

Apply Configuration Changes

# After editing config.yml
sudo systemctl restart meshc2

# After editing service file
sudo systemctl daemon-reload
sudo systemctl restart meshc2

Backup Configuration

# Backup current config
sudo cp /etc/meshc2/config.yml /etc/meshc2/config.yml.backup

# Backup with timestamp
sudo cp /etc/meshc2/config.yml /etc/meshc2/config.yml.$(date +%Y%m%d_%H%M%S)

Log Management

Log Rotation

# Force log rotation
sudo logrotate -f /etc/logrotate.d/meshc2

# Test rotation (dry run)
sudo logrotate -d /etc/logrotate.d/meshc2

# View rotated logs
ls -la /var/log/meshc2/

Clear Logs

# Clear systemd journal for service
sudo journalctl --vacuum-time=1s -u meshc2

# Clear application log (preserve file)
echo "" | sudo tee /var/log/meshc2/meshc2.log

# Remove old rotated logs
sudo find /var/log/meshc2/ -name "*.gz" -mtime +30 -delete

Common Issues & Solutions

Issue: Service Won't Start

Check for port conflicts

# Check if device is in use
lsof /dev/ttyUSB0
fuser /dev/ttyUSB0

# Check USB devices
lsusb
ls -la /dev/ttyUSB*

Check permissions

# Verify service user is in dialout group
groups meshc2

# Check device permissions
ls -la /dev/ttyUSB*

# Fix permissions if needed
sudo usermod -a -G dialout meshc2
sudo chmod 666 /dev/ttyUSB0

Issue: No Meshtastic Device Found

Check device connection

# List USB devices
lsusb | grep -E "(CP210|CH340|FTDI|10c4:ea60)"

# Check kernel messages
dmesg | tail -20

# Check serial devices
ls -la /dev/tty*

# Test device directly
sudo -u meshc2 /opt/meshc2/venv/bin/python -c "import meshtastic; meshtastic.serial_interface.SerialInterface()"

Issue: Service Crashes Repeatedly

Check for crash loops

# View recent crashes
sudo journalctl -u meshc2 -p err --since "10 minutes ago"

# Check restart count
sudo systemctl show meshc2 --property=NRestarts

# Temporarily disable restart
sudo systemctl edit meshc2
# Add under [Service]:
# Restart=no

Issue: High CPU/Memory Usage

Monitor resources

# Real-time monitoring
htop -p $(pgrep -f meshtastic_troubleshoot)

# Check limits
sudo systemctl show meshc2 --property=LimitNPROC,LimitNOFILE,MemoryMax

# Set memory limit (temporary)
sudo systemctl set-property meshc2 MemoryMax=500M

Performance Tuning

Adjust Restart Behavior

# Edit service override
sudo systemctl edit meshc2

# Add these lines:
[Service]
RestartSec=30
StartLimitBurst=3
StartLimitIntervalSec=120

Adjust Logging

# Reduce log verbosity in /etc/meshc2/config.yml
logging:
  level: "WARNING"  # Only warnings and errors

Maintenance Tasks

Weekly Maintenance

# Check service health
sudo systemctl status meshc2

# Review error logs
sudo journalctl -u meshc2 -p err --since "7 days ago"

# Check disk usage
du -sh /var/log/meshc2/

# Rotate logs if needed
sudo logrotate -f /etc/logrotate.d/meshc2

Monthly Maintenance

# Backup configuration
sudo tar -czf /backup/meshc2-config-$(date +%Y%m).tar.gz /etc/meshc2/

# Clean old logs
sudo find /var/log/meshc2/ -name "*.gz" -mtime +30 -delete

# Update system packages
sudo apt update && sudo apt upgrade

# Check for MeshC2 updates
cd /opt/meshc2 && git status

Integration with Monitoring

Systemd Integration

# Create health check script
cat << 'EOF' | sudo tee /usr/local/bin/meshc2-health-check
#!/bin/bash
if systemctl is-active --quiet meshc2; then
    echo "MeshC2 service is running"
    exit 0
else
    echo "MeshC2 service is not running"
    exit 1
fi
EOF

sudo chmod +x /usr/local/bin/meshc2-health-check

Prometheus/Node Exporter

# Add to crontab for monitoring
* * * * * /usr/local/bin/meshc2-health-check || echo "meshc2_service_down 1" > /var/lib/node_exporter/meshc2.prom

Emergency Recovery

Complete Service Reset

#!/bin/bash
# Save this as /usr/local/bin/meshc2-reset

# Stop service
sudo systemctl stop meshc2

# Backup current config
sudo cp /etc/meshc2/config.yml /etc/meshc2/config.yml.backup

# Clear logs
echo "" | sudo tee /var/log/meshc2/meshc2.log

# Reset systemd
sudo systemctl reset-failed meshc2
sudo systemctl daemon-reload

# Start fresh
sudo systemctl start meshc2

Reinstall Service

# Complete removal and reinstall
cd /home/eric/MeshC2
sudo ./uninstall.sh
./install.sh

Contact & Support

For issues or questions:

  • Check logs first: sudo journalctl -u meshc2 -n 100
  • Review this guide for common solutions
  • Check device connection: lsusb | grep -E "(CP210|CH340|FTDI)"
  • Verify configuration: /opt/meshc2/meshc2 --validate-config

Quick Reference Card

# Most Common Commands
sudo systemctl status meshc2    # Check status
sudo systemctl restart meshc2   # Restart service
sudo journalctl -u meshc2 -f    # Follow logs
/opt/meshc2/monitor_messages.sh                  # Monitor messages
sudo nano /etc/meshc2/config.yml                 # Edit config