|
1 |
| -# Placeholder |
| 1 | +# AutoMQ Client Examples |
| 2 | + |
| 3 | +This directory contains comprehensive client examples for interacting with AutoMQ using different programming languages. Each example demonstrates both basic and advanced Kafka operations with performance metrics and best practices optimized for AutoMQ's object storage architecture. |
| 4 | + |
| 5 | +## Common Design Principles |
| 6 | + |
| 7 | +All client examples in this repository share the following characteristics: |
| 8 | + |
| 9 | +🎯 **Community-Recommended Clients**: Each example uses the client library recommended by both AutoMQ and the Apache Kafka community for optimal compatibility and performance. |
| 10 | + |
| 11 | +⚙️ **AutoMQ Best Practices**: All clients are pre-configured with parameters following AutoMQ's recommended [best practices](https://www.automq.com/docs/automq-cloud/getting-started/client-sdk-guide), including optimized batch sizes, linger times, and timeout settings for object storage architecture. |
| 12 | + |
| 13 | +🐳 **Docker Compose Ready**: Every example includes Docker Compose configuration for quick deployment and can be seamlessly integrated with AutoMQ cluster Docker Compose setups. |
| 14 | + |
| 15 | +📊 **Built-in Metrics**: All examples automatically print comprehensive performance metrics including latency measurements, throughput statistics, and operational insights. |
| 16 | + |
| 17 | +🚀 **Production-Ready**: Examples demonstrate real-world patterns with proper error handling, resource management, and concurrent processing capabilities. |
| 18 | + |
| 19 | +## Available Client Examples |
| 20 | + |
| 21 | +### 🔥 [Java Examples](./java/) |
| 22 | + |
| 23 | +Java examples using the Apache Kafka client library with Maven build system. |
| 24 | + |
| 25 | +**Available Examples:** |
| 26 | +- **SimpleMessageExample**: Basic producer/consumer operations |
| 27 | +- **TransactionalMessageExample**: ACID-compliant transactional messaging |
| 28 | +- **HubSpotContactDemo**: Real-world integration example |
| 29 | +- Performance metrics and Docker support included |
| 30 | + |
| 31 | +### 🚀 [Go Examples](./go/) |
| 32 | + |
| 33 | +Go examples using the franz-go client library. |
| 34 | + |
| 35 | +**Available Examples:** |
| 36 | +- **SimpleMessageExample**: Basic producer/consumer operations |
| 37 | +- **TransactionalMessageExample**: Transactional operations with ACID guarantees |
| 38 | +- Performance metrics and Docker support included |
| 39 | + |
| 40 | +### 🐍 [Python Examples](./python/) |
| 41 | + |
| 42 | +Python examples using the kafka-python library. |
| 43 | + |
| 44 | +**Available Examples:** |
| 45 | +- **SimpleMessageExample**: Basic producer/consumer operations |
| 46 | +- Performance metrics and Docker support included |
| 47 | +- Multiple execution methods (direct Python, Docker, shell script) |
| 48 | + |
| 49 | +### ⚡ [C++ Examples](./cpp/) |
| 50 | + |
| 51 | +*Note: C++ examples are currently TODO. Implementation is planned for future releases.* |
| 52 | + |
| 53 | +### 🌐 JavaScript Examples |
| 54 | + |
| 55 | +*Note: JavaScript examples are currently TODO. Implementation is planned for future releases.* |
| 56 | + |
| 57 | +### 🦀 Rust Examples |
| 58 | + |
| 59 | +*Note: Rust examples are currently TODO. Implementation is planned for future releases.* |
| 60 | + |
| 61 | +## Common Features Across All Examples |
| 62 | + |
| 63 | +### 🎯 **Performance Metrics** |
| 64 | +All examples provide detailed performance analytics: |
| 65 | +- **Total Messages**: Configured message count |
| 66 | +- **Messages Sent/Received**: Actual successful operations |
| 67 | +- **Total Execution Time**: End-to-end processing time |
| 68 | +- **Average Produce Latency**: Time from send() to broker acknowledgment |
| 69 | +- **Average End-to-End Latency**: Time from message creation to consumption |
| 70 | +- **Consume Time**: Duration of the consumption phase |
| 71 | + |
| 72 | +### 🔧 **AutoMQ Optimizations** |
| 73 | +All clients are configured with AutoMQ-specific optimizations: |
| 74 | +- **Batch Size**: 1MB for optimal object storage performance |
| 75 | +- **Linger Time**: 100ms for efficient message batching |
| 76 | +- **Max Request Size**: 16MB for large message batches |
| 77 | +- **Metadata Refresh**: 60 seconds for balanced performance |
| 78 | +- **Acknowledgments**: "all" for durability guarantees |
| 79 | + |
| 80 | +### 🐳 **Docker Support** |
| 81 | +Every example includes: |
| 82 | +- Dockerfile with multi-stage builds |
| 83 | +- Docker Compose configuration |
| 84 | +- Environment variable configuration |
| 85 | +- Network integration with AutoMQ clusters |
| 86 | +- Security best practices (non-root execution) |
| 87 | + |
| 88 | +### 🔒 **Production-Ready Features** |
| 89 | +- Comprehensive error handling and retry logic |
| 90 | +- Proper resource management and cleanup |
| 91 | +- Configurable timeouts and connection settings |
| 92 | +- Logging with appropriate levels |
| 93 | +- Thread-safe operations and concurrency handling |
| 94 | + |
| 95 | +## Quick Start |
| 96 | + |
| 97 | +### Prerequisites |
| 98 | +- Running AutoMQ cluster |
| 99 | +- Docker and Docker Compose (for containerized deployment) |
| 100 | +- Language-specific requirements (see individual README files) |
| 101 | + |
| 102 | +### Running Examples |
| 103 | + |
| 104 | +Each language directory contains detailed instructions, but here's a quick overview: |
| 105 | + |
| 106 | +```bash |
| 107 | +# Java |
| 108 | +cd java && mvn clean package && docker compose up --build |
| 109 | + |
| 110 | +# Go |
| 111 | +cd go && go build -o kafka-examples . && ./kafka-examples |
| 112 | + |
| 113 | +# Python |
| 114 | +cd python && ./run.sh |
| 115 | + |
| 116 | +# Or use Docker for any language |
| 117 | +cd <language> && docker compose up --build |
| 118 | +``` |
| 119 | + |
| 120 | +### Configuration |
| 121 | + |
| 122 | +All examples support environment variable configuration: |
| 123 | +- `BOOTSTRAP_SERVERS`: AutoMQ broker addresses (default: localhost:9092) |
| 124 | +- Additional language-specific variables (see individual READMEs) |
| 125 | + |
| 126 | +## Architecture Overview |
| 127 | + |
| 128 | +``` |
| 129 | +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ |
| 130 | +│ Producer │ │ AutoMQ │ │ Consumer │ |
| 131 | +│ │───▶│ Cluster │───▶│ │ |
| 132 | +│ • Batching │ │ │ │ • Group Mgmt │ |
| 133 | +│ • Compression │ │ • Object Store │ │ • Offset Mgmt │ |
| 134 | +│ • Retries │ │ • Partitioning │ │ • Rebalancing │ |
| 135 | +└─────────────────┘ └─────────────────┘ └─────────────────┘ |
| 136 | +``` |
| 137 | + |
| 138 | +## Performance Tuning |
| 139 | + |
| 140 | +For optimal performance with AutoMQ: |
| 141 | + |
| 142 | +1. **Batch Configuration**: Use larger batch sizes (1MB) to reduce object storage operations |
| 143 | +2. **Linger Time**: Set appropriate linger time (100ms) for batching efficiency |
| 144 | +3. **Compression**: Enable compression for better network utilization |
| 145 | +4. **Connection Pooling**: Reuse connections and clients when possible |
| 146 | +5. **Monitoring**: Use the built-in metrics for performance analysis |
| 147 | + |
| 148 | +## Troubleshooting |
| 149 | + |
| 150 | +### Common Issues |
| 151 | + |
| 152 | +1. **Connection Refused**: Verify AutoMQ cluster is running and accessible |
| 153 | +2. **Topic Not Found**: Ensure topics exist or enable auto-creation |
| 154 | +3. **Permission Denied**: Check Kafka ACLs and authentication settings |
| 155 | +4. **High Latency**: Review batch size and linger time configuration |
| 156 | +5. **Memory Issues**: Adjust JVM settings or container memory limits |
| 157 | + |
| 158 | +### Getting Help |
| 159 | + |
| 160 | +- Check individual language README files for specific guidance |
| 161 | +- Review AutoMQ documentation: https://www.automq.com/docs |
| 162 | +- Examine log files in the `logs/` directory of each example |
| 163 | +- Use Docker logs: `docker compose logs -f` |
| 164 | + |
| 165 | +## Contributing |
| 166 | + |
| 167 | +We welcome contributions to improve these examples: |
| 168 | + |
| 169 | +1. Fork the repository |
| 170 | +2. Create a feature branch |
| 171 | +3. Add tests for new functionality |
| 172 | +4. Update documentation |
| 173 | +5. Submit a pull request |
| 174 | + |
| 175 | +## License |
| 176 | + |
| 177 | +These examples are provided under the same license as the AutoMQ project. |
0 commit comments