#340 Part 3: Complete Audit Flow integration and create Audit Interface (#413)

* Add Audit Service to CE and PDP in OE federator

* Update exchange/orchestration-engine/middleware/audit_middleware.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Refactor audit service enums and logging methods for improved clarity and consistency

* Add CONSENT_CHECK to eventTypes in audit service enum configuration

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Thanikan <sthanikan2000@gmail.com>

Author: 3 people

audit-service/README.md

@@ -1,6 +1,6 @@
 # Audit Service
 
-A Go microservice for centralized audit logging across OpenDIF services, providing distributed tracing and comprehensive event tracking.
+A generic Go microservice for centralized audit logging across distributed services, providing distributed tracing and comprehensive event tracking.
 
 [![Go Version](https://img.shields.io/badge/Go-1.21%2B-blue)](https://golang.org/)
 [![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](../LICENSE)
@@ -115,6 +115,31 @@ Copy `.env.example` to `.env` and configure:
 
 For PostgreSQL configuration and advanced settings, see [.env.example](.env.example).
 
+### Event Type Configuration
+
+Event types are configurable via `config/enums.yaml`. This allows you to customize the audit service for your specific use case. The service comes with generic default event types, but you can add project-specific ones.
+
+**Default Event Types:**
+- `MANAGEMENT_EVENT` - Administrative operations
+- `USER_MANAGEMENT` - User-related operations
+- `DATA_FETCH` - Data retrieval operations
+
+**Customizing Event Types:**
+
+Edit `config/enums.yaml` to add your own event types:
+
+```yaml
+enums:
+  eventTypes:
+    - MANAGEMENT_EVENT
+    - USER_MANAGEMENT
+    - DATA_FETCH
+    - YOUR_CUSTOM_EVENT_TYPE
+    - ANOTHER_EVENT_TYPE
+```
+
+See [config/README.md](config/README.md) for detailed configuration options.
+
 ## API Endpoints
 
 ### Core Endpoints
@@ -136,13 +161,13 @@ curl -X POST http://localhost:3001/api/audit-logs \
   -d '{
     "traceId": "550e8400-e29b-41d4-a716-446655440000",
     "timestamp": "2024-01-20T10:00:00Z",
-    "eventType": "POLICY_CHECK",
+    "eventType": "MANAGEMENT_EVENT",
     "eventAction": "READ",
     "status": "SUCCESS",
     "actorType": "SERVICE",
-    "actorId": "orchestration-engine",
+    "actorId": "my-service",
     "targetType": "SERVICE",
-    "targetId": "policy-decision-point"
+    "targetId": "target-service"
   }'
 ```
 
@@ -156,18 +181,9 @@ curl http://localhost:3001/api/audit-logs
 curl http://localhost:3001/api/audit-logs?traceId=550e8400-e29b-41d4-a716-446655440000
 
 # Filter by event type
-curl http://localhost:3001/api/audit-logs?eventType=POLICY_CHECK&status=SUCCESS
+curl http://localhost:3001/api/audit-logs?eventType=MANAGEMENT_EVENT&status=SUCCESS
 ```
 
-See [docs/API.md](docs/API.md) for complete API documentation.
-
-## Documentation
-
-- **[API Documentation](docs/API.md)** - Complete API reference with examples
-- **[Database Configuration](docs/DATABASE_CONFIGURATION.md)** - Database setup and configuration guide
-- **[Architecture](docs/ARCHITECTURE.md)** - Project structure and design patterns
-- **[OpenAPI Spec](openapi.yaml)** - OpenAPI 3.0 specification
-
 ## Development
 
 ### Project Structure
@@ -187,8 +203,6 @@ audit-service/
 └── main.go          # Entry point
 ```
 
-See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for detailed architecture documentation.
-
 ### Running Tests
 
 ```bash
@@ -221,6 +235,49 @@ go build -o audit-service
 go build -ldflags="-X main.Version=1.0.0 -X main.GitCommit=$(git rev-parse HEAD)" -o audit-service
 ```
 
+## Integration
+
+The Audit Service is designed to be integrated into any microservices architecture. It provides a simple REST API that can be called from any service.
+
+### Integration Pattern
+
+1. **Configure your service** to point to the audit service URL
+2. **Send audit events** via HTTP POST to `/api/audit-logs`
+3. **Query audit logs** via HTTP GET from `/api/audit-logs`
+
+### Client Libraries
+
+You can integrate the audit service using:
+
+- **HTTP Client**: Direct HTTP calls to the REST API
+- **Shared Client Package**: Use the `shared/audit` package (if available in your project)
+- **Custom Wrapper**: Create your own client library
+
+### Example Integration
+
+```go
+// Example: Log an audit event from your service
+auditRequest := map[string]interface{}{
+    "traceId":    traceID,
+    "timestamp":  time.Now().UTC().Format(time.RFC3339),
+    "eventType":  "YOUR_EVENT_TYPE",
+    "status":     "SUCCESS",
+    "actorType":  "SERVICE",
+    "actorId":    "your-service-name",
+    "targetType": "RESOURCE",
+    "targetId":   "resource-id",
+}
+
+// POST to http://audit-service:3001/api/audit-logs
+```
+
+### Graceful Degradation
+
+- Services continue to function normally if audit service is unavailable
+- No errors are thrown when audit service URL is not configured
+- Audit operations should be asynchronous (fire-and-forget) to avoid blocking requests
+- Services can be started before audit service is ready
+
 ## Deployment
 
 ### Docker
@@ -243,13 +300,17 @@ docker run -d \
   -e DB_TYPE=postgres \
   -e DB_HOST=postgres \
   -e DB_PASSWORD=your_password \
+  -e DB_NAME=audit_db \
   audit-service
 ```
 
 ### Docker Compose
 
+The audit service includes a `docker-compose.yml` for standalone deployment:
+
 ```bash
-# Start service
+# Deploy audit service
+cd audit-service
 docker compose up -d
 
 # View logs
@@ -266,30 +327,8 @@ docker compose down
 3. **CORS**: Configure `CORS_ALLOWED_ORIGINS` appropriately
 4. **Monitoring**: Monitor service health via `/health` endpoint
 5. **Backup**: Implement database backup strategy
-
-## Integration with OpenDIF Services
-
-The Audit Service integrates with:
-
-- **Orchestration Engine** - Tracks data exchange operations
-- **Portal Backend** - Logs administrative actions
-- **Consent Engine** - Records consent changes
-
-Audit logging is **optional** - services function normally without it.
-
-### Configuration in Other Services
-
-```bash
-# Enable audit logging in orchestration-engine
-export AUDIT_SERVICE_ENABLED=true
-export AUDIT_SERVICE_URL=http://audit-service:3001
-
-# Enable audit logging in portal-backend
-export AUDIT_SERVICE_ENABLED=true
-export AUDIT_SERVICE_URL=http://audit-service:3001
-```
-
-See [../exchange/AUDIT_SERVICE.md](../exchange/AUDIT_SERVICE.md) for integration documentation.
+6. **High Availability**: Consider deploying multiple instances behind a load balancer
+7. **Security**: Implement authentication/authorization if exposing the service publicly
 
 ## Troubleshooting
 
@@ -312,6 +351,12 @@ See [../exchange/AUDIT_SERVICE.md](../exchange/AUDIT_SERVICE.md) for integration
 - Check credentials and SSL settings
 - Verify network connectivity
 
+**Event type validation errors:**
+
+- Check that your event types are defined in `config/enums.yaml`
+- Verify the enum configuration file is being loaded correctly
+- Check service logs for validation error details
+
 See [docs/DATABASE_CONFIGURATION.md](docs/DATABASE_CONFIGURATION.md) for detailed troubleshooting.
 
 ## Contributing
@@ -328,12 +373,4 @@ This project is licensed under the Apache License 2.0 - see [LICENSE](../LICENSE
 
 ## Support
 
-- **Issues**: [GitHub Issues](https://github.com/OpenDIF/opendif-core/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/OpenDIF/opendif-core/discussions)
-- **Documentation**: [OpenDIF Documentation](https://github.com/OpenDIF/opendif-core/tree/main/docs)
-
-## Related Services
-
-- [Orchestration Engine](../exchange/orchestration-engine/) - Data exchange orchestration
-- [Portal Backend](../portal-backend/) - Admin portal backend
-- [Consent Engine](../exchange/consent-engine/) - Consent management
+For issues, questions, or contributions, please use the project's issue tracker and discussion forums.

audit-service/config/config.go

@@ -33,13 +33,13 @@ type Config struct {
 }
 
 // DefaultEnums provides default enum values if config file is not found
+// Note: OpenDIF-specific event types (ORCHESTRATION_REQUEST_RECEIVED, POLICY_CHECK, CONSENT_CHECK, PROVIDER_FETCH)
+// should be added to config/enums.yaml for project-specific configurations
 var DefaultEnums = AuditEnums{
 	EventTypes: []string{
-		"POLICY_CHECK",
 		"MANAGEMENT_EVENT",
 		"USER_MANAGEMENT",
 		"DATA_FETCH",
-		"CONSENT_CHECK",
 	},
 	EventActions: []string{
 		"CREATE",

audit-service/config/enums.yaml

@@ -9,8 +9,9 @@ enums:
     - POLICY_CHECK
     - MANAGEMENT_EVENT
     - USER_MANAGEMENT
-    - DATA_FETCH
     - CONSENT_CHECK
+    - DATA_REQUEST
+    - PROVIDER_FETCH
 
   # Event Action: CRUD operations
   eventActions:
@@ -29,6 +30,7 @@ enums:
     - ADMIN
     - MEMBER
     - SYSTEM
+    - APPLICATION
 
   # Target Type: Types of targets that actions can be performed on
   targetTypes:

audit-service/database/client.go

@@ -68,7 +68,7 @@ func NewDatabaseConfig() *Config {
 
 	// For SQLite: only DB_TYPE=sqlite or DB_PATH count as configuration
 	// DB_HOST is only relevant when DB_TYPE=postgres
-useFileBasedSQLite := dbPathSet || (dbTypeSet && dbTypeStr != "postgres" && dbTypeStr != "postgresql")
+	useFileBasedSQLite := dbPathSet || (dbTypeSet && dbTypeStr != "postgres" && dbTypeStr != "postgresql")
 
 	switch dbTypeStr {
 	case "postgres", "postgresql":
@@ -103,7 +103,7 @@ useFileBasedSQLite := dbPathSet || (dbTypeSet && dbTypeStr != "postgres" && dbTy
 		config.MaxIdleConns = parseIntOrDefault("DB_MAX_IDLE_CONNS", 1)
 
 		// Determine database path based on configuration
-if !useFileBasedSQLite {
+		if !useFileBasedSQLite {
 			// No SQLite configuration at all → in-memory database for quick testing
 			config.DatabasePath = ":memory:"
 			slog.Info("No database configuration found, using in-memory SQLite")

audit-service/v1/database/gorm_repository.go

@@ -77,6 +77,8 @@ func (r *GormRepository) GetAuditLogs(ctx context.Context, filters *AuditLogFilt
 	}
 
 	// Apply pagination and ordering
+	// Note: Results are ordered by timestamp DESC (newest first) for general queries.
+	// For trace-specific queries, use GetAuditLogsByTraceID which orders by ASC (chronological).
 	limit := filters.Limit
 	if limit <= 0 {
 		limit = 100 // default

audit-service/v1/models/audit_log.go

@@ -1,6 +1,7 @@
 package models
 
 import (
+	"database/sql/driver"
 	"encoding/json"
 	"fmt"
 	"sync"
@@ -11,6 +12,71 @@ import (
 	"gorm.io/gorm"
 )
 
+// JSONBRawMessage is a custom type that properly handles JSONB scanning from PostgreSQL
+// It implements sql.Scanner and driver.Valuer interfaces to handle both PostgreSQL JSONB
+// (which can return as string or []byte) and SQLite TEXT (which returns as []byte)
+// This type wraps json.RawMessage to provide database scanning capabilities while maintaining
+// the same JSON marshaling behavior as json.RawMessage.
+type JSONBRawMessage json.RawMessage
+
+// Scan implements the sql.Scanner interface for JSONBRawMessage
+// Handles both PostgreSQL JSONB (string or []byte) and SQLite TEXT ([]byte)
+func (j *JSONBRawMessage) Scan(value interface{}) error {
+	if value == nil {
+		*j = JSONBRawMessage(nil)
+		return nil
+	}
+
+	var bytes []byte
+	switch v := value.(type) {
+	case []byte:
+		bytes = v
+	case string:
+		bytes = []byte(v)
+	default:
+		return fmt.Errorf("cannot scan %T into JSONBRawMessage", value)
+	}
+
+	// Note: We don't validate JSON here to avoid performance overhead.
+	// The database should already contain valid JSON. If validation is needed,
+	// it should be done at the application layer when unmarshaling.
+	*j = JSONBRawMessage(bytes)
+	return nil
+}
+
+// Value implements the driver.Valuer interface for JSONBRawMessage
+func (j JSONBRawMessage) Value() (driver.Value, error) {
+	if len(j) == 0 {
+		return nil, nil
+	}
+	return []byte(j), nil
+}
+
+// MarshalJSON implements json.Marshaler for JSONBRawMessage
+// Delegates to the underlying json.RawMessage behavior
+func (j JSONBRawMessage) MarshalJSON() ([]byte, error) {
+	if len(j) == 0 {
+		return []byte("null"), nil
+	}
+	return []byte(j), nil
+}
+
+// UnmarshalJSON implements json.Unmarshaler for JSONBRawMessage
+// Delegates to the underlying json.RawMessage behavior
+func (j *JSONBRawMessage) UnmarshalJSON(data []byte) error {
+	if j == nil {
+		return fmt.Errorf("JSONBRawMessage: UnmarshalJSON on nil pointer")
+	}
+	*j = JSONBRawMessage(data)
+	return nil
+}
+
+// GormDataType returns the GORM data type for JSONBRawMessage
+// This helps GORM understand the database column type
+func (JSONBRawMessage) GormDataType() string {
+	return "jsonb"
+}
+
 // Audit log status constants (not configurable via YAML as they are core to the system)
 const (
 	StatusSuccess = "SUCCESS"
@@ -66,9 +132,9 @@ type AuditLog struct {
 	TargetID   *string `gorm:"type:varchar(255)" json:"targetId,omitempty"` // resource_id or service_name
 
 	// Metadata (Payload without PII/sensitive data)
-	RequestMetadata    json.RawMessage `gorm:"type:text" json:"requestMetadata,omitempty"`    // Request payload without PII/sensitive data
-	ResponseMetadata   json.RawMessage `gorm:"type:text" json:"responseMetadata,omitempty"`   // Response or Error details
-	AdditionalMetadata json.RawMessage `gorm:"type:text" json:"additionalMetadata,omitempty"` // Additional context-specific data
+	RequestMetadata    JSONBRawMessage `gorm:"type:jsonb" json:"requestMetadata,omitempty"`    // Request payload without PII/sensitive data
+	ResponseMetadata   JSONBRawMessage `gorm:"type:jsonb" json:"responseMetadata,omitempty"`   // Response or Error details
+	AdditionalMetadata JSONBRawMessage `gorm:"type:jsonb" json:"additionalMetadata,omitempty"` // Additional context-specific data
 
 	// BaseModel provides CreatedAt
 	BaseModel

audit-service/v1/models/request_dtos.go

@@ -1,9 +1,5 @@
 package models
 
-import (
-	"encoding/json"
-)
-
 // CreateAuditLogRequest represents the request payload for creating a generalized audit log
 // This matches the final SQL schema with unified actor/target approach
 type CreateAuditLogRequest struct {
@@ -27,7 +23,9 @@ type CreateAuditLogRequest struct {
 	TargetID   *string `json:"targetId,omitempty"`             // resource_id or service_name
 
 	// Metadata (Payload without PII/sensitive data)
-	RequestMetadata    json.RawMessage `json:"requestMetadata,omitempty"`    // Request payload without PII/sensitive data
-	ResponseMetadata   json.RawMessage `json:"responseMetadata,omitempty"`   // Response or Error details
-	AdditionalMetadata json.RawMessage `json:"additionalMetadata,omitempty"` // Additional context-specific data
+	// Using JSONBRawMessage instead of json.RawMessage to avoid type conversion
+	// JSONBRawMessage implements json.Unmarshaler, so it works seamlessly with JSON decoding
+	RequestMetadata    JSONBRawMessage `json:"requestMetadata,omitempty"`    // Request payload without PII/sensitive data
+	ResponseMetadata   JSONBRawMessage `json:"responseMetadata,omitempty"`   // Response or Error details
+	AdditionalMetadata JSONBRawMessage `json:"additionalMetadata,omitempty"` // Additional context-specific data
 }