implement dynamic emr clusters mode

[yanivkrol]

users will now be able to specify EMR cluster name/id/arn dynamically in the query. to enable this, set dyanmic_emr_cluster_mode in config.yaml to true. for usage with name, the cluster must be active (since cluster name can be resued)

• added "dyanmic_emr_cluster_mode" boolean in config this mode can't be used together with static servers specification

• all tool calls now require "server_spec" parameter

servers_spec = {
"static_server_spec": {
"server_name": str,
"default_client": bool
},
"dynamic_emr_server_spec": {
"emr_cluster_arn": str,
"emr_cluster_id": str,
"emr_cluster_arn": str
}
}

in static mode, the static_server_spec is used.
in dyanmic mode the dynamic_emr_server_spec is used.

• dynamically created spark clients are cached:

  • by arn: lifetime
  • by id: lifetime
  • by name: for the session

• created EMRclient to find the relevant cluster when needed

🔄 Pull Request

📝 Description

Brief description of changes and motivation.

🎯 Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 📖 Documentation update
• 🧪 Test improvement
• 🔧 Refactoring (no functional changes)

🧪 Testing

• ✅ All existing tests pass (task test)
• 🔬 Tested with MCP Inspector
• 📊 Tested with sample Spark data
• 🚀 Tested with real Spark History Server (if applicable)

🔬 Test Commands Run

# Example:
# task test
# npx @modelcontextprotocol/inspector uv run -m spark_history_mcp.core.main

🛠️ New Tools Added (if applicable)

• Tool Name: new_tool_name
• Purpose: What it does
• Usage: Example parameters

📸 Screenshots (if applicable)

✅ Checklist

• 🔍 Code follows project style guidelines
• 🧪 Added tests for new functionality
• 📖 Updated documentation (README, TESTING.md, etc.)
• 🔧 Pre-commit hooks pass
• 📝 Added entry to CHANGELOG.md (if significant change)

📚 Related Issues

Fixes #86
Related to #(issue number)

🤔 Additional Context

---

🎉 Thank you for contributing! Your effort helps make Spark monitoring more intelligent.

[nabuskey]

Are you still interested in continuing this work? If so, let's discuss in an issue.

[yanivkrol]

Are you still interested in continuing this work? If so, let's discuss in an issue.

It wasn't reviewed for a long time so some work obviously needs to be done, we can discuss (you can continue in the existing issue I guess). My PR as a fork is actually used in our company internally and is very useful.

It might be beneficial if the existing PR is reviewed before rebasing