Fix TNS Decoder Rejection Bugs & Remove Obsolete Profiles

[nschimme]

This PR resolves a critical compatibility issue where generated AAC bitstreams would trigger frame rejections and audio dropouts in standard decoders like FFmpeg and FAAD.

The root cause was that the encoder allowed the Temporal Noise Shaping (TNS) long-block filter order to scale up to 20 for MPEG-4 streams at low sample rates. Per specification, the standard AAC-LC (Low Complexity) profile enforces a hard cap of 12.

Because libfaac is effectively a dedicated Low Complexity encoder—and the experimental code paths for MAIN, LTP, and SSR profiles were already disabled in the top-level configuration—this PR cascades that spec enforcement into a thorough cleanup of those dead, unmaintained profile paths.

---

🔍 Detailed Changes

1. Core TNS Spec Fix

• Enforced Cap: Reduced TNS_MAX_ORDER from 20 to 12 in coder.h.
• Refactored TnsInit: Eliminated complex, nested conditional switches that allowed the order to drift past spec constraints based on sample rates. It now statically applies safe, spec-compliant low-complexity bounds.

2. Codebase Refactoring & Profile Stripping

• FilterBank Streamlining: Removed the LTP-specific overlap_select logic. The function now cleanly processes normal overlapped data paths without checking for experimental non-overlapped modes.
• Macro Elimination: Deleted the obsolete MOVERLAPPED and MNON_OVERLAPPED definitions from filtbank.h.
• Frontend Cleanup: Simplified the CLI reporter in main.c to unconditionally state Object type: Low Complexity, removing the broken selection block.

---

📊 Testing & Verification

• Decoder Compatibility: Output bitstreams were validated against FFmpeg and FAAD. Frame drops and encoder rejection errors are entirely resolved.
• Audio Quality Metrics: Tested via ViSQOL at 16kHz to ensure the strict filter capping didn't degrade audio fidelity. The baseline audio actually saw a measurable objective improvement:
  • Baseline MOS: 4.02
  • Post-Fix MOS: 4.38

Benchmark shows at default parameters (TNS-off) no regressions which is expected: https://github.com/nschimme/faac/actions/runs/26182013325

---

💭 Reviewer Notes

Note on ABI Continuity: While the logic implementing the unsupported profiles has been stripped out to ensure the bug doesn't regress, the underlying constants themselves have been preserved in global headers where necessary to prevent breaking existing API boundaries for external consumers.

[fabiangreffrath]

Thank you!