TL;DR: We propose an effective framework, NeuroSketch, for neural decoding via systematic architectural optimization.

In this study, we propose NeuroSketch, an effective framework for neural decoding via systematic architecture optimization. Starting with the basic architecture study, we find that CNN-2D outperforms other architectures in neural decoding tasks and explore its effectiveness from temporal and spatial perspectives. Building on this, we optimize the architecture from macro- to micro-level, achieving improvements in performance at each step. The exploration process and model validations take over 5,000 experiments spanning three distinct modalities (visual, auditory, and speech), three types of brain signals (EEG, SEEG, and ECoG), and eight diverse decoding tasks. Experimental results indicate that NeuroSketch achieves state-of-the-art (SOTA) performance across all evaluated datasets, positioning it as a powerful tool for neural decoding. The complete experimental results are available at https://github.com/Galaxy-Dawn/NeuroSketch_Results.

🚀 News

• 📣 2025/12/11: We introduced a new variant of NeuroSketch for speech decoding, named NeuroSketch‑SD. This version contains approximately 20M parameters, offering a higher model capacity for decoding speech information from brain signals. Specifically, we have optimized the feature dimension across stages, the downsampling strategy, the convolutional design, the FFN structure, and the activation function to enhance decoding performance.
• 📣 2025/12/9: We made NeuroSketch public!

Roadmap

Our study is guided by two core questions that aim to systematically investigate how to design an effective framework for neural decoding:

• Q1: Which basic model architecture is best suited to capture temporal and spatial patterns in brain signals?

• Q2: Based on an appropriate architecture, how can we improve neural decoding performance through macro-to-micro architectural optimization?

As shown in Figure 1, by answering the above questions, we consistently improve the decoding performance at each optimization step.

Model Architecture

After the optimizations mentioned above, we have developed an effective framework, NeuroSketch. The overall architecture is shown in Figure 2(c). Like CNN-2D, NeuroSketch employs a 2D input representation, which is then processed by a stem layer to initiate feature extraction. The following forward propagation process is divided into four stages to progressively capture features from low-level to high-level. Following the four stages, the resulting feature is passed through a generalized mean (GeM) pooling layer that aggregates features along the channel and temporal dimensions. The pooled representation is then fed into a linear layer for the final classification. Additional details of the architecture and implementation can be found in Appendix D.2.

Main Results

As shown in Figure 3, NeuroSketch outperforms all selected baselines, underscoring its strong capacity to model brain signals across diverse decoding scenarios.

⚡Quick Start

📁 Dataset Preparation

Dataset	URL
Du-IN	Download
Chisco	Download
FacesHouses	Download
ThingsEEG	Download
SEED-DV	Apply
OpenMIIR	Download

Note: SEED-DV dataset requires manual application approval before download.

🔧Setup

1. Clone Repository

git clone https://github.com/Galaxy-Dawn/NeuroSketch.git

2. Configure Paths

cd ./run/conf/dir/

Modify local.yaml to set up your local environment paths in YAML format.

# Directory Configuration

# Path configurations for data and output management

data_dir:          # Path to raw downloaded datasets (required)
processed_dir:     # Path for processed/stage data storage (required)
output_dir:        # Training output directory (recommended: /root/NeuroSketch/output)
logging_dir:       # Training logs directory (recommended: /root/NeuroSketch/output/log)
model_dir:         # Weights & Biases (wandb) output directory (recommended: /root/NeuroSketch/output/train)
model_save_dir:    # Directory for saved model checkpoints
sub_dir:           "./"  # Relative path supplement (default: current directory)

3. Prepare dataset

cd ./run/scripts/prepare_data
# Run individual dataset scripts

Outputs will be saved to your configured processed_dir in local.yaml

🔬 Experiments

1.Roadmap Part

cd ./run/scripts/roadmap
# Run individual scripts

2.Experiment Part

cd ./run/scripts/exp
# Run individual scripts

📖 Citation

If you find this work useful for your research or applications, please cite using this BibTeX:

@misc{zhang2025neurosketcheffectiveframeworkneural,
      title={NeuroSketch: An Effective Framework for Neural Decoding via Systematic Architectural Optimization}, 
      author={Gaorui Zhang and Zhizhang Yuan and Jialan Yang and Junru Chen and Li Meng and Yang Yang},
      year={2025},
      eprint={2512.09524},
      archivePrefix={arXiv},
      primaryClass={q-bio.NC},
      url={https://arxiv.org/abs/2512.09524},