Skip to content

Hyperparameter tuning for custom strategies #329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 290 commits into from
Sep 3, 2025
Merged

Hyperparameter tuning for custom strategies #329

merged 290 commits into from
Sep 3, 2025

Conversation

fjwillemsen
Copy link
Collaborator

@fjwillemsen fjwillemsen commented Sep 3, 2025 •
edited
Loading

This PR combines the hyperparameter tuning with the constrained optimization algorithms, new differential evolution implementation, and support for custom strategies and hyperparameter tuning thereof. It also brings updated hyperparameter defaults after extensive tuning.
In addition, it brings several performance improvements to the handling and lookup of neighbors in the search space:

  • Simultaneous caching of various neighbor methods.
  • Specific methods for finding a random neighbor.
  • Partial neighbor caching, which works by the fact that each neighbor lookup also produces partial information on the neighbors of that neighbor.

This pull request encompasses the following pull requests:

@fjwillemsen
Setup structure for Tensorspace in Searchspace
009cf01
@fjwillemsen
Implemented mappings and conversions to and from tensor to parameter …
33983f7 
…configuration
@fjwillemsen
Improved efficiency of acquisition function by removing evaluated con…
f3fc81b 
…figurations
@fjwillemsen
Removed Ax, added BOTorch as dependency
a5a0471
@fjwillemsen
Convenience script for benchmarking BO
9429539
@fjwillemsen
Added objective, tuning direction and hyperparameter tuning language …
176b8f5 
…selection support
@fjwillemsen
Completed implementation of mixed-type handling and handling of inval…
196af62 
…id and evaluated configurations
@fjwillemsen
Added docstrings, improved formatting
55a5c1a
@fjwillemsen
Extended strategies test to test for ability to handle non-numeric an…
d64f783 
…d mixed parameter values
@fjwillemsen
Mixed-type parameters are not converted to numeric constraints
e95ab30
@fjwillemsen
CostFunc can now encode and decode non-numeric configurations for str…
10a6a5c 
…ategies that require only numerics
@fjwillemsen
Fixed logging statements, improved formatting
6ae3ba6
@fjwillemsen
Improved the performance of get_bounds
4873a20
@fjwillemsen
Applied non-numeric encoding in differential evolution to handle non-…
bae7e96 
…numeric parameter values
@fjwillemsen
Implemented automatic conversion to multiple types for encoded tensor…
7eb7ef7 
… parameter lookup
@fjwillemsen
Added tests for Searchspace tensor encoding and conversion
91d3ce4
@fjwillemsen
Seperated strategies and runners test cache file
80d514e
@fjwillemsen
Implemented handling of categorical parameters
a489252
@fjwillemsen
Implemented variational GP and likelihood
68aee14
@fjwillemsen
Using LogExpectedImprovement to avoid stability issues
b9c012d
@fjwillemsen
Implemented tensor space bounds in searchspace
41ce663
@fjwillemsen
Implemented normalization for input features
07ef1d4
@fjwillemsen
Tensorspace is reduced by removing inconsequential parameters
721d072
@fjwillemsen
Extended strategies tests to include single parameter value
2434b3b
@fjwillemsen
Fixed an indexing error for tensorspace bounds
1679751
@fjwillemsen
Extended searchspace tests to include single parameter value
2b816a6
@fjwillemsen
Implemented additional acquisition functions, reduced number of reini…
c417585 
…tializations
@fjwillemsen
Implemented division of tensorspace into chunks for faster optimization
3d53b29
@fjwillemsen
Switch to fit_gpytorch_mll_torch for faster fitting, use approximate …
3ed43a6 
…mode
@fjwillemsen
Implemented running BO on GPU / Apple Silicon, settable precision
559813f
fjwillemsen and others added 23 commits July 17, 2025 22:55
@fjwillemsen
Implemented passing keyword arguments like meta strategy and time lim…
a89af66 
…it to hyperparameter tuning CLI
@fjwillemsen
Improved matching problem size check
8bd5991
@fjwillemsen
Implemented a discrete version of Latin Hypercube sampling that is (s…
89891a8 
…emi) constraint-aware
@fjwillemsen
Implemented efficiently representing the full search space as the ind…
b91a650 
…ices of the parameter values
@fjwillemsen
Implemented getting the numpy array as numerical values, where non-nu…
fa293cf 
…merical values are replaced with their index
@fjwillemsen
Implemented retrieving the true parameter configuration from a mixed …
0812b2b 
…numerical configuration
@fjwillemsen
Implemented parameter index-, numeric- and mixed- 2D views of the sea…
21d63bc 
…rch space
@fjwillemsen
Created additional tests for new and improved numpy views of the sear…
66f3f69 
…ch space
@fjwillemsen
Implemented a fast distributed random sample technique suitable for t…
3b1aeed 
…he discrete constraint search spaces
@fjwillemsen
Various improvements to and other searchspace functions
ce55392
@fjwillemsen
Performance improvement to __get_random_neighbor_adjacent
5ab83f8
@fjwillemsen
Implemented test for get_distributed_random_sample_indices
3a38865
@fjwillemsen
Implemented a new neighbor method for getting the neighbors clostest …
c43634b 
…to the parameter indices of the given configuration
@fjwillemsen
Wrote test for closest param indices neighbor method
0fe2ea5
@fjwillemsen
Implemented a LHS sampler, automatic adjustement to smaller integer s…
d0a5ba6 
…izes where possible for numpy arrays
@fjwillemsen
Implemented tests for the LHS sampler and the true index bounds
ce29740
@fjwillemsen
Using LHS and closest-param-indices in new diff_evo for much better p…
16c8245 
…erformance
@fjwillemsen
Disabled full validate on load for hyperparameter tuning
c044ef8
@fjwillemsen
Extended hyperparameters for optimization algorithms in paper
cde7823
@fjwillemsen
Updated required python-constraint version
f56c1fd
@fjwillemsen
Set new hyperparameter values for various strategies after more exten…
2097092 
…sive training
@fjwillemsen
Tweaked number of valid search iterations for hyperparameter tuning
ad41cfe
@nikivanstein
Hyperparametertuning custom strategies (#325)
301c421 
Adding default costfunc kwargs to allow LlaMEA to generate algorithms without these settings.
@fjwillemsen fjwillemsen marked this pull request as ready for review September 3, 2025 10:18
@sonarqubecloud
Copy link

sonarqubecloud bot commented Sep 3, 2025

Quality Gate Passed Quality Gate passed

Issues
114 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.2% Duplication on New Code

See analysis details on SonarQube Cloud

benvanwerkhoven
benvanwerkhoven approved these changes Sep 3, 2025
View reviewed changes
Copy link
Collaborator

@benvanwerkhoven benvanwerkhoven left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work!

@benvanwerkhoven benvanwerkhoven merged commit 5a48c0b into master Sep 3, 2025
4 checks passed
@benvanwerkhoven benvanwerkhoven mentioned this pull request Sep 3, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@benvanwerkhoven benvanwerkhoven benvanwerkhoven approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@fjwillemsen @benvanwerkhoven @nikivanstein