Question about hyperparameter selection during testing

[KIKO-9]

Hi, I’ve noticed that some hyperparameters have a big impact on the results during testing. For example, in phonon calculations, the displacement parameter (default 0.03) — if I set it to 0.1, the KSRME actually gets lower. Similarly, the fmax used during relaxation also affects performance. When submitting model results for benchmarking, do I have to stick to the benchmark’s default settings, or can I use the configuration that gives the best results for my model?

[janosh]

you can report $\kappa_\text{SRME}$ at multiple $d$ values, any superset of displacement in {0.01, 0.03} is allowed. see discussion in #193

[KIKO-9]

Thanks for your reply!
If I test with the set {0.01, 0.03, 0.05, 0.1} and submit the $κ_{srme}$ for all four displacements, which one will be shown on the website?

[janosh]

0.01 is the default since that's what the DFT reference data uses but longterm there should be a UI for people to select which displacement value they want to see the metrics for

[xvzemin]

May I ask, in DFT calculations the displacement is 0.01, but in the default UIP test it is 0.03.
Is there any deeper significance behind this difference?

[janosh]

no significance whatsoever. it so happens that some models are insensitive or noisy in their force predictions at very small atomic displacements, which caused a lot of model developers to experiment with predicting lattice thermal conductivity at larger atomic displacements, usually 0.03 and/or 0.05. the reason many submitted models now use 0.03 was simply that i committed a test script where i forgot to revert the value of 0.03 to 0.01 before committing which was then copied multiple times by model author's before i noticed my mistake. the physical reason not to increase the displacement distance too much is that the further you displace the atoms, the more higher level anharmonic PES contributions may become important (for some materials more than others) which first of all are not captured in the k_SRME calculation (only considers 2nd and 3rd force constants) and which are also absent from the DFT reference data. so you let the model make predictions for a regime where more complex interactions might come into play but compare against predictions from DFT not in that same regime

[xvzemin]

Thanks for your reply! Your explanation is very clear.