Force errors question about qe.traj type of data

[AlexanderOnly]

I'm training with data from qe.traj , but the force error has been hard to drop.
Even I trained with 100w or 200w steps，the force error is still greater than 0.5eV.

# batch      l2_tst        l2_trn         l2_e_tst     l2_e_trn     l2_f_tst     l2_f_trn         lr
 145100    8.69e+00  9.35e+00    3.65e-03  2.33e-03    6.76e-01  7.27e-01    6.6e-04
 145200    8.76e+00  5.09e+00    6.11e-03  1.02e-02    6.81e-01  3.96e-01    6.6e-04
 145300    8.68e+00  4.14e+00    2.93e-03  5.78e-03    6.75e-01  3.22e-01    6.6e-04
 145400    8.73e+00  1.01e+01    5.93e-03  8.11e-03    6.79e-01  7.90e-01    6.6e-04
 145500    8.68e+00  8.93e+00    6.12e-03  3.71e-03    6.75e-01  6.95e-01    6.6e-04
 145600    8.73e+00  9.73e+00    1.13e-02  1.12e-02    6.79e-01  7.57e-01    6.6e-04
 145700    8.67e+00  5.62e+00    4.48e-03  1.86e-03    6.75e-01  4.37e-01    6.6e-04

I suspect it's a data problem. So I used the example from dpdata for training.
It can see that the energy dropped quickly, but the force error was still hard to drop.

# batch      l2_tst        l2_trn         l2_e_tst     l2_e_trn     l2_f_tst     l2_f_trn         lr
  16000    1.69e+01  1.25e+02    1.05e+00  8.57e-01    6.63e-01  4.91e+00    6.5e-04
  16100    1.69e+01  1.25e+02    1.01e+00  8.49e-01    6.63e-01  4.91e+00    6.5e-04
  16200    1.69e+01  1.25e+02    1.01e+00  8.35e-01    6.63e-01  4.91e+00    6.5e-04
  16300    1.69e+01  1.25e+02    9.74e-01  8.22e-01    6.63e-01  4.91e+00    6.5e-04
  16400    1.69e+01  1.25e+02    9.67e-01  8.12e-01    6.63e-01  4.91e+00    6.5e-04
  16500    1.69e+01  1.25e+02    9.17e-01  7.98e-01    6.63e-01  4.91e+00    6.5e-04
  16600    1.69e+01  1.25e+02    9.00e-01  7.86e-01    6.63e-01  4.91e+00    6.5e-04
  16700    1.69e+01  1.25e+02    8.66e-01  7.69e-01    6.63e-01  4.91e+00    6.5e-04
  16800    1.69e+01  1.25e+02    8.46e-01  7.64e-01    6.63e-01  4.91e+00    6.5e-04
  16900    1.69e+01  1.25e+02    8.12e-01  7.41e-01    6.63e-01  4.91e+00    6.5e-04

Here's my input files:

{
    "model": {
        "type_map": [
            "O",
            "H"
        ],
        "descriptor": {
            "type": "se_a",
            "sel": [
              68,80
            ],
            "rcut_smth": 0.5,
            "rcut": 5.0,
            "neuron": [
                25,50,100
            ],"axis_neuron": 16,
            "resnet_dt": false
        },
        "fitting_net": {
            "neuron": [
            240,240,240],
            "resnet_dt": true
        },
        "data_stat_nbatch": 10
    },
    "loss": {
        "start_pref_e": 0.02,
        "limit_pref_e": 1,
        "start_pref_f": 1000,
        "limit_pref_f": 1,
        "start_pref_v": 0,
        "limit_pref_v": 0,
        "start_pref_ae": 0,
        "limit_pref_ae": 0
    },
    "learning_rate": {
        "type": "exp",
        "start_lr": 0.001,
        "decay_steps": 5000,
        "decay_rate": 0.96,
        "stop_lr": 1e-08
    },
    "training": {
        "systems": [
        "../traj"
],
        "set_prefix": "set",
        "stop_batch": 400000,
        "batch_size": "auto",
        "disp_file": "lcurve.out",
        "disp_freq": 100,
        "numb_test": "10%",
        "save_freq": 1000,
        "save_ckpt": "model.ckpt",
        "load_ckpt": "model.ckpt",
        "disp_training": true,
        "time_training": true,
        "profiling": false,
        "profiling_file": "timeline.json"
    }
}

Training data that from dpdata is attached here.
traj.zip

[AlexanderOnly]

I try to change 'length_convert' to 1, and it works.