FTS3 job status updating failed when using multi-hop transfer

[zhangxt-ihep]

Hello,
We are using multi-hop for IHEP -> JINR transfer for JUNO experiment in IHEP's DIRAC instance. The transfer will go through CNAF storage, like: IHEP->CNAF->JINR. Basically it works fine, and most of the transfers could be successfully handled.
But we still met an issue: When the transfer failed in IHEP->CNAF part (as figure 1 shows), the FTS3Agent can no longer update job status (as figure 2 shows). Then the transformation requests got stuck with status "Submitted". Neither re-submit, nor cancelled, just stay in submitted.
Is their any methods to re-activate those stuck requests?

Thank you,
Xuantong

[chaen]

Can you please get the output of

import fts3.rest.client.easy as fts3
context = fts3.Context(endpoint=<yourFTS>)
fts3.get_job_status(context, <yourFtsGUID>, list_files=True)

Also, can you please copy paste errors rather than screenshots ? It's easier to search :-)

[zhangxt-ihep]

Here is a typical transfer status. We are using the multihop transfer, the transfer will go through IHEP->CNAF->JINR.
In this failure, when it failed at IHEP->CNAF part, the CNAF->JINR part state would be "Not_used".
But in DIRAC/DataManagementSystem/Client/FTS3Job.py, line 181, the for-loop cannot handle the 2 file states with "Failed" and "Not_used" pair. It just goes to line 221 and raise an error. Then, this transfer will get stuck, and no longer resumed.

[zhangxuantong@lxlogin003:155 16:49 ~]$ fts-rest-transfer-status -s https://fts3.ihep.ac.cn:8446 03861268-a80c-11f0-8acc-7cde78c49152 -j | jq .
{
"source_se": "https://junoeos01.ihep.ac.cn",
"vo_name": "juno",
"overwrite_flag": "Y",
"retry_delay": 0,
"source_spacetoken": null,
"dest_se": "https://juno-se-dr01.jinr.ru",
"reason": "One or more files failed. Please have a look at the details for more information",
"job_finished": "2025-10-13T08:41:51",
"os_project_id": null,
"destination_spacetoken": null,
"job_state": "FAILED",
"submit_time": "2025-10-13T08:10:00",
"copy_pin_lifetime": -1,
"job_type": "H",
"priority": 3,
"bring_online": -1,
"max_time_in_queue": null,
"verify_checksum": "t",
"cancel_job": false,
"archive_timeout": -1,
"submit_host": "fts3.ihep.ac.cn",
"internal_job_params": "nostreams:1",
"job_metadata": {
"operationID": 9146792,
"rmsReqID": 11009597,
"sourceSE": "IHEP-JUNOEOS",
"targetSE": "JINR-DCACHE",
"useTokens": false,
"auth_method": "certificate"
},
"job_id": "03861268-a80c-11f0-8acc-7cde78c49152",
"user_dn": "/C=CN/O=HEP/O=IHEP/OU=PHYS/CN=Kairui Huang",
"dst_file_report": false,
"cred_id": "5446836be3531bcf",
"retry": 3,
"http_status": "200 Ok",
"files": [
{
"dest_surl_uuid": null,
"source_surl": "https://junoeos01.ihep.ac.cn:9000/eos/juno/dirac/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd",
"archive_start_time": null,
"retry": 0,
"log_file": "/var/log/fts3/transfers/2025-10-13/junoeos01.ihep.ac.cn__xfer-archive.cr.cnaf.infn.it/2025-10-13-0841__junoeos01.ihep.ac.cn__xfer-archive.cr.cnaf.infn.it__8890253857__03861268-a80c-11f0-8acc-7cde78c49152",
"vo_name": "juno",
"dest_surl": "https://xfer-archive.cr.cnaf.infn.it:8443/juno/storm/dirac/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd",
"archive_finish_time": null,
"user_filesize": 293300104,
"activity": "default",
"source_se": "https://junoeos01.ihep.ac.cn",
"staging_host": null,
"internal_file_params": "nostreams:1,timeout:1160,buffersize:0",
"file_metadata": {
"desc": "MultiHop 29457036"
},
"scitag": null,
"log_debug": 0,
"dest_se": "https://xfer-archive.cr.cnaf.infn.it",
"reason": "TRANSFER [5] TRANSFER ERROR: Copy failed (3rd pull). Last attempt: Transfer failure: SocketTimeoutException while fetching https://junoeos01.ihep.ac.cn:9000/eos/juno/dirac/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd: Read timed out",
"pid": 2658338,
"staging_metadata": null,
"src_token_id": null,
"file_id": 8890253857,
"recoverable": true,
"priority": 3,
"filesize": 293300104,
"archive_metadata": null,
"staging_finished": null,
"dst_token_id": null,
"hashed_id": 64402,
"file_state": "FAILED",
"checksum": "ADLER32:9331e14a",
"tx_duration": 60.698,
"staging_start": null,
"file_state_initial": "",
"file_index": 0,
"transfer_host": "fts3.ihep.ac.cn",
"finish_time": "2025-10-13T08:41:51",
"throughput": 0.0,
"selection_strategy": "auto",
"job_id": "03861268-a80c-11f0-8acc-7cde78c49152",
"start_time": "2025-10-13T08:40:49",
"bringonline_token": null
},
{
"dest_surl_uuid": null,
"source_surl": "https://xfer-archive.cr.cnaf.infn.it:8443/juno/storm/dirac/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd",
"archive_start_time": null,
"retry": 0,
"log_file": null,
"vo_name": "juno",
"dest_surl": "https://juno-se-dr01.jinr.ru:2880/pnfs/jinr.ru/data/juno/juno/juno-reprod/ReProd25A/J25.6.0/9908-9943/9913/RUN.9913.JUNODAQ.Physics.ds-2.global_trigger.20250907120159.381_J25.5.0.esd",
"archive_finish_time": null,
"user_filesize": 293300104,
"activity": "default",
"source_se": "https://xfer-archive.cr.cnaf.infn.it",
"staging_host": null,
"internal_file_params": null,
"file_metadata": {
"desc": "Transfer 29457036",
"fileID": 29457036
},
"scitag": null,
"log_debug": null,
"dest_se": "https://juno-se-dr01.jinr.ru",
"reason": null,
"pid": null,
"staging_metadata": null,
"src_token_id": null,
"file_id": 8890253858,
"recoverable": null,
"priority": 3,
"filesize": null,
"archive_metadata": null,
"staging_finished": null,
"dst_token_id": null,
"hashed_id": 64402,
"file_state": "NOT_USED",
"checksum": "ADLER32:9331e14a",
"tx_duration": null,
"staging_start": null,
"file_state_initial": "",
"file_index": 1,
"transfer_host": null,
"finish_time": null,
"throughput": null,
"selection_strategy": "auto",
"job_id": "03861268-a80c-11f0-8acc-7cde78c49152",
"start_time": null,
"bringonline_token": null
}
],
"dm": []
}

[zhangxiaomei]

Another problem related is that the request jobs containing these multi-hop transfers got stuck in "scheduled" status after 2 attempts:

How to deal with those requests?