How to improve performance of RequestManagementSystem

[zhangxiaomei]

Many operations (replicate, remove, register....) are using RequestManagementSystem. Sometimes two many requests slows down its performance. I would like to know:

1. Is the configuration parameters reasonable for RequestManagementSystem?
   
2. How to scale up the RequestExecutingAgent?
3. Why many cancelled requests can't be cleaned by the CleanReqDBAgent?
   
4. It is useful to increase groupSize for the requests to reduce burden of RequestManagementSystem?

[nifuki]

Just to add: the RMS components are listed as scalable in the Dublications table, so it should be possible to improve performance by running additional instances.

[zhangxiaomei]

That is what I want to know. How to do it, especially running additional instances in another machine?

[fstagni]

Hi, few answers:

1. for RequestExecutionAgent, you can pump up:

• RequestsPerCycle (e.g. put 300)
• BulkRequest (e.g. again 300)
  For CleanReqDBAgent you can increase DeleteLimit and KickLimit easily (e.g. 10000 for both)

2. you can add another instance(s) of RequestExecutingAgent on a second machine, but maybe with the adjustments above you'll be alright

3. Maybe the agent is too slow with your current settings. Also, I'd switch the option of CleanReqDBAgent DeleteFailed to True

4. probably yes

[fstagni]

Service or agent is the same. Use the sysadmin-cli : https://dirac.readthedocs.io/en/latest/AdministratorGuide/HowTo/SystemAdministratorInterface.html

[zhangxiaomei]

The CleanReqDBAgent has done a lot of cleaning successfully after changing DeleteFailed to True, but after it finished cleaning, there are still some quite old "cancelled" Requests. Do you know what happened to those requests?

[fstagni]

I can't know just by that screenshot. You can investigate further with dirac-rms-request script

[zhangxiaomei]

I think I found the reason. These requests belongs to some transformations which can't be correctly archived. For example, the request 4400002 [1] belongs to the transformation 4030, which has problems to be automatically archived [2]. Since these transformations keep in "completed" status, the related requests keep being updated.
[1]
(base) [zhangxm@lxlogin003 2025]$ dirac-rms-request 4400002

Request name='00004030_00095479' ID=4400002 Status='Failed' ('Canceled' in DB)
Created 2025-01-20 08:21:51, Updated 2025-04-29 13:43:52, NotBefore 2025-01-20 08:29:05
Owner: '/C=CN/O=HEP/O=IHEP/OU=CC/CN=Xiaomei Zhang', Group: juno_user
[0] Operation Type='ReplicateAndRegister' ID=7639539 Order=0 Status='Failed'
TargetSE: JINR-EOS - Created 2025-01-20 08:21:51, Updated 2025-01-23 13:55:32
[01] ID=50217905 LFN='/juno/Commissioning_Dryrun/OSIRIS/2023/0908/TCP_7150_1_09-08-1001_1008_1000' Status='Failed' Checksum='441bd167' Attempts=2

[2]
(base) [zhangxm@lxlogin003 2025]$ dirac-transformation-archive 4030

Will consider the following transformation types: ['Calibration-JUNO', 'DataReconstruction-JUNO', 'DataStripping-JUNO', 'ElecSimulation-JUNO', 'MCSimulation-JUNO', 'MCStripping-JUNO', 'Merge-JUNO', 'Removal-JUNO', 'Replication-JUNO', 'Transfer-JUNO']
Will search for directories in the following locations: ['MetadataCatalog', 'TransformationDB']
Will use TransformationID as metadata tag name for TransformationID
Will archive Completed transformations after 1 days
Will remove logs found on storage element: LogSE
Archiving transformation 4030
Found 95642 tasks for transformation

[zhangxiaomei]

I think I have solved both transformation cleaning and requests cleaning. The problems of transformation cleaning is due to miss shifterProxy and the problems of request cleaning is due of continuous update from the transformation.