Clean-up feature for Scipion projects

[JuhaHuiskonen]

The file number in large projects can become hard to manage. It would make sense to have a feature to delete those files that are not registered as outputs required by Scipion and that are not generally needed.

Each protocol could define files:

• that are necessary for Scipion workflow (will never be deleted)
• files that are useful in some cases outside of Scipion but not used in the workflow (deleted only if 'full' clean is selected, e.g. those diagnostic files that can't be visualised by Scipion)
• intermediate files that are very rarely needed (deleted if 'quick' or 'full' clean is selected, e.g. iterations 2...N-1)

The user could then select all the protocols they want to clean and select either of the two options (quick or full).

[pconesa]

Thanks Juga. We clearly need to do something. What I'm not sure is if it's more hassle for the developers to flag all possible output files or just delete those that are not final, like all iteration files in iterative protocols. El mié., 6 may. 2020 16:39, Juha Huiskonen <notifications@github.com> escribió:

…

The file number in large projects can become hard to manage. It would make sense to have a feature to delete those files that are not registered as outputs required by Scipion and that are not generally needed. Each protocol could define files: - that are necessary for Scipion workflow (will never be deleted) - files that are useful in some cases outside of Scipion but not used in the workflow (deleted only if 'full' clean is selected, e.g. those diagnostic files that can't be visualised by Scipion) - intermediate files that are very rarely needed (deleted if 'quick' or 'full' clean is selected, e.g. iterations 2...N-1) The user could then select all the protocols they want to clean and select either of the two options (quick or full). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2078>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF7ZYIL2ATS644MQDEOSF3RQFZCXANCNFSM4M2P7XPA> .

[JuhaHuiskonen]

The protocols that increase the file count the most seem to be RELION's particle polishing and CTF refinement (especially in projects with many many movies - as these protocols create several diagnostic files for each movie). Refinement and classification runs produces significantly fewer files.

[pconesa]

@azazellochg , @delarosatrevin , It might be a good time now maybe to review the amount of files relion keeps. If you have the chance, rethink if files kept are really useful for a regular user.

I'm mention here @dmaluenda to be aware of this for xmipp methods.

[delarosatrevin]

Yes, I plan to have a look at it...but maybe it would be good time to define one cleanUp method in base Protocol class...and make then actionable from the GUI. So it is ready for when each protocol decides to implement it. I have some ideas for a general implementation. We could discuss about it.

…

On Thu, May 7, 2020 at 8:21 PM Pablo Conesa ***@***.***> wrote: @azazellochg <https://github.com/azazellochg> , @delarosatrevin <https://github.com/delarosatrevin> , It might be a good time now maybe to review the amount of files relion keeps. If you have the chance, rethink if files kept are really useful for a regular user. I'm mention here @dmaluenda <https://github.com/dmaluenda> to be aware of this for xmipp methods. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2078 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAM6WG3OMFHCKNUFMHECQTRQL33HANCNFSM4M2P7XPA> .

[pconesa]

Anything that will make life easy for developers is welcome.

…

On 7/5/20 20:57, Jose Miguel de la Rosa Trevin wrote: Yes, I plan to have a look at it...but maybe it would be good time to define one cleanUp method in base Protocol class...and make then actionable from the GUI. So it is ready for when each protocol decides to implement it. I have some ideas for a general implementation. We could discuss about it. On Thu, May 7, 2020 at 8:21 PM Pablo Conesa ***@***.***> wrote: > @azazellochg <https://github.com/azazellochg> , @delarosatrevin > <https://github.com/delarosatrevin> , It might be a good time now maybe > to review the amount of files relion keeps. If you have the chance, rethink > if files kept are really useful for a regular user. > > I'm mention here @dmaluenda <https://github.com/dmaluenda> to be aware of > this for xmipp methods. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#2078 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/ACAM6WG3OMFHCKNUFMHECQTRQL33HANCNFSM4M2P7XPA> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2078 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF7ZYLWRT7JYYNUJNRBUXDRQMABJANCNFSM4M2P7XPA>.