Version2025 #1
Draft
Version2025 #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…management while used in other threads
Publish schedule running status
Add a cache manager and api
Script to get a file
Cache use lock_guard and unique_lock Move step Executor specific data to ExecutorData ns_Executor::Local::CheckFinishedSteps iter on pid instead of using -1 Fix ns_Schedule::Schedule::~Schedule() ns_Schedule::Schedule::ScheduleLoop export step finished TasksManager save/restore status, replace atomic with a lock
Function now have $INPATH to access data files joigned at task creation LocalExecutor / executor.sh only update global env if step has no sibling Add an run_id to have, being a number between 0 to max nb of sibling - 1 scripts/get_file.sh add -q to be use in scripting Cache save data if loading data did some cleaning in file list Use weakly_canonical instead of canonical to store the path of an file that may not exist server/parts_handler support data in array Fix ns_Executor::Local::CheckFinishedSteps checking timed out steps Fix ns_Schedule::Schedule::ScheduleLoop timed out steps are now put in stepsDone
Update sample to make work ttf campaign Update executor.sh to provide more variable to running functions Add support for global configuration : config.json Config file are not validate in Load but in Validate instead, to allow build of config sample file Move Schdeule Tasks mangement into TasksManager Schedule reset status.json file at startup Save task.json in user_data/task_id/task_id.json Improve Schedule::ScheduleLoop Create Task element Move some step data in Task, step now refer their Task TasksManager now track managed Tasks id in his status file Fix Local initialisation of cpu cores Local now pin child process to specific cores Server now export api : tasks_running Improve handle of api rest response
Refactor core selection and cleanup logic; improve configuration handling - Sample/campaign.conf: improve how tlspuffin is killed when objective is reached - Rename all CPU-related fields to 'core' (Step, Local, LocalConfig) - Step: - Renamed ReadFromJSON to ReadFromTaskJSON - Moved cores_ to LocalData - FinalClean now delegates to task_ instead of executor_ - Task: - Stores Executor-specific data - Added FinalClean method - TaskManager: fix status save to occur *after* task deletion - LocalConfig: - Validate cores using CoresStats - Support new core config format: `excludeCores` + `nbCores` (optional) or `cores` - Add linux_cores.hxx/cxx: - Provides CoresStats and CoresMonitor for runtime core availability - Local: - Uses CoresMonitor to select most idle cores if config.nbCores != 0 - Fixed FinalClean to save path correctly - Cache/Config and Validate: - Store default (non-canonical) config path if config file is missing - Add `cpu.cxx` sample to test CoresStats and CoresMonitor behavior
api/server:
- Accepts `args[]` form fields via multipart HTTP requests.
- Validates that each `args[]` entry contains `=` and fails on duplicate keys.
- Passes parsed args to `ScheduleAPI::AddTask` and `Schedule::AddTask`.
schedule/task:
- Added `args_` member to `Task` for storing string key-value pairs.
- Serialized `args_` to JSON as array of { "key", "value" } objects.
- Serialized `root_steps_` as a JSON array of unique step IDs.
- Added `Task::ToJSON(...)` to centralize task serialization logic.
schedule/step:
- Refactored `Step::ToJSON` to embed full `Task` JSON under `"task"` field.
- Removed direct inclusion of task fields (e.g. files_path, run_path, etc.).
- Normalized JSON field name from `"nbCores"` to `"nbcores"` in deserialization.
executor/local:
- At first execution step, creates `.taskenv` file with one line per task arg.
Move run path management from Executor to Task Reorganize job folders and environment files under Task responsibility Move FinalClean from Executor to Task Relocate API endpoints from / to /api prefix Add API endpoint to retrieve job outputs (running and completed) Fix a missing lock in cache operations Add Schedule::GetExecutor(...) method Add string ID to Step class Add Executor::GatherFilesToLocal(...) virtual method for step file retrieval Support final_storage_path in job configuration with variable substitution (task_id and global variables) Add CORS support for API requests
Rename artifacts in campaign.conf Add Quatro HTML report generation in campaign.conf Script Rename STEP_ID to STEP_NAME for clarity Add report_generate scripts for Quarto HTML building Server Task manages a run_path where Executors store results after each execution Move run_path, logs_path, env_path and output_path from Executor to Task Task prepares run directory before first step execution Task maintains JSON file tracking executed steps, stored with final results for traceability Add step naming via 'id' parameter in configuration Preserve subdirectories when moving artifact directories
Change in task description. Added a configurations section to centralize and declare configurations. Declared configurations can be used in step run, they can be globally override in a step or in a specific run. Can still be declared and used in a specific run. Add step_configurations to store global configurations and compute overridden final configurations Step parameters were reworked, instead of being passed as args of executor.sh, they are injected in the script with their given name and value Add a publish section to export results FinalStoring actions are moved from Task to Publisher Use a enum for output state instead of a int Add a Task constructor Script: Added install of python required component for quatro scripts Web interface Started a web ui to manage tasks
moved src to src/scheduler
Add handler to cancel task or step, to request files and provide error Add ManageCORS to handle CORS request Update endpoints to match better rest API design File serving is to provide html ui Add html in server config to provides location of ui files At start will try restore last saved state Improve retrieve of step output Remove shutdown step at schedule loop end Add ns_Schedule::Publish::ToJSON Fix handlers still open when doing exec Add some ui pre work
can handle some variable in JSON description JSON description can setup a default task name handle task name provided by api when create a new task move ns_Schedule::Publish::ResolveVariables/ns_Schedule::Task::ResolveVariables in utils/variables handle SIGUSER1 to allow user to choice if task are shutdown at exit or not ns_Schedule::Task::PrepareToRun now prepare variables and resolve them ns_Executor::Local set itself as child subreaper improve wait end of process delete run folder once step done Shutdown now wait all childrens Analyzer Add commit cli option Remove unused code Adapt to publish server storage layout
* include reserve_port binary and install if missing * moved get_file_sh.h from src/scheduler/schedule/executor to src/scheduler/schedule * updated samples: added use of reserve_port and ipcrm for tlspuffin * improve GetOutput and all related functions * added toolsPath in config as the location to store general tools (get_file, reserve_port) * added task_id field when saving to JSON step, if no other task info was saved * request handler for running tasks now exports tasksmanager.json instead of status.json (to include all task information, not just running steps) Publish * added PublishAPI::Storage() * Server handle http(s) request for files * added request handler to manager errors Reserve_port * binary to allocate a free TCP port on 127.0.0.1 Misc * removed analyze (already present in scripts/reports_generate)
* added std::ios::app flag when reopening a task .steps.json file * added exit code value lost in Step (to handle lost processes) * Step now reloads ExecutorData only after all other fields have been reloaded * added an attribute in LocalData to better monitor lost processes if the scheduler is restarted * Local::CheckFinishedSteps can now track lost processes, if the scheduler is restarted
* add monitorsPath_ to Schedule::Config (runs/monitors), created by Monitor constructor * monitor files named as taskid-stepid.txt for unique identification per step * replace previous monitoring approach with inotify file-based system * implement StartMonitor in executor.sh allowing jobs, with monitoring configuration, to initiate monitoring at runtime * executor.sh provide StartMonitor to allow a step to run his monitoring * add sample monitoring functions and test cases in samples * add message_from_run field to steps for real-time monitoring feedback * add logs utilities for better debugging and support multithread use of output (LOGI, LOGE, LOGW macros) * update .gitignore pattern for embeded directories
CMake * add CMakeTextEmbedding.cmake for embedding text files as C++ headers * update CMakeUtils.cmake for text embedding support All embeded ressources are now at src/embeded Publisher * moved HTML/CSS/JS files to folder outside src, in html/publisher Scripts * refactor executor.sh with simplified execution flow * add functions.sh with reusable shell utilities * extract common functions from executor.sh to functions.sh Samples * renames campaign.conf in campaign.sh and experiment.json in campaign.json * add test_0.json and test_0.sh for basic testing * update monitoring.sh / monitoring.json Schedule * improve GetOutput, now also return file current size * moved access to files logic in src/utils/file.* * add cache port information for constructor to allow Executor::Local to provide it as executor.sh parameter * Executor::LocalData and Executor::LocalTaskData refactored to not store many time the same information * small refactor of executor Local * add Monitor::GetMessage to retrieve last monitor informations * Fix issue when trying to remove a no monitored step * Monitor::GetChange update message_from_run_ attributes of Step * monitor_path belong to Step instead of Monitor, as each step have a specific monitor file * fix wrong userPath read in Config object from Schedule * improve stability and fix bugs when cancel Step or Taks * Schedule implements ns_Executor::ExecutorsProvider interface * refactored the constructor part of Step, added end_processed_ to make easier to know if a task is done or not * added Step::MarkCancel * refactored the constructor part of Task, now also store the artefacts_path_ and the list of all steps * added Task::Cancel and a lock metadata_index_lock_ * TaskManager delegate most the Task building to Task / Step * json returned by RequestHandler is marked UTF-8
CMake * add CMakeTextEmbedding.cmake for embedding text files as C++ headers * update CMakeUtils.cmake for text embedding support All embeded ressources are now at src/embeded Publisher * moved HTML/CSS/JS files to folder outside src, in html/publisher Scripts * refactor executor.sh with simplified execution flow * add functions.sh with reusable shell utilities * extract common functions from executor.sh to functions.sh Samples * renames campaign.conf in campaign.sh and experiment.json in campaign.json * add test_0.json and test_0.sh for basic testing * update monitoring.sh / monitoring.json Schedule * improve GetOutput, now also return file current size * moved access to files logic in src/utils/file.* * add cache port information for constructor to allow Executor::Local to provide it as executor.sh parameter * Executor::LocalData and Executor::LocalTaskData refactored to not store many time the same information * small refactor of executor Local * add Monitor::GetMessage to retrieve last monitor informations * Fix issue when trying to remove a no monitored step * Monitor::GetChange update message_from_run_ attributes of Step * monitor_path belong to Step instead of Monitor, as each step have a specific monitor file * fix wrong userPath read in Config object from Schedule * improve stability and fix bugs when cancel Step or Taks * Schedule implements ns_Executor::ExecutorsProvider interface * refactored the constructor part of Step, added end_processed_ to make easier to know if a task is done or not * added Step::MarkCancel * refactored the constructor part of Task, now also store the artefacts_path_ and the list of all steps * added Task::Cancel and a lock metadata_index_lock_ * TaskManager delegate most the Task building to Task / Step * json returned by RequestHandler is marked UTF-8
remove: get_file.sh is no longer embedded, functionality moved to functions.sh add: SetCache in functions.sh to add a file to the cache server add: EndDirectChild in functions.sh to properly kill child processes change: monitor is killed at end of run, then run a last time change: when removing a file monitoring, update the monitor with the file content fix: cancel of Task not running is now properly managed fix: some rare cases of cancellation doing nothing
Executor/Local * add: some job information to the runtime bash environement Publish * add: restore lost goal field Request_Handler * add: support of download for txt file Samples * update: use new runtime bash environement * add: perf task * add: vulnerabilies-groupA task * add: dynamic use of C puts if available Doc * add: list of commit to check with perf/vuln
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
2025 rewrite of puffin-bench
Schedule/Run Task server
Display goal results server