Commit Graph

18 Commits

Author SHA1 Message Date
Simone Gotti 2c3e6bf9e4 wal: update and rename to datamanager
* Rename to datamanager since it handles a complete "database" backed by an
objectstorage and etcd

* Don't write every single entry as a single file but group them in a single
file. In future improve this to split the data in multiple files of a max size.
2019-04-26 16:00:03 +02:00
Simone Gotti 41e333d7ec *: rename "lts" to "ost"
`lts` was choosen to reflect a "long term storage" but currently it's just an
object storage implementation. So use this term and "ost" as its abbreviation
(to not clash with "os").
2019-04-27 15:16:48 +02:00
Simone Gotti 07bc4a21ff runservice scheduler: automatically remove dynamic executors 2019-04-24 13:25:41 +02:00
Simone Gotti 8bde2f2bc0 runservice: implement caching
Add `save_cache` and `restore_cache steps`
2019-04-13 14:58:56 +02:00
Simone Gotti 3928851c10 runservice: rename Run.RunTasks to Run.Tasks 2019-04-12 17:45:38 +02:00
Simone Gotti da27348a1d runservice: implement run setup errors
Add the ability to define a run with a setuperror phase.

When the run setup has errors client could submit a run with a list of setup
errors. In such case the run will be created in the setuperror phase.

Setup errors are currently generated by the webhook receiver and the run service
when it checks the run config for possible issues.
2019-04-09 16:51:37 +02:00
Simone Gotti 671b89d391 runservice: merge RunConfig and RunData
* Use just RunConfig
* Use StaticEnvironment vs Environment in RunConfig to distinguish between env
that won't change at run recreation from env that could change at every
recreation
* The RunCreate api will just receive the runtasks instead of a runconfig (more
right)
2019-04-09 18:11:00 +02:00
Simone Gotti 3642be6f21 */api: Use helpers for error handling
* client: always parse the json error message field and return its contents
* Use ErrBadRequest and ErrNotFound in every handler and command
* Gateway: by default pass underlying service error (configstore, runservice) to
client keeping the status code and message. In future, if some errors must be
masked, we should change the specific parts that need special handling.
2019-04-09 14:53:00 +02:00
Simone Gotti 643dfe4072 runservice api: improve response handling
* Command: use ErrBadRequest
* Always return a json message also on error. For internal errors return a
generic "internal server error" message to not leak the real internal error to
clients
* Return 201 Created on resource creation
* Return 204 No Content on resource deletion and other action with no json
output
2019-04-08 18:04:42 +02:00
Simone Gotti 7d787c5f77 *: implement task approval 2019-04-08 17:29:57 +02:00
Simone Gotti 04f3905ea1 client: fix content type header case 2019-04-08 12:28:15 +02:00
Simone Gotti f3781c9087 *: fix rest methods
* use POST instead of PUT for resource creation
* use PUT instead of POST for resource special actions
2019-04-08 08:54:45 +02:00
Simone Gotti e46766829c runservice: rework store and readdb logic
* Remove all the small index files on the lts
* Keep on s3 only a full index of all runs containing the runid, grouppath and phase
  million of runs can take only some hundred of megabytes
* Periodically create a new dump of the index
2019-03-29 12:15:48 +01:00
Simone Gotti 48ab496beb *: add api to query last run per group 2019-03-29 12:00:18 +01:00
Simone Gotti 8f4a5b29b9 *: implement setup step 2019-03-13 15:48:35 +01:00
Simone Gotti f09602cdc3 *: implement run stop 2019-03-08 10:02:37 +01:00
Simone Gotti 9d2c133817 runservice implement initial basic run restart 2019-03-04 16:11:18 +01:00
Simone Gotti 36fc79dfc6 runservice: initial commit 2019-02-21 15:54:50 +01:00