Commit Graph

12 Commits

Author SHA1 Message Date
Simone Gotti f5cf3b9fa7 runservice: check changegroup name 2019-04-29 10:12:34 +02:00
Simone Gotti 2c3e6bf9e4 wal: update and rename to datamanager
* Rename to datamanager since it handles a complete "database" backed by an
objectstorage and etcd

* Don't write every single entry as a single file but group them in a single
file. In future improve this to split the data in multiple files of a max size.
2019-04-26 16:00:03 +02:00
Simone Gotti 41e333d7ec *: rename "lts" to "ost"
`lts` was choosen to reflect a "long term storage" but currently it's just an
object storage implementation. So use this term and "ost" as its abbreviation
(to not clash with "os").
2019-04-27 15:16:48 +02:00
Simone Gotti 8bde2f2bc0 runservice: implement caching
Add `save_cache` and `restore_cache steps`
2019-04-13 14:58:56 +02:00
Simone Gotti 3928851c10 runservice: rename Run.RunTasks to Run.Tasks 2019-04-12 17:45:38 +02:00
Simone Gotti 751361daea runservice: refactor scheduling logic
* split functions in sub parts to ease future testing
* save run fewer times
* rework events logic to considere both run phase and result changes (emit an
event on every phase or result change)
2019-04-10 14:48:47 +02:00
Simone Gotti 671b89d391 runservice: merge RunConfig and RunData
* Use just RunConfig
* Use StaticEnvironment vs Environment in RunConfig to distinguish between env
that won't change at run recreation from env that could change at every
recreation
* The RunCreate api will just receive the runtasks instead of a runconfig (more
right)
2019-04-09 18:11:00 +02:00
Simone Gotti fc891409ca update wal and readdb 2019-04-01 12:54:43 +02:00
Simone Gotti e46766829c runservice: rework store and readdb logic
* Remove all the small index files on the lts
* Keep on s3 only a full index of all runs containing the runid, grouppath and phase
  million of runs can take only some hundred of megabytes
* Periodically create a new dump of the index
2019-03-29 12:15:48 +01:00
Simone Gotti 21447fc59d etcd: allow specifying a revision for a get 2019-03-29 11:37:22 +01:00
Simone Gotti 8f4a5b29b9 *: implement setup step 2019-03-13 15:48:35 +01:00
Simone Gotti 36fc79dfc6 runservice: initial commit 2019-02-21 15:54:50 +01:00