Commit Graph

463 Commits

Author SHA1 Message Date
Simone Gotti 32a08ec5c8
Merge pull request #179 from sgotti/runservice_logshandler_improve_errors
runservice: improve errors in logsHandler
2019-11-14 09:56:24 +01:00
Simone Gotti 66e182a55d runservice: improve errors in logsHandler
* return errNotExist in readTaskLogs when the executor task doesn't exist: so
the client will receive a 404 instead of a 500 (since a generic error will be
mapped to a 500).
* Wrap the errNotExist returned by readTaskLogs with a new ErrNotExits reporting
"log doesn't exist"
2019-11-13 15:50:20 +01:00
Simone Gotti 07cde065c8 runservice: use etcd mutex TryLock on fetching
When fetching avoid concurrent fetches from multiple runservices using an etcd
mutex TryLock.
2019-11-13 11:53:54 +01:00
Simone Gotti 9fd4b662a8
Merge pull request #175 from camandel/api_logarchived
gateway: add api to get log status
2019-11-13 11:45:31 +01:00
Carlo Mandelli 8ed813e722 gateway: add api to get log status 2019-11-13 10:01:51 +01:00
Simone Gotti 5ab9f7c970 *: use etcd mutex TryLock
etcd PR 11104 (https://github.com/etcd-io/etcd/pull/11104) implemented mutex
TryLock. Since it's only available in etcd master just copy relevant code and
add a TODO to remove it when updating the etcd client to a version implementing
TryLock.

Use TryLock everywhere where it'll be useful.
2019-11-12 22:27:17 +01:00
Simone Gotti 24a9563872
Merge pull request #167 from sgotti/datamanager_writedatasnapshot_skip_already_checkpointed_wals
datamanager: skip already applied wals in writeDataSnapshot
2019-11-12 14:57:42 +01:00
Simone Gotti d679254516 readdb: improve HandleEvents goroutine exiting
Rename errCh to doneCh (error is not needed) and always send to it when one of
the HandleEvents functions exits (not only on error).

This will ensure that all the goroutines will be stopped also if one of them
returns without an error.
2019-11-12 11:03:21 +01:00
Simone Gotti 1e70e3404b datamanager: skip already applied wals in writeDataSnapshot
As an optimization don't apply already applied wals.
2019-11-12 11:02:52 +01:00
Simone Gotti dfd0f8c806 datamanager tests: increase etcd waitdown timeout 2019-11-12 10:47:28 +01:00
Simone Gotti 72f279c4c3 *: improve error handling
* objectstorage: remove `types` package and move `ErrNotExist` in base package
* objectstorage: Implement .Is and add helper `IsErrNotExist` for `ErrNotExist`
* util: Rename `ErrNotFound` to `ErrNotExist`
* util: Add `IsErr*` helpers and use them in place of `errors.Is()`
* datamanager: add `ErrNoDataStatus` to report when there's not data status in ost
* runservice/common: remove `ErrNotExist` and use errors in util package
2019-11-11 12:17:35 +01:00
Simone Gotti 5af07d0852 objectstorage: use a single package
remove all the subpackages and just use a single package
2019-11-08 16:31:48 +01:00
Simone Gotti 35e1ec0e15 datamanager: remove old storage wals
Remove all wals not required by the existing data status files and not existing
in etcd.
2019-11-08 15:39:17 +01:00
Simone Gotti 9c0eb3d7ef datamanager: refactor ReadWal
make ReadWal directly return a *WalHeader
2019-11-08 13:24:43 +01:00
Simone Gotti 9c1f3b2a69 datamanager: check wal previouswalsequence is correct in initEtcd 2019-11-07 17:05:40 +01:00
Simone Gotti acd62a3f90 datamanager: don't create ost wal checkpointed files
currently creating .checkpointed files in the ost isn't useful.
We already have the data snapshot that reports the last checkpointed wal.
2019-11-07 10:32:41 +01:00
Simone Gotti 4fcb067052 datamanager: clean old data files
keep the last n (now set to 3) data status files and remove all other data status
files and unneeded data files.
2019-11-07 10:30:31 +01:00
Simone Gotti 52cb683267 datamanager: fix index creation on multiple data files
When during a checkpoint more than one file is created the entries position in
the index is not right since it's not reset at every new index.

Fix it and add related tests.
2019-11-06 15:33:40 +01:00
Carlo Mandelli aab2321d58 services: check config only for enabled services 2019-11-05 09:25:22 +01:00
Simone Gotti 4b4416fc99 datamanager: add data sequence to data file name
When creating a datafile name make it start with the current data sequence. This
is useful in future to know which data sequence created a new data file.
2019-11-04 09:23:12 +01:00
Simone Gotti e06dc332e2 sequence: add tests for String and Parse methods
Add tests to sequence String() and Parse(string) methods.
2019-10-31 16:53:57 +01:00
Simone Gotti e18794764e go.mod: update dependencies
Update all the updatable dependencies
2019-10-29 09:31:38 +01:00
Carlo Mandelli 7a51404b71 run config: add tty option for run steps 2019-10-28 16:58:54 +01:00
Simone Gotti 1eb16886d8 objectstorage: add WriteObject tests
Test WriteObject using different size values: unspecified, equal to the buffer
size or less than the buffer size.
2019-10-25 15:16:42 +02:00
Simone Gotti 2e520dae55
Merge pull request #154 from sgotti/objectstorage_object_size
objectstorage: return object size in objectinfo
2019-10-25 12:23:48 +02:00
Simone Gotti 4c88bb75a3
Merge pull request #153 from sgotti/objectstorage_posix_limitreader_only_size_gt_0
objectstorage posix: use limitreader only when size is specified.
2019-10-25 12:23:35 +02:00
Simone Gotti ae1e92b115 objectstorage: return object size in objectinfo
Return object size in object info.
2019-10-25 10:53:36 +02:00
Simone Gotti 58f68601e6 objectstorage posix: use limitreader only when size is specified.
Use limitreader only when size is specified (greater or equal to 0).
When size is unknown (less than 0) limitreader will immediately return EOF
instead of writing the whole data.
2019-10-25 10:26:59 +02:00
Simone Gotti 0388003d09 objectstorage s3: use limitreader in write object
If size is specified limit reads to size bytes.
2019-10-25 10:06:22 +02:00
Simone Gotti a0450a5e69
Merge pull request #146 from sgotti/gitea_fix_getref
gitea: use GetRepoRefs instead of GetRepoRef
2019-10-23 10:29:23 +02:00
Simone Gotti f0e4bbfeeb gitea: use GetRepoRefs instead of GetRepoRef
Looks like GetRepoRef doesn't correcly handle gitea repo refs response expecting
a single entry. Instead, at least with latest gitea version, the response is
always an array of refs. So use GetRepoRefs.
2019-10-22 10:17:00 +02:00
Simone Gotti 446e626f9f gateway: fix project create run http method
Make it a POST instead of a PUT.
2019-10-22 09:50:01 +02:00
Carlo Mandelli 6fccb935c4 docker: mount multiple volumes 2019-10-17 09:21:46 +02:00
Simone Gotti 3d0c68b5fc gitsources: don't set branch value when in a pull request
We were passing the source branch name as the Branch value in the webhook data.

This patch will just delete this assignment. If in future it's needed let's add
it with a different name to not cause confusion.
2019-10-14 22:42:08 +02:00
Simone Gotti fa4b41ab74 when: match only the current ref type
Only match the current ref type, ie: don't match a branch when the ref type is a
tag or pull request.

Ref is always matched because it's not related to a specific ref type.
2019-10-14 17:08:44 +02:00
Simone Gotti 7d62481415 *: implement ability to add tmpfs volumes to containers
* Add a generic container volume option that currently only support tmpfs. In
future it could be expanded to use of host volumes or other kind of volumes (if
supported by the underlying executor)

* Implement creation of tmpfs volumes in docker and k8s drivers.
2019-10-08 16:36:23 +02:00
Carlo Mandelli 4fc8f3ebed cmd: add version command 2019-10-05 12:11:18 +02:00
Simone Gotti a7ca2848e6 datamanager: remove some logs from tests 2019-10-02 09:27:55 +02:00
Simone Gotti 11da186913 gateway: add parentRef field to project group update api
and make all the request fields optional
2019-10-01 14:39:45 +02:00
Simone Gotti 973cfe8770 configstore: implement project group move 2019-10-01 12:03:57 +02:00
Simone Gotti e2a0fedfb8 configstore: disable root project group deletion 2019-10-01 10:26:17 +02:00
Simone Gotti 8b9464486d gateway: add parentRef field to project update api
and make all the request fields optional
2019-09-27 17:10:04 +02:00
Simone Gotti fe9e3e8317 configstore: implement project move 2019-09-27 11:07:49 +02:00
Carlo Mandelli cd8175c156 add clone options 2019-09-25 09:29:39 +02:00
Simone Gotti f5c0e91f39 gateway: api implement get secrets removeoverridden
Implement missing removeoverridden option for get secrets.
2019-09-20 15:39:21 +02:00
Carlo Mandelli 436aa8f1de skip run with special commit 2019-09-18 11:53:22 +02:00
Simone Gotti 714da3ffe3 executor: add missing mutex unlock
Add missing mutex unlock that will cause deadlocks.
2019-09-18 09:49:38 +02:00
Simone Gotti 9cfb21d365 gateway: run api return more step data
For every step return the step type.

For a run step also return the shell and exit status.
2019-09-17 17:43:48 +02:00
Simone Gotti 39829f1ec4 runservice: save step exitstatus in run.
For every step save also the command exit status.
2019-09-17 14:35:37 +02:00
Simone Gotti 12b02143b2 runservice: don't save executor task data in etcd
Reorganize ExecutorTask to better distinguish between the task Spec and
the Status.

Split the task Spec in a sub part called ExecutorTaskSpecData that contains
tasks data that don't have to be saved in etcd because it contains data that can
be very big and can be generated starting from the run and the runconfig.
2019-09-17 12:03:43 +02:00