Commit Graph

380 Commits

Author SHA1 Message Date
Simone Gotti d91ec09d7d config: add starlark config support
Handle `.agola/config.star` files in starlark config format.
To provide a context like done for jsonnet we require that the starlark agola
config file contains a main function that will receive a config context as a
dict.
We also had to implement our own json conversion from a starlark dict since go
starlark removed its own function.
2020-03-10 13:29:20 +01:00
Simone Gotti 7094de908b run: export also ref type environment variable to runs 2020-03-09 10:47:58 +01:00
Simone Gotti 627f40987e run: export also pull request id environment variable to runs 2020-03-09 09:42:51 +01:00
Simone Gotti 1820c0247c config: provide jsonnet context top level argument 2020-03-06 11:19:49 +01:00
Simone Gotti 0e2b01a586 runservice: correctly handle skipped tasks in fetcher
skip fetching of tasks with status skipped, not only tasks marked as skip.
This avoid many wrong an noisy logs of type "executor task with id taskid
doesn't exist. This shouldn't happen. Skipping fetching"
2020-03-02 10:40:59 +01:00
Simone Gotti eb180da914
Merge pull request #225 from sgotti/runservice_fix_handling_of_wrong_executortask_status
runservice: fix handling of wrong executortask status
2020-03-02 10:26:32 +01:00
Simone Gotti 4da7c23bb8
Merge pull request #224 from sgotti/executor_use_cancellable_context_in_executetask
executor: use cancellable context in executetask
2020-03-02 10:26:16 +01:00
Simone Gotti 382705bde9
Merge pull request #223 from sgotti/executor_fix_stopping_of_not_running_task
executor: fix stopping of not running tasks
2020-03-02 09:45:39 +01:00
Simone Gotti 19611c18e7 runservice: fix handling of wrong executortask status
updateRunTaskStatus should also accept transitions from not started to a
finished state like "success", "failed", "stopped" since we could miss some
status updates from the executor for many reasons.
2020-02-28 13:02:35 +01:00
Simone Gotti e4507446ed executor: use cancellable context in executetask
Use a cancellable context to handle running task stop.
When the context is done the pod will be stopped.
2020-02-28 10:52:36 +01:00
Simone Gotti 97d145a9d3 executor: fix stopping of not running tasks
When a related runningTask doesn't exist and the executor task status is
running just report it as failed ignoring if it's marked to stop.
2020-02-28 10:45:07 +01:00
Simone Gotti 3ac018e6e5 runservice: use all scheduled tasks in scheduleRun
rename activeExecutorTasks to scheduledExecutorTasks and don't filter out
finished tasks.
In some logic we need all the scheduled tasks and not only the not finished
ones.
2020-02-28 09:56:12 +01:00
Simone Gotti a4e280cd7d
Merge pull request #222 from sgotti/executor_fix_reporting_of_stopped_tasks_and_steps
executor: fix reporting of stopped tasks and steps
2020-02-28 09:55:40 +01:00
Simone Gotti 19b8c7f427 executor: fix reporting of stopped tasks and steps
In executeTask set the executor task and step phase to stop if task spec Stop is
true.
2020-02-27 17:38:00 +01:00
Simone Gotti 88dbca15a3 executor: serialize task handling
taskUpdater will be called serially and won't block. It'll execute a goroutine
for executing the task and for sending the task state to the scheduler.

executeTask will just start task execution, all the logic of choosing if
starting a task is moved inside taskUpdater

In this way we avoid concurrency issues when handling the same executorTask
in parallel
2020-02-27 17:19:42 +01:00
Simone Gotti 145c87b4c0 runservice: minimize scheduling of tasks that will be queued by the executor
Since the executor only periodically updates its state we could end up
scheduling much more tasks than the executor ActiveTasksLimit. This will happen
in the case of many parallel tasks that can all start at the same time.

To avoid this also considere the executor tasks saved in etcd that represent
the real view of scheduled tasks.
2020-02-27 11:03:03 +01:00
Simone Gotti 5dd9e587fe runservice: mark not running tasks as skipped when run marked to stop
Currently when a run is marked to stop we are going to stop currently running
tasks and then their childs will be marked as skipped.

But tasks not depending on a stopped task (root task or childs with a finished
parent) that are just waiting for an executor slot, will be scheduled when
there will be a free slot also if the run is marked to stop (and then the
scheduler will stop them after some seconds).

This patch will mark all not started tasks as skipped when the run is marked to
stop.
2020-02-26 16:45:09 +01:00
Simone Gotti eb48e73a54 gateway: move authentication apis to /api/v1alpha/auth
Move the various authentication apis to /api/v1alpha/auth since they should be
versioned like other apis.
2020-02-19 10:48:14 +01:00
Carlo Mandelli 182eb14b20 cmd: project option to disable passing variables to PR from forked repo 2020-01-28 09:02:37 +01:00
Carlo Mandelli d049782e29 tests: add unique name for logs of the third ConfigStore instance 2020-01-16 13:54:13 +01:00
Simone Gotti 2de91549a3 tests: improve services logging
During tests provide a zaptest Logger so all services output will be redirected
to golang testing logger.

When multiple services of the same type are provided add a unique name field to
distinguish them.
2020-01-15 12:30:34 +01:00
Simone Gotti ecf355721f docker: create a toolbox volume for every pod
Instead of doing the current hack of copying the agola toolbox inside the host
tmp dir (always done but only needed when running the executor inside a docker
container) that has different issues (like tmp file removal done by
tmpwatch/systemd-tmpfiles), use a solution similar to the k8s driver: for every
pod create a volume containing the agola-toolbox and remove it at pod removal.

We could also use a single "global" volume but we should handle cases like
volume removal (i.e. a docker volume prune command). So for now just create a
dedicated per pod volume.
2020-01-10 12:25:12 +01:00
Carlo Mandelli 3e47bc601a gateway/runservice: add api to delete step logs 2019-11-18 10:34:56 +01:00
Simone Gotti 7e8f7155d7 runservice: improve errors in logsHandler
return errNotExist in readTaskLogs when the run,task or step doesn't exist.
2019-11-15 15:50:58 +01:00
Simone Gotti f7d0950ca1 *: write and flush header on log handlers
Explicitly write and flush the headers in the various services LogHandlers.

Currently the 200 response and the other headers will be automatically written
by the golang http implementation only when we send something in the body. But if
there's nothing to send (no logs yet written) the client will never receive the
headers and cannot know if the request was successful.
2019-11-14 10:52:45 +01:00
Simone Gotti 32a08ec5c8
Merge pull request #179 from sgotti/runservice_logshandler_improve_errors
runservice: improve errors in logsHandler
2019-11-14 09:56:24 +01:00
Simone Gotti 66e182a55d runservice: improve errors in logsHandler
* return errNotExist in readTaskLogs when the executor task doesn't exist: so
the client will receive a 404 instead of a 500 (since a generic error will be
mapped to a 500).
* Wrap the errNotExist returned by readTaskLogs with a new ErrNotExits reporting
"log doesn't exist"
2019-11-13 15:50:20 +01:00
Simone Gotti 07cde065c8 runservice: use etcd mutex TryLock on fetching
When fetching avoid concurrent fetches from multiple runservices using an etcd
mutex TryLock.
2019-11-13 11:53:54 +01:00
Simone Gotti 9fd4b662a8
Merge pull request #175 from camandel/api_logarchived
gateway: add api to get log status
2019-11-13 11:45:31 +01:00
Carlo Mandelli 8ed813e722 gateway: add api to get log status 2019-11-13 10:01:51 +01:00
Simone Gotti 5ab9f7c970 *: use etcd mutex TryLock
etcd PR 11104 (https://github.com/etcd-io/etcd/pull/11104) implemented mutex
TryLock. Since it's only available in etcd master just copy relevant code and
add a TODO to remove it when updating the etcd client to a version implementing
TryLock.

Use TryLock everywhere where it'll be useful.
2019-11-12 22:27:17 +01:00
Simone Gotti d679254516 readdb: improve HandleEvents goroutine exiting
Rename errCh to doneCh (error is not needed) and always send to it when one of
the HandleEvents functions exits (not only on error).

This will ensure that all the goroutines will be stopped also if one of them
returns without an error.
2019-11-12 11:03:21 +01:00
Simone Gotti 72f279c4c3 *: improve error handling
* objectstorage: remove `types` package and move `ErrNotExist` in base package
* objectstorage: Implement .Is and add helper `IsErrNotExist` for `ErrNotExist`
* util: Rename `ErrNotFound` to `ErrNotExist`
* util: Add `IsErr*` helpers and use them in place of `errors.Is()`
* datamanager: add `ErrNoDataStatus` to report when there's not data status in ost
* runservice/common: remove `ErrNotExist` and use errors in util package
2019-11-11 12:17:35 +01:00
Simone Gotti 5af07d0852 objectstorage: use a single package
remove all the subpackages and just use a single package
2019-11-08 16:31:48 +01:00
Simone Gotti 9c0eb3d7ef datamanager: refactor ReadWal
make ReadWal directly return a *WalHeader
2019-11-08 13:24:43 +01:00
Carlo Mandelli aab2321d58 services: check config only for enabled services 2019-11-05 09:25:22 +01:00
Simone Gotti e18794764e go.mod: update dependencies
Update all the updatable dependencies
2019-10-29 09:31:38 +01:00
Carlo Mandelli 7a51404b71 run config: add tty option for run steps 2019-10-28 16:58:54 +01:00
Simone Gotti 446e626f9f gateway: fix project create run http method
Make it a POST instead of a PUT.
2019-10-22 09:50:01 +02:00
Carlo Mandelli 6fccb935c4 docker: mount multiple volumes 2019-10-17 09:21:46 +02:00
Simone Gotti fa4b41ab74 when: match only the current ref type
Only match the current ref type, ie: don't match a branch when the ref type is a
tag or pull request.

Ref is always matched because it's not related to a specific ref type.
2019-10-14 17:08:44 +02:00
Simone Gotti 7d62481415 *: implement ability to add tmpfs volumes to containers
* Add a generic container volume option that currently only support tmpfs. In
future it could be expanded to use of host volumes or other kind of volumes (if
supported by the underlying executor)

* Implement creation of tmpfs volumes in docker and k8s drivers.
2019-10-08 16:36:23 +02:00
Carlo Mandelli 4fc8f3ebed cmd: add version command 2019-10-05 12:11:18 +02:00
Simone Gotti 11da186913 gateway: add parentRef field to project group update api
and make all the request fields optional
2019-10-01 14:39:45 +02:00
Simone Gotti 973cfe8770 configstore: implement project group move 2019-10-01 12:03:57 +02:00
Simone Gotti e2a0fedfb8 configstore: disable root project group deletion 2019-10-01 10:26:17 +02:00
Simone Gotti 8b9464486d gateway: add parentRef field to project update api
and make all the request fields optional
2019-09-27 17:10:04 +02:00
Simone Gotti fe9e3e8317 configstore: implement project move 2019-09-27 11:07:49 +02:00
Simone Gotti f5c0e91f39 gateway: api implement get secrets removeoverridden
Implement missing removeoverridden option for get secrets.
2019-09-20 15:39:21 +02:00
Carlo Mandelli 436aa8f1de skip run with special commit 2019-09-18 11:53:22 +02:00