Commit Graph

37 Commits

Author SHA1 Message Date
Simone Gotti 4da7c23bb8
Merge pull request #224 from sgotti/executor_use_cancellable_context_in_executetask
executor: use cancellable context in executetask
2020-03-02 10:26:16 +01:00
Simone Gotti e4507446ed executor: use cancellable context in executetask
Use a cancellable context to handle running task stop.
When the context is done the pod will be stopped.
2020-02-28 10:52:36 +01:00
Simone Gotti 97d145a9d3 executor: fix stopping of not running tasks
When a related runningTask doesn't exist and the executor task status is
running just report it as failed ignoring if it's marked to stop.
2020-02-28 10:45:07 +01:00
Simone Gotti a4e280cd7d
Merge pull request #222 from sgotti/executor_fix_reporting_of_stopped_tasks_and_steps
executor: fix reporting of stopped tasks and steps
2020-02-28 09:55:40 +01:00
Simone Gotti 19b8c7f427 executor: fix reporting of stopped tasks and steps
In executeTask set the executor task and step phase to stop if task spec Stop is
true.
2020-02-27 17:38:00 +01:00
Simone Gotti 88dbca15a3 executor: serialize task handling
taskUpdater will be called serially and won't block. It'll execute a goroutine
for executing the task and for sending the task state to the scheduler.

executeTask will just start task execution, all the logic of choosing if
starting a task is moved inside taskUpdater

In this way we avoid concurrency issues when handling the same executorTask
in parallel
2020-02-27 17:19:42 +01:00
Simone Gotti 2de91549a3 tests: improve services logging
During tests provide a zaptest Logger so all services output will be redirected
to golang testing logger.

When multiple services of the same type are provided add a unique name field to
distinguish them.
2020-01-15 12:30:34 +01:00
Simone Gotti ecf355721f docker: create a toolbox volume for every pod
Instead of doing the current hack of copying the agola toolbox inside the host
tmp dir (always done but only needed when running the executor inside a docker
container) that has different issues (like tmp file removal done by
tmpwatch/systemd-tmpfiles), use a solution similar to the k8s driver: for every
pod create a volume containing the agola-toolbox and remove it at pod removal.

We could also use a single "global" volume but we should handle cases like
volume removal (i.e. a docker volume prune command). So for now just create a
dedicated per pod volume.
2020-01-10 12:25:12 +01:00
Simone Gotti e18794764e go.mod: update dependencies
Update all the updatable dependencies
2019-10-29 09:31:38 +01:00
Carlo Mandelli 7a51404b71 run config: add tty option for run steps 2019-10-28 16:58:54 +01:00
Simone Gotti 7d62481415 *: implement ability to add tmpfs volumes to containers
* Add a generic container volume option that currently only support tmpfs. In
future it could be expanded to use of host volumes or other kind of volumes (if
supported by the underlying executor)

* Implement creation of tmpfs volumes in docker and k8s drivers.
2019-10-08 16:36:23 +02:00
Simone Gotti 714da3ffe3 executor: add missing mutex unlock
Add missing mutex unlock that will cause deadlocks.
2019-09-18 09:49:38 +02:00
Simone Gotti 39829f1ec4 runservice: save step exitstatus in run.
For every step save also the command exit status.
2019-09-17 14:35:37 +02:00
Simone Gotti 12b02143b2 runservice: don't save executor task data in etcd
Reorganize ExecutorTask to better distinguish between the task Spec and
the Status.

Split the task Spec in a sub part called ExecutorTaskSpecData that contains
tasks data that don't have to be saved in etcd because it contains data that can
be very big and can be generated starting from the run and the runconfig.
2019-09-17 12:03:43 +02:00
Simone Gotti 6ee76274d7 executor: set the container exec user in every step 2019-09-12 10:55:07 +02:00
Simone Gotti 51e9a32db7 runconfig: set task default shell
Currently, if no shell is defined in the task and in the step, the executor will
use an hardcoded default shell.

This will cause changed run behavior if we add an option to globally set the
agola default shell.

To avoid this set the task shell to the default shell inside the runconfig if
it's empty so future executions will always use this value.
2019-09-11 18:51:18 +02:00
Simone Gotti 6d7410cfce runservice: remove run step user
Defining an option to override the user for a run step is too much fine grained
and, for consistency, will require to do the same also for the other steps
(clone, *workspace etc...).

Remove it since it's probably enough to define it at the task level.
2019-09-11 15:02:08 +02:00
Simone Gotti e31b0b47ef executor: listen on wildcard address
Since the current logic is to use the first available private ip address as the
advertized address we have to listen on wildcard since a different host provided
in web.ListenAddress will make the executor unreachable.

In future improve this to let the user to manually define the bind and the
advertized address (perhaps using go-sockaddr templates like done by consul) to
also support nat between the schedulers and the executors.
2019-08-06 13:42:42 +02:00
Simone Gotti c1ff28ef9f *: export clients and related types
Export clients and related packages.

The main rule is to not import internal packages from exported packages.

The gateway client and related types are totally decoupled from the gateway
service (not shared types between the client and the server).

Instead the configstore and the runservice client currently share many types
that are now exported (decoupling them will require that a lot of types must be
duplicated and the need of functions to convert between them, this will be done
in future when the APIs will be declared as stable).
2019-08-02 12:02:01 +02:00
Simone Gotti d0c5621201 util: remove time.go
The same function is already provided by pointer.go
2019-08-01 14:14:56 +02:00
Simone Gotti e48a28d5b9
Merge pull request #71 from sgotti/executor_set_task_endtime_when_marking_task_failed
executor: set task endTime when marking as failed
2019-07-29 14:43:13 +02:00
Simone Gotti 0ecfc24def executor: fix typo in setting task endTime when setup failed
There was a typo so we weren't setting the task endTime when the setup step
failed.

Also unify all logic to just use `et` (instead of a mix of `et` or `rt.et`)
2019-07-29 10:00:32 +02:00
Simone Gotti 1707be9528 executor: set task endTime when marking as failed
Add missing set of task endTime when the executor is marking the task as failed
due to no related running task (usually after executor restart).
2019-07-29 09:58:17 +02:00
Simone Gotti 6f3798e8fe *: use sleep timer in loops
So we'll react instantly to a context cancel instead of waiting on time.Sleep
returning.
2019-07-25 16:22:54 +02:00
Simone Gotti 929a6fb654 service/*: log error only if nil 2019-07-04 15:50:37 +02:00
Simone Gotti f152b6a6da executor: fix linter errors
Fix errors reported by default golangci-lint linters
2019-07-02 15:18:52 +02:00
Simone Gotti 8d67844cc4 *: use vanity url
use agola.io domain
2019-07-01 11:40:20 +02:00
Simone Gotti 0296d594b5 executor: add option to allow privileged containers
* add a config option allowPrivilegedContainers
* fail task setup if privileged containers are requested but they aren't
allowed.
* report if privileged containers are allowed to the runservice
2019-06-13 18:31:08 +02:00
Simone Gotti 57446f7dcd executor: fix task status update when runservice is unavailable
* don't remove the runningTask when executeTask finishes but just mark the
runningTask a not executing
* add a loop to periodically update executorTask status and remove the
runningTask if not executing and status update was successful
* remove runningTask when it disappears from the runservice
2019-06-13 12:39:34 +02:00
Simone Gotti d6feb10e8f executor: rename/refactor executeTasksInternal to executeTaskSteps
* directly pass runningTask instead of executorTask
2019-06-13 12:34:31 +02:00
Simone Gotti 629cf2be82 executor: reuse taskPath function 2019-06-12 16:16:53 +02:00
Simone Gotti 9b2ce717c7 *: migrate to "golang.org/x/xerrors"
Just a raw replace of "github.com/pkg/errors".

Next steps will improve errors (like remote errors, api errors, not exist errors
etc...) to leverage its functionalities
2019-05-23 11:23:14 +02:00
Simone Gotti 821e371cd8 executor: don't start task if max tasks limit has been reached 2019-05-15 15:06:46 +02:00
Simone Gotti caa9d74b72 runservice: implement additional service containers
This was already defined in the config but not implemented in the executor and
drivers.

All the containers defined in the runtime after the first one will be "service"
containers. They will share the same network namespace with the other containers
in the "pod" so they can communicate between themself on loopback
2019-05-10 17:31:23 +02:00
Simone Gotti 7adca3ea30 runservice drivers: define ExecConfig AttachStdin 2019-05-10 16:50:31 +02:00
Simone Gotti 4e785e4851 runservice: build and use multiple toolboxes per architecture 2019-05-09 12:36:30 +02:00
Simone Gotti 1e34dca95d runservice: split and simplify scheduler and executor naming
Also if they are logically part of the runservice the names runserviceExecutor
and runserviceScheduler are long and quite confusing for an external user

Simplify them separating both the code parts and updating the names:

runserviceScheduler -> runservice
runserviceExecutor -> executor
2019-05-07 23:56:10 +02:00