Commit Graph

517 Commits

Author SHA1 Message Date
Simone Gotti c1da3ab566 *: Improve error handling
* Create an APIError that should only be used for api returned errors.
  It'll wrap an error and can have different Kinds and optional code and
  message.
* The http handlers will use the first APIError available in the
  error chain and generate a json response body containing the code and
  the user message. The wrapped error is internal and is not sent in the
  response.
  If no api error is available in the chain a generic internal
  server error will be returned.
* Add a RemoteError type that will be created from remote services calls
  (runservice, configstore). It's similar to the APIError but a
  different type to not propagate to the caller response and it'll not
  contain any wrapped error.
* Gateway: when we call a remote service, by default, we'll create a
  APIError using the RemoteError Kind (omitting the code and the
  message that usually must not be propagated).
  This is done for all the remote service calls as a starting point, in
  future, if this default behavior is not the right one for a specific
  remote service call, a new api error with a different kind and/or
  augmented with the calling service error codes and user messages could
  be created.
* datamanager: Use a dedicated ErrNotExist (and converting objectstorage
  ErrNotExist).
2022-02-25 16:11:19 +01:00
Simone Gotti 87f182a0c9 *: use errors.Is/errors.As to handle wrapped error checking
Enable golangci-lint errorlint linter to check proper use of errors.Is
and error.As instead of direct comparison or error type casting.
2022-02-24 17:07:29 +01:00
Simone Gotti bed726f2d2 ci: update golangci-lint to v1.44.2
Fix errors reported by new/updated linters
2022-02-24 15:22:39 +01:00
Simone Gotti b7c797bde1 gateway: use dedicated auth context keys type
Use auth context keys dedicated type instead of strings and avoid code
duplication by moving shared code under a common package.
2022-02-24 13:41:57 +01:00
Tullio 7160194104 gateway: add api to get authenticated user organizations
gateway: add api to get authenticated user organizations

gateway: add api to get authenticated user organizations
2022-02-23 15:31:34 +01:00
Simone Gotti 576c09775f go.mod: replace https://github.com/satori/go.uuid
Replace https://github.com/satori/go.uuid with maintained version at
https://github.com/gofrs/uuid

Since the new version uuid.NewV4 returns an error when failing to read
from the random source reader we use uuid.Must to panic on error since
it's considered an unrecoverable error.

In future, if needed, we could handle the error instead of panicking.
2022-02-21 09:43:32 +01:00
alessandro.pinna aa65b93342 ci: update gitea to v1.15.11 2022-02-08 15:07:42 +01:00
alessandro.pinna 9251a2a046 enhancement: automatic cleanup of old repos/branches 2022-02-08 11:34:23 +01:00
alessandro.pinna e517e4d243 runconfig: test task generation with on_skipped dependencies 2022-02-01 10:01:54 +01:00
raeyulca 88bef4480d add missing case statement 2022-01-18 23:14:50 -08:00
alessandro.pinna e600ca078e config: fix executor ActiveTasksLimit yaml tag 2021-10-11 09:48:25 +02:00
Simone Gotti 64680468fd *: migrate to golang-jwt/jwt
Migrate to the new community maintained version of github.com/dgrijalva/jwt-go
2021-09-10 12:33:59 +02:00
Yves-Gaël Billet 3dbf9a923f fix: gitea fetch remote repository with pagination
Actual implementation doesn't fetch all repository. Gitea API has
pagination.

This patch works wwith gitea pagination for repository listing

Signed-off-by: Yves-Gaël BILLET <yg.billet@gmail.com>
2021-06-04 23:17:41 +02:00
Simone Gotti d9d6b995da executor: add init image config authentication options
Add config options to define docker registry authentication for init image
pulling.
2021-05-26 08:57:21 +02:00
Simone Gotti afece4d846 executor: use busybox:stable as default init image
Use "busybox:stable" as default init image name so it won't be checked/pulled
any time.
2021-05-25 14:03:55 +02:00
Simone Gotti b403fd558a executor/docker: pull init image only when needed
Use the same logic of k8s and pull init image only when has the latest (or
empty) tag or it doesn't exist.
2021-05-25 12:59:16 +02:00
Simone Gotti 21a63af6f5 executor: add config option to define the init image
* Add an executor config option to define the init image to use.
2021-05-25 11:46:44 +02:00
Simone Gotti 0544586ade *: call ListenAndServeTLS when tls is enabled in config 2021-03-19 10:53:16 +01:00
Simone Gotti f598fd0f4b gitserver: fix fetchfile error handling
since the git command output is redirected to the http response body, the golang
http server automatically adds a 200 status before sending the body. If the git
command fails we cannot return an http error anymore but must close the
connection to let the client know that the request failed.
2021-03-15 13:19:13 +01:00
Carlo Mandelli 45eb092871 Remove WaitingApproval for stopped tasks 2021-02-03 19:45:27 +01:00
Finn-Thorben Sell 8cb8973846 add content-type header for /config.js 2020-10-30 19:41:52 +01:00
6543 c6f09610f1
migrate 2020-05-22 12:07:53 +02:00
Simone Gotti 8a156b936e config: check max config size
limit config size to 1MiB.
2020-03-17 11:28:25 +01:00
Simone Gotti d91ec09d7d config: add starlark config support
Handle `.agola/config.star` files in starlark config format.
To provide a context like done for jsonnet we require that the starlark agola
config file contains a main function that will receive a config context as a
dict.
We also had to implement our own json conversion from a starlark dict since go
starlark removed its own function.
2020-03-10 13:29:20 +01:00
Simone Gotti 7094de908b run: export also ref type environment variable to runs 2020-03-09 10:47:58 +01:00
Simone Gotti 627f40987e run: export also pull request id environment variable to runs 2020-03-09 09:42:51 +01:00
Simone Gotti 1820c0247c config: provide jsonnet context top level argument 2020-03-06 11:19:49 +01:00
Simone Gotti e20abf053c runconfig: disable password authentication in clone step
If for some reasons the ssh public key auth fails, avoid the clone step to block
during a git clone waiting for a password.
2020-03-05 16:11:20 +01:00
Simone Gotti 0e2b01a586 runservice: correctly handle skipped tasks in fetcher
skip fetching of tasks with status skipped, not only tasks marked as skip.
This avoid many wrong an noisy logs of type "executor task with id taskid
doesn't exist. This shouldn't happen. Skipping fetching"
2020-03-02 10:40:59 +01:00
Simone Gotti eb180da914
Merge pull request #225 from sgotti/runservice_fix_handling_of_wrong_executortask_status
runservice: fix handling of wrong executortask status
2020-03-02 10:26:32 +01:00
Simone Gotti 4da7c23bb8
Merge pull request #224 from sgotti/executor_use_cancellable_context_in_executetask
executor: use cancellable context in executetask
2020-03-02 10:26:16 +01:00
Simone Gotti 382705bde9
Merge pull request #223 from sgotti/executor_fix_stopping_of_not_running_task
executor: fix stopping of not running tasks
2020-03-02 09:45:39 +01:00
Simone Gotti 19611c18e7 runservice: fix handling of wrong executortask status
updateRunTaskStatus should also accept transitions from not started to a
finished state like "success", "failed", "stopped" since we could miss some
status updates from the executor for many reasons.
2020-02-28 13:02:35 +01:00
Simone Gotti e4507446ed executor: use cancellable context in executetask
Use a cancellable context to handle running task stop.
When the context is done the pod will be stopped.
2020-02-28 10:52:36 +01:00
Simone Gotti 97d145a9d3 executor: fix stopping of not running tasks
When a related runningTask doesn't exist and the executor task status is
running just report it as failed ignoring if it's marked to stop.
2020-02-28 10:45:07 +01:00
Simone Gotti 3ac018e6e5 runservice: use all scheduled tasks in scheduleRun
rename activeExecutorTasks to scheduledExecutorTasks and don't filter out
finished tasks.
In some logic we need all the scheduled tasks and not only the not finished
ones.
2020-02-28 09:56:12 +01:00
Simone Gotti a4e280cd7d
Merge pull request #222 from sgotti/executor_fix_reporting_of_stopped_tasks_and_steps
executor: fix reporting of stopped tasks and steps
2020-02-28 09:55:40 +01:00
Simone Gotti 19b8c7f427 executor: fix reporting of stopped tasks and steps
In executeTask set the executor task and step phase to stop if task spec Stop is
true.
2020-02-27 17:38:00 +01:00
Simone Gotti 88dbca15a3 executor: serialize task handling
taskUpdater will be called serially and won't block. It'll execute a goroutine
for executing the task and for sending the task state to the scheduler.

executeTask will just start task execution, all the logic of choosing if
starting a task is moved inside taskUpdater

In this way we avoid concurrency issues when handling the same executorTask
in parallel
2020-02-27 17:19:42 +01:00
Simone Gotti 145c87b4c0 runservice: minimize scheduling of tasks that will be queued by the executor
Since the executor only periodically updates its state we could end up
scheduling much more tasks than the executor ActiveTasksLimit. This will happen
in the case of many parallel tasks that can all start at the same time.

To avoid this also considere the executor tasks saved in etcd that represent
the real view of scheduled tasks.
2020-02-27 11:03:03 +01:00
Simone Gotti 5dd9e587fe runservice: mark not running tasks as skipped when run marked to stop
Currently when a run is marked to stop we are going to stop currently running
tasks and then their childs will be marked as skipped.

But tasks not depending on a stopped task (root task or childs with a finished
parent) that are just waiting for an executor slot, will be scheduled when
there will be a free slot also if the run is marked to stop (and then the
scheduler will stop them after some seconds).

This patch will mark all not started tasks as skipped when the run is marked to
stop.
2020-02-26 16:45:09 +01:00
Simone Gotti eb48e73a54 gateway: move authentication apis to /api/v1alpha/auth
Move the various authentication apis to /api/v1alpha/auth since they should be
versioned like other apis.
2020-02-19 10:48:14 +01:00
Simone Gotti ed53183431 go.mod: update dependencies
Update all the updatable dependencies
2020-02-18 13:55:50 +01:00
Simone Gotti dad7447989 gitsources: handle skipverify also in oauth2 requests
Pass a custom http client set to skip tls verification if required to oauth2
calls.
2020-02-11 21:49:32 +01:00
Simone Gotti 0611b5cc67 github: handle nil user email 2020-02-11 15:59:30 +01:00
Simone Gotti 59463944db github: use the provided api url
we were always setting the public github url in the github client instead of the
provided api url.
2020-02-11 09:10:09 +01:00
Carlo Mandelli 182eb14b20 cmd: project option to disable passing variables to PR from forked repo 2020-01-28 09:02:37 +01:00
Carlo Mandelli d049782e29 tests: add unique name for logs of the third ConfigStore instance 2020-01-16 13:54:13 +01:00
Simone Gotti 2de91549a3 tests: improve services logging
During tests provide a zaptest Logger so all services output will be redirected
to golang testing logger.

When multiple services of the same type are provided add a unique name field to
distinguish them.
2020-01-15 12:30:34 +01:00
Simone Gotti ecf355721f docker: create a toolbox volume for every pod
Instead of doing the current hack of copying the agola toolbox inside the host
tmp dir (always done but only needed when running the executor inside a docker
container) that has different issues (like tmp file removal done by
tmpwatch/systemd-tmpfiles), use a solution similar to the k8s driver: for every
pod create a volume containing the agola-toolbox and remove it at pod removal.

We could also use a single "global" volume but we should handle cases like
volume removal (i.e. a docker volume prune command). So for now just create a
dedicated per pod volume.
2020-01-10 12:25:12 +01:00