Commit Graph

475 Commits

Author SHA1 Message Date
Simone Gotti
2676770336 userdirectrun: add options to define variables
Add a --var and --var-file options (repeatable multiple times) to define the
variables to be used in the run.
2019-08-06 16:58:00 +02:00
Simone Gotti
252bd95a58
Merge pull request #84 from sgotti/executor_listen_wildcard
executor: listen on wildcard address
2019-08-06 15:18:46 +02:00
Simone Gotti
e31b0b47ef executor: listen on wildcard address
Since the current logic is to use the first available private ip address as the
advertized address we have to listen on wildcard since a different host provided
in web.ListenAddress will make the executor unreachable.

In future improve this to let the user to manually define the bind and the
advertized address (perhaps using go-sockaddr templates like done by consul) to
also support nat between the schedulers and the executors.
2019-08-06 13:42:42 +02:00
Simone Gotti
7623202048
Merge pull request #78 from sgotti/config_run_when
config: add run when field
2019-08-06 11:25:22 +02:00
Simone Gotti
db742a6cd6 config: add run when field
Don't create a run if a when condition is defined and it doesn't match.
2019-08-05 16:07:47 +02:00
Simone Gotti
4fe4631729
Merge pull request #83 from sgotti/user_direct_run_custom_branch_tag_ref
userdirectrun: allow setting destination branch/tag/ref
2019-08-05 16:06:44 +02:00
Simone Gotti
4ec0b33eb4 userdirectrun: allow setting destination branch/tag/ref
Allow setting the destination branch/tag/ref so users can test the run
conditions based on the branch/tag/ref.

To simulate a pull request an user can define a ref that matches one of these
regular expressions: `refs/pull/(\d+)/head`, `refs/merge-requests/(\d+)/head`
2019-08-05 14:45:34 +02:00
Simone Gotti
0da52e2d8a
Merge pull request #82 from sgotti/tests_test_also_clone_step
tests: test also clone step
2019-08-05 14:33:19 +02:00
Simone Gotti
c17772040b tests: test also clone step
Also test clone step so we are sure that the clone url is correct.
2019-08-05 13:33:58 +02:00
Simone Gotti
1c96b5fbff
Merge pull request #81 from sgotti/docker_driver_use_fixed_client_api_version
docker driver: use fixed client api version
2019-08-04 23:56:17 +02:00
Simone Gotti
f082ee52db
Merge pull request #80 from sgotti/docker_driver_use_toolbox_exec
docker driver: use toolbox exec
2019-08-04 23:55:57 +02:00
Simone Gotti
df66cfc736 docker driver: use fixed client api version
Set the client required api version to 1.26. In this way we'll work with docker
>= 1.13.1
2019-08-04 23:38:20 +02:00
Simone Gotti
b3672bf927 docker driver: use toolbox exec
Older version of docker doesn't support the exec api Env and WorkingDir options.

Support these versions by doing the same we already do with the k8s driver: use
the `toolbox exec` command that will set the provided Env, change the cwd to the
WorkingDir and the exec the wanted command.
2019-08-04 18:09:34 +02:00
Simone Gotti
b333b6f423
Merge pull request #77 from sgotti/export_api_client_types
*: export clients and related types
2019-08-02 14:41:50 +02:00
Simone Gotti
c1ff28ef9f *: export clients and related types
Export clients and related packages.

The main rule is to not import internal packages from exported packages.

The gateway client and related types are totally decoupled from the gateway
service (not shared types between the client and the server).

Instead the configstore and the runservice client currently share many types
that are now exported (decoupling them will require that a lot of types must be
duplicated and the need of functions to convert between them, this will be done
in future when the APIs will be declared as stable).
2019-08-02 12:02:01 +02:00
Simone Gotti
a52b52865c
Merge pull request #76 from sgotti/configstore_internal_types
configstore: move configstore types inside configstore package
2019-08-02 10:58:26 +02:00
Simone Gotti
9371e407ee
Merge pull request #75 from sgotti/remove_util_time
util: remove time.go
2019-08-02 10:42:37 +02:00
Simone Gotti
fd26e617b3 configstore: move configstore types inside configstore package
Since they're not types common to all the services but belongs to the
configstore.

Next step will be to make them local to the configstore and not directly used by
other services since these types are also stored.
2019-08-02 10:05:47 +02:00
Simone Gotti
d0c5621201 util: remove time.go
The same function is already provided by pointer.go
2019-08-01 14:14:56 +02:00
Simone Gotti
e48a28d5b9
Merge pull request #71 from sgotti/executor_set_task_endtime_when_marking_task_failed
executor: set task endTime when marking as failed
2019-07-29 14:43:13 +02:00
Simone Gotti
43e445f3aa
Merge pull request #70 from sgotti/executor_fix_task_endtime_setup_step_failed
executor: fix typo in setting task endTime when setup failed
2019-07-29 14:43:02 +02:00
Simone Gotti
fe90eecf62
Merge pull request #69 from sgotti/runservice_fix_executor_delete
runservice: fix/improve executor delete logic
2019-07-29 14:42:41 +02:00
Simone Gotti
b81ad4cd8c runservice: fix/improve executor delete logic
* Don't fail tasks inside the delete executor action, just delete the executor
from etcd

* The scheduler, when detecting a task without a related executor will mark the
task as failed and correctly set end time of the task and its steps.
2019-07-29 12:06:15 +02:00
Simone Gotti
504cd3d6dc
Merge pull request #68 from sgotti/runservice_maintenance_export_import
runservice: maintenance/export/import
2019-07-29 12:05:38 +02:00
Simone Gotti
f812597410 runservice: maintenance/export/import
Implement runservice maintenance mode and export/import.

When runservice is set in maintenance mode it'll start only the maintenance and
export/import handlers.

Setting maintenance mode will set a key in etcd so all the runservice instances
will detect it and enter in maintenance mode. This is done asyncronously so it
could take some time (future improvements will add some api to show all the
runservice states)

Export is always available and will export the datamanager contents. Currently
only datamanager contents are exported (no logs and workspace archives).

Import is available only during maintenance, given a datamanager export will
import it and reset etcd to this import state.
2019-07-29 11:52:30 +02:00
Simone Gotti
1034b3afdb
Merge pull request #67 from sgotti/configstore_maintenance_export_import
configstore: maintenance/export/import
2019-07-29 11:11:02 +02:00
Simone Gotti
0ecfc24def executor: fix typo in setting task endTime when setup failed
There was a typo so we weren't setting the task endTime when the setup step
failed.

Also unify all logic to just use `et` (instead of a mix of `et` or `rt.et`)
2019-07-29 10:00:32 +02:00
Simone Gotti
1707be9528 executor: set task endTime when marking as failed
Add missing set of task endTime when the executor is marking the task as failed
due to no related running task (usually after executor restart).
2019-07-29 09:58:17 +02:00
Simone Gotti
fafa5188c2 configstore: maintenance/export/import
Implement configstore maintenance mode and export/import.

When configstore is set in maintenance mode it'll start only the maintenance and
export/import handlers.

Setting maintenance mode will set a key in etcd so all the configstore instances
will detect it and enter in maintenance mode. This is done asyncronously so it
could take some time (future improvements will add some api to show all the
configstore states)

Export is always available and will export the datamanager contents.

Import is available only during maintenance, given a datamanager export will
import it and reset etcd to this import state.
2019-07-26 10:55:04 +02:00
Simone Gotti
f3fa229f6c util: add GoWait function
GoWait will increase the provided waitGroup on start and execute a goroutine
that will run the provided functions and then decrease the waitGroup
2019-07-26 10:55:04 +02:00
Simone Gotti
e2526a6399
Merge pull request #66 from sgotti/util_backoff_use_context
util: use context in backoff
2019-07-26 10:54:54 +02:00
Simone Gotti
bd035e9840 util: use context in backoff 2019-07-26 10:36:11 +02:00
Simone Gotti
335d1a2aad
Merge pull request #65 from sgotti/readdb_close_open_on_run
readdb: close and open readdb on Run
2019-07-26 10:18:29 +02:00
Simone Gotti
ceafc2ef98 readdb: close and open readdb on Run 2019-07-25 17:59:54 +02:00
Simone Gotti
d18c878d4d
Merge pull request #64 from sgotti/use_sleep_timers_loops
*: use sleep timer in loops
2019-07-25 17:56:46 +02:00
Simone Gotti
6f3798e8fe *: use sleep timer in loops
So we'll react instantly to a context cancel instead of waiting on time.Sleep
returning.
2019-07-25 16:22:54 +02:00
Simone Gotti
9481b8d67d
Merge pull request #62 from sgotti/db_use_context
db: use context functions
2019-07-25 16:22:10 +02:00
Simone Gotti
b8c2b4020a db: use context functions
Use the go sql context functions (ExecContext, QueryContext etc...)

The context is saved inside Tx so the library users should only pass it one time
to the db.Do function.
2019-07-25 14:49:53 +02:00
Simone Gotti
5c87d89c72
Merge pull request #63 from sgotti/datamanager_export_import
datamanager: implement import/export
2019-07-25 14:49:16 +02:00
Simone Gotti
3404cb94b9 datamanager: implement import/export
* export: exports the newest data checkpoint. It forces a checkpoint before
exporting (currently no wals are exported)

* import: cleans up etcd, creates a new datasnaphot from the provided import stream
and then initializes etcd. Currently no old data is removed from the object
storage but it's just ignored.
2019-07-25 11:12:49 +02:00
Simone Gotti
af4aa58903
Merge pull request #61 from sgotti/datamanager_maintenance_mode
datamanager: implement maintenance mode
2019-07-25 10:58:40 +02:00
Simone Gotti
3987caf664 datamanager: implement maintenance mode
when datamanager is started in maintenance mode no goroutines are scheduled and
no initial etcd initialization is done
2019-07-24 17:37:27 +02:00
Simone Gotti
e0a6e78cf3
Merge pull request #60 from sgotti/go_mod_update_xerrors
go mod: update golang.org/x/xerrors
2019-07-24 17:00:34 +02:00
Simone Gotti
5a16b72adb
Merge pull request #59 from sgotti/db_retry_on_sqlite_locked_err
db: retry on sqlite locked error
2019-07-24 17:00:14 +02:00
Simone Gotti
2800f6cd6e go mod: update golang.org/x/xerrors 2019-07-24 15:23:33 +02:00
Simone Gotti
3297244d5d db: retry on sqlite locked error
Since we are using the shared cache with the lock notify we won't receive
SQLITE_BUSY errors but we could receive SQLITE_LOCKED errors due to deadlocks or
locked tables on concurrent read and write transactions.

This patch catches this kind of errors and retries the tx until maxTxRetries.
2019-07-24 12:20:33 +02:00
Simone Gotti
77ee8d9e7d
Merge pull request #58 from sgotti/readdb_fix_deadlock
readdb: fix deadlock in Run method
2019-07-23 15:47:05 +02:00
Simone Gotti
19706fe485
Merge pull request #57 from sgotti/runservice_stop_also_if_result_is_not_set
runservice: stop run also if result is not set
2019-07-23 15:09:58 +02:00
Simone Gotti
85876310af readdb: fix deadlock in Run method
In runservice readdb Run method we could end with a deadlock if two of the
goroutines that call HandleEvents.* try to write to the errCh at the same
time before the errCh is read. If this happens one of the two will be blocked on
writing to the channel but the read won't happen since it'll blocked by
wg.Wait().

Fix this doing:
* use a buffered channel large as the number of executed goroutines.
* create a new errCh at every loop (so we'll ignore later errors after the first
one)

Note: we could also use a non blocking send to avoid this situation but we
should also start the wg.Wait before the goroutines or earlier errors could be
lost causing another kind of hang.
2019-07-23 14:56:26 +02:00
Simone Gotti
75d68b2b52 runservice: stop run also if result is not set 2019-07-23 12:11:01 +02:00