Commit Graph

651 Commits

Author SHA1 Message Date
Simone Gotti abb44b483b
Merge pull request #197 from camandel/tests_unique_name_cs3
tests: add unique name for logs of the third ConfigStore instance
2020-01-17 09:19:40 +01:00
Carlo Mandelli d049782e29 tests: add unique name for logs of the third ConfigStore instance 2020-01-16 13:54:13 +01:00
Simone Gotti fd9a69bbb2
Merge pull request #194 from sgotti/tests_improve_services_logging
tests: improve services logging
2020-01-15 15:01:19 +01:00
Simone Gotti 2de91549a3 tests: improve services logging
During tests provide a zaptest Logger so all services output will be redirected
to golang testing logger.

When multiple services of the same type are provided add a unique name field to
distinguish them.
2020-01-15 12:30:34 +01:00
Simone Gotti 180d8dd819
Merge pull request #193 from sgotti/executor_docker_toolbox_volume
docker: create a toolbox volume for every pod
2020-01-13 09:57:55 +01:00
Simone Gotti ecf355721f docker: create a toolbox volume for every pod
Instead of doing the current hack of copying the agola toolbox inside the host
tmp dir (always done but only needed when running the executor inside a docker
container) that has different issues (like tmp file removal done by
tmpwatch/systemd-tmpfiles), use a solution similar to the k8s driver: for every
pod create a volume containing the agola-toolbox and remove it at pod removal.

We could also use a single "global" volume but we should handle cases like
volume removal (i.e. a docker volume prune command). So for now just create a
dedicated per pod volume.
2020-01-10 12:25:12 +01:00
Simone Gotti a438a065a1
Merge pull request #189 from sgotti/datamanager_dont_wait_etcd_down
datamanager tests: don't wait for etcd down
2019-12-02 15:06:16 +01:00
Simone Gotti b8d12d7962
Merge pull request #188 from camandel/add_taskid_run_list
cmd: add TaskID to run list output
2019-12-02 13:41:06 +01:00
Simone Gotti eafa4d1381 datamanager tests: don't wait for etcd down
It is causing some timeout errors since there can be another instance from
another test run in parallel started on the same port.
2019-12-02 13:32:57 +01:00
Carlo Mandelli 8bbdc0a36e cmd: add TaskID to run list output 2019-12-02 08:51:11 +01:00
Simone Gotti 2232e2895e
Merge pull request #185 from camandel/add_run_list_info
cmd: add details to run list
2019-11-29 16:21:45 +01:00
Carlo Mandelli a15a769610 cmd: add details to run list 2019-11-29 14:52:04 +01:00
Simone Gotti 05984f0f62
Merge pull request #187 from camandel/cli_log_management
cmd: add log get and delete subcommands
2019-11-29 10:41:31 +01:00
Carlo Mandelli 8aade1859b cmd: add log get and delete subcommands 2019-11-28 13:39:00 +01:00
Simone Gotti cc70a55a59
Merge pull request #186 from camandel/fix_tests_api_delete
gateway: fix api used by GetRunTask
2019-11-27 08:52:37 +01:00
Carlo Mandelli 4464c9a6f2 gateway: fix api used by GetRunTask 2019-11-25 14:42:34 +01:00
Simone Gotti c55f1f2e16
Merge pull request #184 from camandel/fix_change_visibility
cmd: fix project visibility switch
2019-11-21 15:22:05 +01:00
Carlo Mandelli 42e302df50 cmd: fix project visibility switch 2019-11-21 14:44:50 +01:00
Simone Gotti 074f346c78
Merge pull request #182 from sgotti/release_v0.4.0
Release v0.4.0
2019-11-19 10:40:30 +01:00
Simone Gotti e077ac8ba9 Release v0.4.0 2019-11-19 09:52:33 +01:00
Simone Gotti f5998a78d4
Merge pull request #157 from camandel/api_delete_logs
gateway/runservice: add api to delete step logs
2019-11-18 13:58:25 +01:00
Carlo Mandelli 3e47bc601a gateway/runservice: add api to delete step logs 2019-11-18 10:34:56 +01:00
Simone Gotti 3ff18510a6
Merge pull request #181 from sgotti/runservice_logshandler_improve_errors_02
runservice: improve errors in logsHandler
2019-11-15 16:46:40 +01:00
Simone Gotti 7e8f7155d7 runservice: improve errors in logsHandler
return errNotExist in readTaskLogs when the run,task or step doesn't exist.
2019-11-15 15:50:58 +01:00
Simone Gotti 5fe3ceb60c
Merge pull request #180 from sgotti/api_write_flush_logs_headers
*: write and flush header on log handlers
2019-11-15 10:18:28 +01:00
Simone Gotti f7d0950ca1 *: write and flush header on log handlers
Explicitly write and flush the headers in the various services LogHandlers.

Currently the 200 response and the other headers will be automatically written
by the golang http implementation only when we send something in the body. But if
there's nothing to send (no logs yet written) the client will never receive the
headers and cannot know if the request was successful.
2019-11-14 10:52:45 +01:00
Simone Gotti 32a08ec5c8
Merge pull request #179 from sgotti/runservice_logshandler_improve_errors
runservice: improve errors in logsHandler
2019-11-14 09:56:24 +01:00
Simone Gotti eeacc296d0
Merge pull request #177 from sgotti/runservice_use_trylock_on_fetching
runservice: use etcd mutex TryLock on fetching
2019-11-13 18:02:34 +01:00
Simone Gotti 66e182a55d runservice: improve errors in logsHandler
* return errNotExist in readTaskLogs when the executor task doesn't exist: so
the client will receive a 404 instead of a 500 (since a generic error will be
mapped to a 500).
* Wrap the errNotExist returned by readTaskLogs with a new ErrNotExits reporting
"log doesn't exist"
2019-11-13 15:50:20 +01:00
Simone Gotti 07cde065c8 runservice: use etcd mutex TryLock on fetching
When fetching avoid concurrent fetches from multiple runservices using an etcd
mutex TryLock.
2019-11-13 11:53:54 +01:00
Simone Gotti 9fd4b662a8
Merge pull request #175 from camandel/api_logarchived
gateway: add api to get log status
2019-11-13 11:45:31 +01:00
Simone Gotti 4497e196ba
Merge pull request #176 from sgotti/use_etcd_mutex_trylock
*: use etcd mutex TryLock
2019-11-13 11:32:29 +01:00
Carlo Mandelli 8ed813e722 gateway: add api to get log status 2019-11-13 10:01:51 +01:00
Simone Gotti 5ab9f7c970 *: use etcd mutex TryLock
etcd PR 11104 (https://github.com/etcd-io/etcd/pull/11104) implemented mutex
TryLock. Since it's only available in etcd master just copy relevant code and
add a TODO to remove it when updating the etcd client to a version implementing
TryLock.

Use TryLock everywhere where it'll be useful.
2019-11-12 22:27:17 +01:00
Simone Gotti 24a9563872
Merge pull request #167 from sgotti/datamanager_writedatasnapshot_skip_already_checkpointed_wals
datamanager: skip already applied wals in writeDataSnapshot
2019-11-12 14:57:42 +01:00
Simone Gotti 2076b171eb
Merge pull request #172 from sgotti/readdb_improve_handleevents_goroutines_exit
readdb: improve HandleEvents goroutine exiting
2019-11-12 14:57:22 +01:00
Simone Gotti d679254516 readdb: improve HandleEvents goroutine exiting
Rename errCh to doneCh (error is not needed) and always send to it when one of
the HandleEvents functions exits (not only on error).

This will ensure that all the goroutines will be stopped also if one of them
returns without an error.
2019-11-12 11:03:21 +01:00
Simone Gotti 1e70e3404b datamanager: skip already applied wals in writeDataSnapshot
As an optimization don't apply already applied wals.
2019-11-12 11:02:52 +01:00
Simone Gotti aff44f7e89
Merge pull request #174 from sgotti/datamanager_test_increas_etcd_waitdown_timeout
datamanager tests: increase etcd waitdown timeout
2019-11-12 11:02:12 +01:00
Simone Gotti dfd0f8c806 datamanager tests: increase etcd waitdown timeout 2019-11-12 10:47:28 +01:00
Simone Gotti 581a610523
Merge pull request #169 from sgotti/improve_error_handling
*: improve error handling
2019-11-11 14:08:39 +01:00
Simone Gotti 72f279c4c3 *: improve error handling
* objectstorage: remove `types` package and move `ErrNotExist` in base package
* objectstorage: Implement .Is and add helper `IsErrNotExist` for `ErrNotExist`
* util: Rename `ErrNotFound` to `ErrNotExist`
* util: Add `IsErr*` helpers and use them in place of `errors.Is()`
* datamanager: add `ErrNoDataStatus` to report when there's not data status in ost
* runservice/common: remove `ErrNotExist` and use errors in util package
2019-11-11 12:17:35 +01:00
Simone Gotti 89eb87a827
Merge pull request #170 from sgotti/objectstorage_use_single_package
objectstorage: use a single package
2019-11-11 10:24:54 +01:00
Simone Gotti 5af07d0852 objectstorage: use a single package
remove all the subpackages and just use a single package
2019-11-08 16:31:48 +01:00
Simone Gotti 35e1ec0e15 datamanager: remove old storage wals
Remove all wals not required by the existing data status files and not existing
in etcd.
2019-11-08 15:39:17 +01:00
Simone Gotti 9fbfeb9d13
Merge pull request #168 from sgotti/datamanager_refactor_readwal
datamanager: refactor ReadWal
2019-11-08 14:26:55 +01:00
Simone Gotti 9c0eb3d7ef datamanager: refactor ReadWal
make ReadWal directly return a *WalHeader
2019-11-08 13:24:43 +01:00
Simone Gotti cd5c5a20ed
Merge pull request #165 from sgotti/datamanager_check_wal_previousequence_is_correct
datamanager: check wal previouswalsequence is correct in initEtcd
2019-11-08 09:40:44 +01:00
Simone Gotti 9c1f3b2a69 datamanager: check wal previouswalsequence is correct in initEtcd 2019-11-07 17:05:40 +01:00
Simone Gotti 33e2d50036
Merge pull request #164 from sgotti/datamanager_dont_create_ost_checkpointed_files
datamanager: don't create ost wal checkpointed files
2019-11-07 14:48:12 +01:00