Commit Graph

89 Commits

Author SHA1 Message Date
Simone Gotti c1ff28ef9f *: export clients and related types
Export clients and related packages.

The main rule is to not import internal packages from exported packages.

The gateway client and related types are totally decoupled from the gateway
service (not shared types between the client and the server).

Instead the configstore and the runservice client currently share many types
that are now exported (decoupling them will require that a lot of types must be
duplicated and the need of functions to convert between them, this will be done
in future when the APIs will be declared as stable).
2019-08-02 12:02:01 +02:00
Simone Gotti fd26e617b3 configstore: move configstore types inside configstore package
Since they're not types common to all the services but belongs to the
configstore.

Next step will be to make them local to the configstore and not directly used by
other services since these types are also stored.
2019-08-02 10:05:47 +02:00
Simone Gotti fafa5188c2 configstore: maintenance/export/import
Implement configstore maintenance mode and export/import.

When configstore is set in maintenance mode it'll start only the maintenance and
export/import handlers.

Setting maintenance mode will set a key in etcd so all the configstore instances
will detect it and enter in maintenance mode. This is done asyncronously so it
could take some time (future improvements will add some api to show all the
configstore states)

Export is always available and will export the datamanager contents.

Import is available only during maintenance, given a datamanager export will
import it and reset etcd to this import state.
2019-07-26 10:55:04 +02:00
Simone Gotti ceafc2ef98 readdb: close and open readdb on Run 2019-07-25 17:59:54 +02:00
Simone Gotti 6f3798e8fe *: use sleep timer in loops
So we'll react instantly to a context cancel instead of waiting on time.Sleep
returning.
2019-07-25 16:22:54 +02:00
Simone Gotti b8c2b4020a db: use context functions
Use the go sql context functions (ExecContext, QueryContext etc...)

The context is saved inside Tx so the library users should only pass it one time
to the db.Do function.
2019-07-25 14:49:53 +02:00
Simone Gotti 85876310af readdb: fix deadlock in Run method
In runservice readdb Run method we could end with a deadlock if two of the
goroutines that call HandleEvents.* try to write to the errCh at the same
time before the errCh is read. If this happens one of the two will be blocked on
writing to the channel but the read won't happen since it'll blocked by
wg.Wait().

Fix this doing:
* use a buffered channel large as the number of executed goroutines.
* create a new errCh at every loop (so we'll ignore later errors after the first
one)

Note: we could also use a non blocking send to avoid this situation but we
should also start the wg.Wait before the goroutines or earlier errors could be
lost causing another kind of hang.
2019-07-23 14:56:26 +02:00
Simone Gotti 3a963ef95f readdb: error if there's no wal in etcd 2019-07-18 16:44:28 +02:00
Simone Gotti 3f64bda0cc readdb: save walSequence provided by data file 2019-07-18 16:44:28 +02:00
Simone Gotti 16820e9033 readdb: insert current wal sequence after checking wal status 2019-07-18 16:44:28 +02:00
Simone Gotti 86d822a247 service: handle cors config and use it only on gateway
* Don't make cors enabled on all (*) by default.
* Handle related web.allowedOrigins options
* Only the gateway api should be called by a browser so setup the cors handler
only on it
2019-07-13 23:15:00 +02:00
Simone Gotti 276c045c3b configstore: add parenttype to related tables
Add the parenttype column to project, projectgroup, secret, variable tables.
2019-07-09 10:16:45 +02:00
Simone Gotti 8b7a5602e0 configstore: fix project/projectgroup rename
project: check name exists only if the name has changed
projectgroup: add missing duplicate name check
2019-07-08 18:12:42 +02:00
Simone Gotti 289e691a58 configstore: implement secret update 2019-07-08 10:32:32 +02:00
Simone Gotti ac0c1fc2bd configstore: implement variable update 2019-07-06 15:25:24 +02:00
Simone Gotti 28c44ce1fc configstore: update/fix remotesource rename
* Override the provided remotesource id with the current one (it could not be
provided or provided with a different id but the remotesource ref is the way to
get the current remote source).

* When changing remotesource name check that a remote source with the new name
does not already exist.
2019-07-06 14:40:31 +02:00
Simone Gotti 2c042fb4bd configstore: add remote source tests 2019-07-06 14:39:37 +02:00
Simone Gotti 929a6fb654 service/*: log error only if nil 2019-07-04 15:50:37 +02:00
Simone Gotti ea025638ff configstore: set base path for datamanager
Don't put datamanager base dirs inside the root of the ost but use a base path.

Let's do it now before releasing since this is a breaking change that requires
moving the ost data to the new path
2019-07-03 17:19:40 +02:00
Simone Gotti d989fe9639 datamanager: always handle basepath
Currently we aren't setting a basepath and it wasn't always correctly handled.
Fix missing basepath handling and improve tests to also use a non empty
basepath.
2019-07-03 17:03:37 +02:00
Simone Gotti f1908c3a16 configstore: sync readdb run method with the one of runservice 2019-07-02 15:44:33 +02:00
Simone Gotti 5643dd5dcd configstore: fix linter errors
Fix errors reported by default golangci-lint linters
2019-07-02 14:46:00 +02:00
Simone Gotti 8d67844cc4 *: use vanity url
use agola.io domain
2019-07-01 11:40:20 +02:00
Simone Gotti 5e21089baa *: remove unneeded logging
remove many log.Info entries that where old debugging entries and move some of
them to the Debug level.
2019-06-14 11:28:00 +02:00
Simone Gotti 6e8d467c80 util: add ErrInternal
ErrInternal is an internal error that should be provided to the user (http api
will return a 500 with the error message)

It'll be used for any kind of error that are not auth or bad requests (like
errors to communicate to another service)
2019-06-11 10:59:21 +02:00
Simone Gotti 22bd181fc8 datamanager: implement data files splitting
split data files in multiple files of a max size (default 10Mib)
In this way every data snapshot will change only the datafiles that have some
changes instead of the whole single file.
2019-06-03 16:17:27 +02:00
Simone Gotti 6d095cbe50 readdb: ensure that we apply only etcd committed wals
Ensure that we apply only etcd commited wals to avoid doing an unuseful insert
when the wal becomes committed storage.
2019-06-03 18:02:09 +02:00
Simone Gotti 4d7605a86b gateway: improve ErrFromRemote handling
Don't create an ErrFromRemote wrapping the returned error but
wrap the ErrFromRemote

Also use xerrors Is/As to get the underlying error to return to api clients
while maintaining context for logging
2019-05-23 12:59:11 +02:00
Simone Gotti 9b2ce717c7 *: migrate to "golang.org/x/xerrors"
Just a raw replace of "github.com/pkg/errors".

Next steps will improve errors (like remote errors, api errors, not exist errors
etc...) to leverage its functionalities
2019-05-23 11:23:14 +02:00
Simone Gotti 8f1225da76 *: implement remote source update 2019-05-23 10:29:03 +02:00
Simone Gotti b3867fb7ca objectstorage: add posix standard storage
rename the previous posix storage to posixflat and make it currently not user
selectable (since I'm not sure it's really worth using it).

The new posix storage uses the filesystem without any escaping so it's not a
real flat namespace.

This isn't a real issue since also minio is not a flat namespace and we are so
forced to use it like a hierarchycal filesystem.
2019-05-21 15:17:53 +02:00
Simone Gotti 6c11ab0596 *: implement update projectgroup 2019-05-14 17:53:48 +02:00
Simone Gotti 21e706a150 configstore: add get org members api/action 2019-05-14 12:57:53 +02:00
Simone Gotti aeda922a7d *: rename deleteOrgMember to removeOrgMember 2019-05-14 11:20:09 +02:00
Simone Gotti c5cdf01332 *: add organizations visibility 2019-05-14 10:56:17 +02:00
Simone Gotti d551dcb820 configstore: implement projectgroup delete 2019-05-13 00:23:57 +02:00
Simone Gotti 5c15eb4db7 configstore: move getvariables to action 2019-05-13 00:23:08 +02:00
Simone Gotti 295ed6e39a configstore: move get secret(s) to action 2019-05-13 00:22:23 +02:00
Simone Gotti 0f9445aabe configstore: move get projecgroup projects/subgroups to action 2019-05-13 00:21:10 +02:00
Simone Gotti c9fb7258ec configstore: fix delete remote source action 2019-05-10 00:02:37 +02:00
Simone Gotti b0fe7e4115 *: implement delete organization member 2019-05-09 17:05:13 +02:00
Simone Gotti 620bae68df *: implement add/update org member 2019-05-09 16:47:22 +02:00
Simone Gotti e1d0318c9b configstore: add UpdateProject api 2019-05-09 15:33:57 +02:00
Simone Gotti 79c1a60a36 configstore: update CreateProject action
* move validation to ValidateProject function
* remove wrong project group check
2019-05-09 15:32:27 +02:00
Simone Gotti 44d5b0f25a *: rename ConfigStore to Configstore 2019-05-07 23:42:42 +02:00
Simone Gotti 649c42f75b gitsources: create secret and webhook secret
Use the webhook secret on webhook creation and check it and webhook receive
2019-05-07 18:29:31 +02:00
Simone Gotti 2675aee333 configstore: generate User and Project secret 2019-05-07 17:16:42 +02:00
Simone Gotti ca5b5f3a7e configstore: rename command(handler) to action(handler)
Since we're going to migrate all actions (also queries that now are implemented
in the api handlers) there
2019-05-03 23:35:25 +02:00
Simone Gotti af67198dec configstore: report project/projectgroup owners
Return project and projectgroup owner type (user or org) and their id.
2019-05-03 23:17:07 +02:00
Simone Gotti 81d656b7a3 configstore: implement organization members 2019-05-03 17:40:07 +02:00