Commit Graph

22 Commits

Author SHA1 Message Date
Simone Gotti 5af07d0852 objectstorage: use a single package
remove all the subpackages and just use a single package
2019-11-08 16:31:48 +01:00
Simone Gotti 9c0eb3d7ef datamanager: refactor ReadWal
make ReadWal directly return a *WalHeader
2019-11-08 13:24:43 +01:00
Simone Gotti c1ff28ef9f *: export clients and related types
Export clients and related packages.

The main rule is to not import internal packages from exported packages.

The gateway client and related types are totally decoupled from the gateway
service (not shared types between the client and the server).

Instead the configstore and the runservice client currently share many types
that are now exported (decoupling them will require that a lot of types must be
duplicated and the need of functions to convert between them, this will be done
in future when the APIs will be declared as stable).
2019-08-02 12:02:01 +02:00
Simone Gotti ceafc2ef98 readdb: close and open readdb on Run 2019-07-25 17:59:54 +02:00
Simone Gotti 6f3798e8fe *: use sleep timer in loops
So we'll react instantly to a context cancel instead of waiting on time.Sleep
returning.
2019-07-25 16:22:54 +02:00
Simone Gotti b8c2b4020a db: use context functions
Use the go sql context functions (ExecContext, QueryContext etc...)

The context is saved inside Tx so the library users should only pass it one time
to the db.Do function.
2019-07-25 14:49:53 +02:00
Simone Gotti 85876310af readdb: fix deadlock in Run method
In runservice readdb Run method we could end with a deadlock if two of the
goroutines that call HandleEvents.* try to write to the errCh at the same
time before the errCh is read. If this happens one of the two will be blocked on
writing to the channel but the read won't happen since it'll blocked by
wg.Wait().

Fix this doing:
* use a buffered channel large as the number of executed goroutines.
* create a new errCh at every loop (so we'll ignore later errors after the first
one)

Note: we could also use a non blocking send to avoid this situation but we
should also start the wg.Wait before the goroutines or earlier errors could be
lost causing another kind of hang.
2019-07-23 14:56:26 +02:00
Simone Gotti 3a963ef95f readdb: error if there's no wal in etcd 2019-07-18 16:44:28 +02:00
Simone Gotti 3f64bda0cc readdb: save walSequence provided by data file 2019-07-18 16:44:28 +02:00
Simone Gotti 16820e9033 readdb: insert current wal sequence after checking wal status 2019-07-18 16:44:28 +02:00
Simone Gotti 04ef20464d runservice: add getruns filter by result 2019-07-05 10:32:51 +02:00
Simone Gotti d989fe9639 datamanager: always handle basepath
Currently we aren't setting a basepath and it wasn't always correctly handled.
Fix missing basepath handling and improve tests to also use a non empty
basepath.
2019-07-03 17:03:37 +02:00
Simone Gotti 19793db0c2 runservice: fix linter errors
Fix errors reported by default golangci-lint linters
2019-07-02 14:53:01 +02:00
Simone Gotti 8d67844cc4 *: use vanity url
use agola.io domain
2019-07-01 11:40:20 +02:00
Simone Gotti 5e21089baa *: remove unneeded logging
remove many log.Info entries that where old debugging entries and move some of
them to the Debug level.
2019-06-14 11:28:00 +02:00
Simone Gotti 22bd181fc8 datamanager: implement data files splitting
split data files in multiple files of a max size (default 10Mib)
In this way every data snapshot will change only the datafiles that have some
changes instead of the whole single file.
2019-06-03 16:17:27 +02:00
Simone Gotti 6d095cbe50 readdb: ensure that we apply only etcd committed wals
Ensure that we apply only etcd commited wals to avoid doing an unuseful insert
when the wal becomes committed storage.
2019-06-03 18:02:09 +02:00
Simone Gotti c0a165de31 runservice: fix query when grouping by path
When grouping by group path we have to apply the filter to the group by using
HAVING and not using WHERE
2019-05-23 23:21:45 +02:00
Simone Gotti 9b2ce717c7 *: migrate to "golang.org/x/xerrors"
Just a raw replace of "github.com/pkg/errors".

Next steps will improve errors (like remote errors, api errors, not exist errors
etc...) to leverage its functionalities
2019-05-23 11:23:14 +02:00
Simone Gotti b3867fb7ca objectstorage: add posix standard storage
rename the previous posix storage to posixflat and make it currently not user
selectable (since I'm not sure it's really worth using it).

The new posix storage uses the filesystem without any escaping so it's not a
real flat namespace.

This isn't a real issue since also minio is not a flat namespace and we are so
forced to use it like a hierarchycal filesystem.
2019-05-21 15:17:53 +02:00
Simone Gotti b95fb98f3c runservice: move RunEvent to types 2019-05-15 09:40:32 +02:00
Simone Gotti 1e34dca95d runservice: split and simplify scheduler and executor naming
Also if they are logically part of the runservice the names runserviceExecutor
and runserviceScheduler are long and quite confusing for an external user

Simplify them separating both the code parts and updating the names:

runserviceScheduler -> runservice
runserviceExecutor -> executor
2019-05-07 23:56:10 +02:00