Commit Graph

15 Commits

Author SHA1 Message Date
Simone Gotti 35e1ec0e15 datamanager: remove old storage wals
Remove all wals not required by the existing data status files and not existing
in etcd.
2019-11-08 15:39:17 +01:00
Simone Gotti 4fcb067052 datamanager: clean old data files
keep the last n (now set to 3) data status files and remove all other data status
files and unneeded data files.
2019-11-07 10:30:31 +01:00
Simone Gotti 4b4416fc99 datamanager: add data sequence to data file name
When creating a datafile name make it start with the current data sequence. This
is useful in future to know which data sequence created a new data file.
2019-11-04 09:23:12 +01:00
Simone Gotti 6f3798e8fe *: use sleep timer in loops
So we'll react instantly to a context cancel instead of waiting on time.Sleep
returning.
2019-07-25 16:22:54 +02:00
Simone Gotti 3987caf664 datamanager: implement maintenance mode
when datamanager is started in maintenance mode no goroutines are scheduled and
no initial etcd initialization is done
2019-07-24 17:37:27 +02:00
Simone Gotti c034819087 datamanager: accept optional datastatus in initEtcd 2019-07-18 16:44:28 +02:00
Simone Gotti 8fbb9fdcbe datamanager: add deleteEtcd method 2019-07-18 16:40:28 +02:00
Simone Gotti c064e490b2 datamanager: add lock around compatchangegroups and initetcd
just to avoid concurrency errors when multiple instances are running
2019-07-10 10:20:03 +02:00
Simone Gotti d989fe9639 datamanager: always handle basepath
Currently we aren't setting a basepath and it wasn't always correctly handled.
Fix missing basepath handling and improve tests to also use a non empty
basepath.
2019-07-03 17:03:37 +02:00
Simone Gotti 49d0238a1e datamanager: fix linter errors
Fix errors reported by default golangci-lint linters
2019-07-02 14:27:51 +02:00
Simone Gotti 8d67844cc4 *: use vanity url
use agola.io domain
2019-07-01 11:40:20 +02:00
Simone Gotti 22bd181fc8 datamanager: implement data files splitting
split data files in multiple files of a max size (default 10Mib)
In this way every data snapshot will change only the datafiles that have some
changes instead of the whole single file.
2019-06-03 16:17:27 +02:00
Simone Gotti 9b2ce717c7 *: migrate to "golang.org/x/xerrors"
Just a raw replace of "github.com/pkg/errors".

Next steps will improve errors (like remote errors, api errors, not exist errors
etc...) to leverage its functionalities
2019-05-23 11:23:14 +02:00
Simone Gotti b85786dc56 datamanager: check changegroup name 2019-04-29 10:12:03 +02:00
Simone Gotti 2c3e6bf9e4 wal: update and rename to datamanager
* Rename to datamanager since it handles a complete "database" backed by an
objectstorage and etcd

* Don't write every single entry as a single file but group them in a single
file. In future improve this to split the data in multiple files of a max size.
2019-04-26 16:00:03 +02:00