Commit Graph

21 Commits

Author SHA1 Message Date
Simone Gotti f7175c4592 datamanager: save previous wal in waldata 2019-07-18 16:44:28 +02:00
Simone Gotti 18c5ae0492 datamanager: error if there're wals but not a datastatus in ost 2019-07-18 16:44:28 +02:00
Simone Gotti c034819087 datamanager: accept optional datastatus in initEtcd 2019-07-18 16:44:28 +02:00
Simone Gotti df716fccc6 datamanager: create a new wal and checkpoint in initEtcd
When doing an initEtcd (new instance or etcd reset) create a new wal (that will
have a new sequence epoch) and do a checkpoint.

In this way:

* readdb will detect that an epoch change and do a full resync
* we always have a data file (also if empty) that provides the last checkpointed
wal. This information could be used by readdb to resync
2019-07-18 16:44:28 +02:00
Simone Gotti cb2a871be6 datamanager: start initEtcd from last datastatus 2019-07-18 16:44:28 +02:00
Simone Gotti 445ef24daa datamanager: add option to force a checkpoint 2019-07-18 16:44:27 +02:00
Simone Gotti 512162bf98 datamanager: clean etcd data before reinitialization 2019-07-18 16:42:17 +02:00
Simone Gotti c064e490b2 datamanager: add lock around compatchangegroups and initetcd
just to avoid concurrency errors when multiple instances are running
2019-07-10 10:20:03 +02:00
Simone Gotti d989fe9639 datamanager: always handle basepath
Currently we aren't setting a basepath and it wasn't always correctly handled.
Fix missing basepath handling and improve tests to also use a non empty
basepath.
2019-07-03 17:03:37 +02:00
Simone Gotti 49d0238a1e datamanager: fix linter errors
Fix errors reported by default golangci-lint linters
2019-07-02 14:27:51 +02:00
Simone Gotti 8d67844cc4 *: use vanity url
use agola.io domain
2019-07-01 11:40:20 +02:00
Simone Gotti 5e21089baa *: remove unneeded logging
remove many log.Info entries that where old debugging entries and move some of
them to the Debug level.
2019-06-14 11:28:00 +02:00
Simone Gotti 22bd181fc8 datamanager: implement data files splitting
split data files in multiple files of a max size (default 10Mib)
In this way every data snapshot will change only the datafiles that have some
changes instead of the whole single file.
2019-06-03 16:17:27 +02:00
Simone Gotti 9b2ce717c7 *: migrate to "golang.org/x/xerrors"
Just a raw replace of "github.com/pkg/errors".

Next steps will improve errors (like remote errors, api errors, not exist errors
etc...) to leverage its functionalities
2019-05-23 11:23:14 +02:00
Simone Gotti b3867fb7ca objectstorage: add posix standard storage
rename the previous posix storage to posixflat and make it currently not user
selectable (since I'm not sure it's really worth using it).

The new posix storage uses the filesystem without any escaping so it's not a
real flat namespace.

This isn't a real issue since also minio is not a flat namespace and we are so
forced to use it like a hierarchycal filesystem.
2019-05-21 15:17:53 +02:00
Simone Gotti bad18bf814 *: report objects size for objectstorage.WriteObject 2019-05-02 09:49:55 +02:00
Simone Gotti 34cfdfeb3b objectstorage: add size option to WriteObject
On s3 limit the max object size to 1GiB when the size is not provided (-1) or
the minio client will calculate a big part size since it tries to use the
maximum object size (5TiB) and will allocate a very big buffer in ram. Also
leave as commented out the previous hack that was firstly creating the file
locally to calculate the size and then put it (for future reference).
2019-05-02 09:47:38 +02:00
Simone Gotti e964aa3537 objectstorage: add persist option to WriteObject
This options is a noop on s3 but on the posix implementation it becomes useful
when there isn't the need to have a persistent file, thus avoiding some fsync
calls.
2019-05-01 15:06:47 +02:00
Simone Gotti 2634f7a3a0 datamanager: add function to get changegroups from etcd 2019-04-29 10:13:13 +02:00
Simone Gotti b85786dc56 datamanager: check changegroup name 2019-04-29 10:12:03 +02:00
Simone Gotti 2c3e6bf9e4 wal: update and rename to datamanager
* Rename to datamanager since it handles a complete "database" backed by an
objectstorage and etcd

* Don't write every single entry as a single file but group them in a single
file. In future improve this to split the data in multiple files of a max size.
2019-04-26 16:00:03 +02:00