Commit Graph

19 Commits

Author SHA1 Message Date
Simone Gotti ed53183431 go.mod: update dependencies
Update all the updatable dependencies
2020-02-18 13:55:50 +01:00
Simone Gotti 72f279c4c3 *: improve error handling
* objectstorage: remove `types` package and move `ErrNotExist` in base package
* objectstorage: Implement .Is and add helper `IsErrNotExist` for `ErrNotExist`
* util: Rename `ErrNotFound` to `ErrNotExist`
* util: Add `IsErr*` helpers and use them in place of `errors.Is()`
* datamanager: add `ErrNoDataStatus` to report when there's not data status in ost
* runservice/common: remove `ErrNotExist` and use errors in util package
2019-11-11 12:17:35 +01:00
Simone Gotti 5af07d0852 objectstorage: use a single package
remove all the subpackages and just use a single package
2019-11-08 16:31:48 +01:00
Simone Gotti 1eb16886d8 objectstorage: add WriteObject tests
Test WriteObject using different size values: unspecified, equal to the buffer
size or less than the buffer size.
2019-10-25 15:16:42 +02:00
Simone Gotti 2e520dae55
Merge pull request #154 from sgotti/objectstorage_object_size
objectstorage: return object size in objectinfo
2019-10-25 12:23:48 +02:00
Simone Gotti 4c88bb75a3
Merge pull request #153 from sgotti/objectstorage_posix_limitreader_only_size_gt_0
objectstorage posix: use limitreader only when size is specified.
2019-10-25 12:23:35 +02:00
Simone Gotti ae1e92b115 objectstorage: return object size in objectinfo
Return object size in object info.
2019-10-25 10:53:36 +02:00
Simone Gotti 58f68601e6 objectstorage posix: use limitreader only when size is specified.
Use limitreader only when size is specified (greater or equal to 0).
When size is unknown (less than 0) limitreader will immediately return EOF
instead of writing the whole data.
2019-10-25 10:26:59 +02:00
Simone Gotti 0388003d09 objectstorage s3: use limitreader in write object
If size is specified limit reads to size bytes.
2019-10-25 10:06:22 +02:00
Simone Gotti 3a7ba2694d internal: fix linter errors
Fix errors reported by default golangci-lint linters
2019-07-02 16:20:53 +02:00
Simone Gotti 8d67844cc4 *: use vanity url
use agola.io domain
2019-07-01 11:40:20 +02:00
Simone Gotti 8e4555373d objectstorage: honor write size in posix objectstorage 2019-06-05 14:56:56 +02:00
Simone Gotti 9b2ce717c7 *: migrate to "golang.org/x/xerrors"
Just a raw replace of "github.com/pkg/errors".

Next steps will improve errors (like remote errors, api errors, not exist errors
etc...) to leverage its functionalities
2019-05-23 11:23:14 +02:00
Simone Gotti b3867fb7ca objectstorage: add posix standard storage
rename the previous posix storage to posixflat and make it currently not user
selectable (since I'm not sure it's really worth using it).

The new posix storage uses the filesystem without any escaping so it's not a
real flat namespace.

This isn't a real issue since also minio is not a flat namespace and we are so
forced to use it like a hierarchycal filesystem.
2019-05-21 15:17:53 +02:00
Simone Gotti 34cfdfeb3b objectstorage: add size option to WriteObject
On s3 limit the max object size to 1GiB when the size is not provided (-1) or
the minio client will calculate a big part size since it tries to use the
maximum object size (5TiB) and will allocate a very big buffer in ram. Also
leave as commented out the previous hack that was firstly creating the file
locally to calculate the size and then put it (for future reference).
2019-05-02 09:47:38 +02:00
Simone Gotti e964aa3537 objectstorage: add persist option to WriteObject
This options is a noop on s3 but on the posix implementation it becomes useful
when there isn't the need to have a persistent file, thus avoiding some fsync
calls.
2019-05-01 15:06:47 +02:00
Simone Gotti 2c3e6bf9e4 wal: update and rename to datamanager
* Rename to datamanager since it handles a complete "database" backed by an
objectstorage and etcd

* Don't write every single entry as a single file but group them in a single
file. In future improve this to split the data in multiple files of a max size.
2019-04-26 16:00:03 +02:00
Simone Gotti 57161477ca objectstorage: return object last modified time 2019-04-15 09:37:34 +02:00
Simone Gotti 7aae1d3e1b Add initial objectstorage implementation 2019-02-21 16:00:48 +01:00