CephFS (librados) support for our Syncwerk (EFSS) software
- or -
Post a project like this1271
€750(approx. $804)
- Posted:
- Proposals: 4
- Remote
- #3026199
- OPPORTUNITY
- Awarded
Description
Experience Level: Expert
Syncwerk is an enterprise file sync and share solution similar to Dropbox or Nextcloud. We are looking for someone who can extend our C based code to store data in CephFS. Currently data is stored on the file system. Storing data on Ceph should be an alternative to storing it on the local file system. Syncwerk is developed for Debian 10 Linux. If required, we can help set up a development environment to quickly start with the actual feature development.
Required experience and skills:
- CephFS (librados)
- Python
- Angular TypeScript
- NodeJS
- Django Rest Framework
- Debian Linux
- Good communication skills
These are the basic requirements:
1. Create a plugin to optionally allow storage in CephFS (https://ceph.io) with librados
2. Make use of rados namespaces to optionally allow storing data of different clients and categories within the same CephFS storage pool
3. Build a migration tool that allows moving data from and to the local file system (both directions)
We store data in our own object storage format on the file system. Data is split into three categories which are stored in different folders on the file system:
1. fs - /var/lib/syncwerk/storage/fs - Stores info about the type. File or directory.
2. commits - /var/lib/syncwerk/storage/commits - Stores all changes similar to Git.
3. blocks - /var/lib/syncwerk/storage/blocks - Stores the actual data of a file in approx 1MB blocks.
Here a demo listing from the data stored on a standard Syncwerk file system:
root@demo:~# tree /var/lib/syncwerk/storage/
/var/lib/syncwerk/storage/
├── blocks
│ └── 9fe50544-1fa0-4760-826e-4bb9113f1fba
│ ├── 97
│ │ └── b30c73d8f843b5be34f3611652a1be9e10d044
│ └── f2
│ └── f01c586ae746631ee271cf4dd3a1005eb4d215
├── commits
│ ├── 1a4f8aee-2ef9-40d5-b900-26ea23925c45
│ │ └── 81
│ │ └── 3cc50dd2ac6eb5147f6fdfc2d7d04491bfefe0
│ ├── 1e4432da-c922-4ebb-a374-ee24e09cd736
│ │ ├── 01
│ │ │ └── 11441155fdab75e910d8d5ec62ea516a66a3bb
│ │ └── 3e
│ │ └── 639c54f0a1fbbfe22c03eaec60f6e14ddef23b
│ ├── 53ac3c1d-bd06-4912-b1f3-8400e71336e2
│ │ ├── 25
│ │ │ └── 79a37c2aa4a8adbedbb7b812123ffb3e7ab748
│ │ └── cb
│ │ └── a010a8d85ce49a7d51e72920d62e66094d0590
│ ├── 6018ac38-8319-40a8-97b8-d67d7dbd0c80
│ │ ├── 00
│ │ │ └── d4bc3353ce42c86d3223a98bfb6ad3b223cf4a
│ │ └── 5d
│ │ └── 485ad9d4d56a749d9afa54a1f4750da8049931
│ ├── 86f7fb7f-7b32-4d24-90e8-eca85ebf5069
│ │ ├── bc
│ │ │ └── 472d8f71638f8605653bce8374f310570c9d42
│ │ └── d3
│ │ └── 1f888ed38d47d0d6e7bb30f9425fc9207c4bcd
│ ├── 8d56407f-8cb0-4177-b0af-7abe2f998b3a
│ │ └── 28
│ │ └── b50bd1adcc6ca65dd7ed7a1316104c35da6df6
│ ├── 928938a7-2ef5-412d-9630-1405ed604be1
│ │ ├── 08
│ │ │ └── 88f3c4c7485fa81309737bc774591bfbc4d2e6
│ │ ├── 30
│ │ │ └── 3721f4109586a8ca567d941204d564b289923c
│ │ └── 63
│ │ └── 007144cfbc1f4667c88173484bec69d6a90e74
│ ├── 9fe50544-1fa0-4760-826e-4bb9113f1fba
│ │ ├── 0e
│ │ │ └── 4831b44f2a812e16e1046a9ec021d61773140b
│ │ ├── 39
│ │ │ └── 706f7b5ac93110b6ebec2d4ba0e32ef34160b1
│ │ └── 49
│ │ └── 06bcf221f817c05150a7a3c1d2266776d92f28
│ ├── ab3da236-61b6-48df-88d6-8bc40a212e40
│ │ └── 60
│ │ └── 72831c7373bff090bbdab47e66026330b6c48f
│ ├── d23b914a-8cb0-4f4e-a35d-7bc22d922b09
│ │ └── 58
│ │ └── 8b30599aeef005b37c5115b082c8c6e19a6f50
│ ├── db19569d-643f-4f64-bbd7-d196a54b8305
│ │ └── 7f
│ │ └── 20baa98091b9d8b7519e08a8e8ff8db1c2f48f
│ └── f46d1146-a6e6-4347-afbf-c1eae8b287c4
│ └── a0
│ └── 45bd9ba79f6f59a7f4d856d842ebdb323a9fee
└── fs
├── 1e4432da-c922-4ebb-a374-ee24e09cd736
│ └── f4
│ └── 0eff467067a83c2397388b4cb16a07e9e1f64b
├── 53ac3c1d-bd06-4912-b1f3-8400e71336e2
│ └── bd
│ └── f788ace702322cc436a53f4eb25e04acd64bb2
├── 6018ac38-8319-40a8-97b8-d67d7dbd0c80
│ └── 85
│ └── 9613ce246aff48a891f886acca818678dd9437
├── 86f7fb7f-7b32-4d24-90e8-eca85ebf5069
│ └── 02
│ └── fae3b4e15c5a468e42f9b7ded01556d860574c
├── 928938a7-2ef5-412d-9630-1405ed604be1
│ └── f7
│ └── cb8eccbb3d6a8b4e94aee5de9d8b1dda2849f4
└── 9fe50544-1fa0-4760-826e-4bb9113f1fba
├── 97
│ └── 85a99aa7254ca34ed5b81111e2c49b6fad5a81
├── c6
│ └── 134c05df189d2237ad2e3c7cce431f339f609a
├── e8
│ └── cd1f9ff5deabc4cac1bac83dc973f59cc6506e
└── ef
└── 5c0362a8128f916b315ad96fbcf157631c563f
53 directories, 31 files
The Syncwerk code base needs to get extended to support storing fs, commits and blocks directly in Ceph pools with librados. For performance reasons some caching with redis and/or memcached may be necessary.
Required experience and skills:
- CephFS (librados)
- Python
- Angular TypeScript
- NodeJS
- Django Rest Framework
- Debian Linux
- Good communication skills
These are the basic requirements:
1. Create a plugin to optionally allow storage in CephFS (https://ceph.io) with librados
2. Make use of rados namespaces to optionally allow storing data of different clients and categories within the same CephFS storage pool
3. Build a migration tool that allows moving data from and to the local file system (both directions)
We store data in our own object storage format on the file system. Data is split into three categories which are stored in different folders on the file system:
1. fs - /var/lib/syncwerk/storage/fs - Stores info about the type. File or directory.
2. commits - /var/lib/syncwerk/storage/commits - Stores all changes similar to Git.
3. blocks - /var/lib/syncwerk/storage/blocks - Stores the actual data of a file in approx 1MB blocks.
Here a demo listing from the data stored on a standard Syncwerk file system:
root@demo:~# tree /var/lib/syncwerk/storage/
/var/lib/syncwerk/storage/
├── blocks
│ └── 9fe50544-1fa0-4760-826e-4bb9113f1fba
│ ├── 97
│ │ └── b30c73d8f843b5be34f3611652a1be9e10d044
│ └── f2
│ └── f01c586ae746631ee271cf4dd3a1005eb4d215
├── commits
│ ├── 1a4f8aee-2ef9-40d5-b900-26ea23925c45
│ │ └── 81
│ │ └── 3cc50dd2ac6eb5147f6fdfc2d7d04491bfefe0
│ ├── 1e4432da-c922-4ebb-a374-ee24e09cd736
│ │ ├── 01
│ │ │ └── 11441155fdab75e910d8d5ec62ea516a66a3bb
│ │ └── 3e
│ │ └── 639c54f0a1fbbfe22c03eaec60f6e14ddef23b
│ ├── 53ac3c1d-bd06-4912-b1f3-8400e71336e2
│ │ ├── 25
│ │ │ └── 79a37c2aa4a8adbedbb7b812123ffb3e7ab748
│ │ └── cb
│ │ └── a010a8d85ce49a7d51e72920d62e66094d0590
│ ├── 6018ac38-8319-40a8-97b8-d67d7dbd0c80
│ │ ├── 00
│ │ │ └── d4bc3353ce42c86d3223a98bfb6ad3b223cf4a
│ │ └── 5d
│ │ └── 485ad9d4d56a749d9afa54a1f4750da8049931
│ ├── 86f7fb7f-7b32-4d24-90e8-eca85ebf5069
│ │ ├── bc
│ │ │ └── 472d8f71638f8605653bce8374f310570c9d42
│ │ └── d3
│ │ └── 1f888ed38d47d0d6e7bb30f9425fc9207c4bcd
│ ├── 8d56407f-8cb0-4177-b0af-7abe2f998b3a
│ │ └── 28
│ │ └── b50bd1adcc6ca65dd7ed7a1316104c35da6df6
│ ├── 928938a7-2ef5-412d-9630-1405ed604be1
│ │ ├── 08
│ │ │ └── 88f3c4c7485fa81309737bc774591bfbc4d2e6
│ │ ├── 30
│ │ │ └── 3721f4109586a8ca567d941204d564b289923c
│ │ └── 63
│ │ └── 007144cfbc1f4667c88173484bec69d6a90e74
│ ├── 9fe50544-1fa0-4760-826e-4bb9113f1fba
│ │ ├── 0e
│ │ │ └── 4831b44f2a812e16e1046a9ec021d61773140b
│ │ ├── 39
│ │ │ └── 706f7b5ac93110b6ebec2d4ba0e32ef34160b1
│ │ └── 49
│ │ └── 06bcf221f817c05150a7a3c1d2266776d92f28
│ ├── ab3da236-61b6-48df-88d6-8bc40a212e40
│ │ └── 60
│ │ └── 72831c7373bff090bbdab47e66026330b6c48f
│ ├── d23b914a-8cb0-4f4e-a35d-7bc22d922b09
│ │ └── 58
│ │ └── 8b30599aeef005b37c5115b082c8c6e19a6f50
│ ├── db19569d-643f-4f64-bbd7-d196a54b8305
│ │ └── 7f
│ │ └── 20baa98091b9d8b7519e08a8e8ff8db1c2f48f
│ └── f46d1146-a6e6-4347-afbf-c1eae8b287c4
│ └── a0
│ └── 45bd9ba79f6f59a7f4d856d842ebdb323a9fee
└── fs
├── 1e4432da-c922-4ebb-a374-ee24e09cd736
│ └── f4
│ └── 0eff467067a83c2397388b4cb16a07e9e1f64b
├── 53ac3c1d-bd06-4912-b1f3-8400e71336e2
│ └── bd
│ └── f788ace702322cc436a53f4eb25e04acd64bb2
├── 6018ac38-8319-40a8-97b8-d67d7dbd0c80
│ └── 85
│ └── 9613ce246aff48a891f886acca818678dd9437
├── 86f7fb7f-7b32-4d24-90e8-eca85ebf5069
│ └── 02
│ └── fae3b4e15c5a468e42f9b7ded01556d860574c
├── 928938a7-2ef5-412d-9630-1405ed604be1
│ └── f7
│ └── cb8eccbb3d6a8b4e94aee5de9d8b1dda2849f4
└── 9fe50544-1fa0-4760-826e-4bb9113f1fba
├── 97
│ └── 85a99aa7254ca34ed5b81111e2c49b6fad5a81
├── c6
│ └── 134c05df189d2237ad2e3c7cce431f339f609a
├── e8
│ └── cd1f9ff5deabc4cac1bac83dc973f59cc6506e
└── ef
└── 5c0362a8128f916b315ad96fbcf157631c563f
53 directories, 31 files
The Syncwerk code base needs to get extended to support storing fs, commits and blocks directly in Ceph pools with librados. For performance reasons some caching with redis and/or memcached may be necessary.
Syncwerk GmbH
100% (22)Projects Completed
14
Freelancers worked with
15
Projects awarded
31%
Last project
8 Jun 2022
Germany
New Proposal
Login to your account and send a proposal now to get this project.
Log inClarification Board Ask a Question
-
There are no clarification messages.
We collect cookies to enable the proper functioning and security of our website, and to enhance your experience. By clicking on 'Accept All Cookies', you consent to the use of these cookies. You can change your 'Cookies Settings' at any time. For more information, please read ourCookie Policy
Cookie Settings
Accept All Cookies