Is the project still alive?

[bitchecker]

Description of problem:
The Red Hat Gluster Storage product has a defined support lifecycle through to 31-Dec-24, after which the Red Hat Gluster Storage product will have reached its EOL. Specifically, RHGS 3.5 represents the final supported RHGS series of releases.

GlusterFS was acquired by Red Hat in 2011 so, it's strange that who bought a project, is going to close a product based on that technology.

So, the question is: Is Glusterfs going to be closed or unmaintained in some way?

[aravindavk]

The Red Hat Gluster Storage is based on upstream Gluster 6.x. Gluster 6.x is already EOL on 12th May 2020 (Ref: https://www.gluster.org/release-schedule/). Only Red Hat customers are affected with RHGS EOL.

Gluster community affected by Red Hat's decision to stop their Gluster based product? Yes. It will definitely impact the Gluster development but the project may not die since there are many maintainers and developers contributing to Gluster outside the Red Hat.

Feature development and the bug fixes may slow down, but I think (and I hope) the project will continue in slow pace.

--
Aravinda
Kadalu Technologies

[bitchecker]

The Red Hat Gluster Storage is based on upstream Gluster 6.x. Gluster 6.x is already EOL on 12th May 2020 (Ref: https://www.gluster.org/release-schedule/). Only Red Hat customers are affected with RHGS EOL.

Gluster community affected by Red Hat's decision to stop their Gluster based product? Yes. It will definitely impact the Gluster development but the project may not die since there are many maintainers and developers contributing to Gluster outside the Red Hat.

I'm completely agreed about the main point RHGS is different from Community, but ff course RH was one of the major developers on the project also because they bought it.

Feature development and the bug fixes may slow down, but I think (and I hope) the project will continue in slow pace.

🤞🏻

[anon314159]

The Red Hat Gluster Storage is based on upstream Gluster 6.x. Gluster 6.x is already EOL on 12th May 2020 (Ref: https://www.gluster.org/release-schedule/). Only Red Hat customers are affected with RHGS EOL.

Gluster community affected by Red Hat's decision to stop their Gluster based product? Yes. It will definitely impact the Gluster development but the project may not die since there are many maintainers and developers contributing to Gluster outside the Red Hat.

Feature development and the bug fixes may slow down, but I think (and I hope) the project will continue in slow pace.

-- Aravinda Kadalu Technologies

While most of this is true, the actual maintainers and those responsible for implementing bug fixes and feature updates are nearly all Red Hat employees. So there's a very legitimate concern that this project will stagnant or get pillaged and then made proprietary. Oh yes, just like your company is doing right now. No? So yes, the phenomenon of leveraging the GPL to essentially take OSS and then close it off for profit is clearly GlusterFS's future.

[mykaul]

@anon314159 , you've joined Github just a month ago, and all you did was raise accusations against two open source projects, GlusterFS and LizardFS (lizardfs/lizardfs#936 ). How about positively contributing to the community instead?

[aravindavk]

The Red Hat Gluster Storage is based on upstream Gluster 6.x. Gluster 6.x is already EOL on 12th May 2020 (Ref: https://www.gluster.org/release-schedule/). Only Red Hat customers are affected with RHGS EOL.
Gluster community affected by Red Hat's decision to stop their Gluster based product? Yes. It will definitely impact the Gluster development but the project may not die since there are many maintainers and developers contributing to Gluster outside the Red Hat.
Feature development and the bug fixes may slow down, but I think (and I hope) the project will continue in slow pace.
-- Aravinda Kadalu Technologies

While most of this is true, the actual maintainers and those responsible for implementing bug fixes and feature updates are nearly all Red Hat employees. So there's a very legitimate concern that this project will stagnant or get pillaged and then made proprietary. Oh yes, just like your company is doing right now. No? So yes, the phenomenon of leveraging the GPL to essentially take OSS and then close it off for profit is clearly GlusterFS's future.

Where are you seeing my company selling Gluster based proprietary product? At Kadalu Technologies, we offer Gluster Consultancy, Feature development and the stable packages. All the fixes/enhancements are contributed back to the main project itself. We also have couple of projects based on Gluster core file system (Ex: Kadalu Storage for Kubernetes, Kadalu Storage for non-k8s deployments). All projects/products based on GlusterFS are Open Source. Please clarify where we are closing the source and selling it.

[sac]

The Red Hat Gluster Storage is based on upstream Gluster 6.x. Gluster 6.x is already EOL on 12th May 2020 (Ref: https://www.gluster.org/release-schedule/). Only Red Hat customers are affected with RHGS EOL.
Gluster community affected by Red Hat's decision to stop their Gluster based product? Yes. It will definitely impact the Gluster development but the project may not die since there are many maintainers and developers contributing to Gluster outside the Red Hat.
Feature development and the bug fixes may slow down, but I think (and I hope) the project will continue in slow pace.
-- Aravinda Kadalu Technologies

While most of this is true, the actual maintainers and those responsible for implementing bug fixes and feature updates are nearly all Red Hat employees. So there's a very legitimate concern that this project will stagnant or get pillaged and then made proprietary. Oh yes, just like your company is doing right now. No? So yes, the phenomenon of leveraging the GPL to essentially take OSS and then close it off for profit is clearly GlusterFS's future.

GlusterFS is licensed under GPLv3 (Not BSD). A simple search would have shown you that one cannot close the source.
And you accuse a company of doing something without bothering to know what they are trying to do. As @aravindavk pointed out we would be interested to know how you concluded that Kadalu is closing the code.

[pranithk]

So, the question is: Is Glusterfs going to be closed or unmaintained in some way?

It is definitely going through a slow period because Red Hat is stopping Gluster based product. But the project is living on. Some of the maintainers from Red Hat moved to 3 different companies. My company's experiences I talked about with another Maintainer @sanjurakonde at fosdem 2023 last year. https://archive.fosdem.org/2023/schedule/event/sds_lessons_learnt_glusterfs/

Hoping more companies show interest in glusterfs and either hire more developers or buy support from Kadalu in the coming years so that things get better.

[amarts]

Is Glusterfs going to be closed or unmaintained in some way?

putting my thoughts on this, considering I am one of the maintainers not inside RHT/IBM since at least 5 years. For sure the frequency of jumping on issues and fixing them pro-actively has reduced. But, if there is a PR from any of the other maintainers, then it gets in quicker, as we do watch out all PRs.
There may be some delay in 'approval' for PRs from new comers as it depends on specific module's maintainers, and there are many modules which are individually maintained.

If there is a new feature, then there should be more discussions to get them in, and may take time. But any CVE/Security or blocker issues would be looked at immediately.

One thing which will reduce (and in many cases, has reduced already) is looking at issues, as it takes significant time for each developer, and each of them would be already moved into different companies/projects. This is where, most users would be advised to work with those developers/companies which provide support.

[anon314159]

The Red Hat Gluster Storage is based on upstream Gluster 6.x. Gluster 6.x is already EOL on 12th May 2020 (Ref: https://www.gluster.org/release-schedule/). Only Red Hat customers are affected with RHGS EOL.
Gluster community affected by Red Hat's decision to stop their Gluster based product? Yes. It will definitely impact the Gluster development but the project may not die since there are many maintainers and developers contributing to Gluster outside the Red Hat.
Feature development and the bug fixes may slow down, but I think (and I hope) the project will continue in slow pace.
-- Aravinda Kadalu Technologies

While most of this is true, the actual maintainers and those responsible for implementing bug fixes and feature updates are nearly all Red Hat employees. So there's a very legitimate concern that this project will stagnant or get pillaged and then made proprietary. Oh yes, just like your company is doing right now. No? So yes, the phenomenon of leveraging the GPL to essentially take OSS and then close it off for profit is clearly GlusterFS's future.

GlusterFS is licensed under GPLv3 (Not BSD). A simple search would have shown you that one cannot close the source. And you accuse a company of doing something without bothering to know what they are trying to do. As @aravindavk pointed out we would be interested to know how you concluded that Kadalu is closing the code.

That's cute but I can list on one hand how many FOSS based SDS stacks that leveraged the GPL but where acquired by a private company and then immediately made proprietary in direct violation of the FSF's GPL. Rules do not matter, when they are not enforced. When is the last time the FSF, copyright or patent holders pushed back against this shady practice? Essentially it's free software development that private companies can exploit with little cost. I completely understand that free doesn't imply cost or monetary gain, only the distribution, as there's no such thing as a fee lunch but it doesn't mean the abuse of FOSS isn't happening at all. Essentially it's the stripping of previous features in favor of support subscriptions based on capacity or the aforementioned previous features but at a price. Providing support services are one thing, but forking from a main branch, privatizing it, removing features and charging money favors the notification of all patent/copyright holders. I farely certain they wouldn't be too happy about their code being straight misused in such a manner. This process is usually a slow burn against FOSS in favor of socialized software but privatized profiteering/closed sourcing it over a certain period of time. In essence companies can claim immunity while simultaneously taking but not actually contributing.

Examples:

JuiceFS
RozoFS
OpenIO
IPFS
OpenAFS
GlusterFS's Future (ex. Kadalu Storage)

Interesting but dated article related to abusing the "commons clause" for profit.
https://techcrunch.com/2018/11/29/the-crusade-against-open-source-abuse/?guccounter=1&guce_referrer=aHR0cHM6Ly9kdWNrZHVja2dvLmNvbS8&guce_referrer_sig=AQAAAINdr8XrxxU1beOpShJ3mouQ5ir2CizxULE-27aIJ80dVoAHq8qdt_HjVibMNo5kXa_D-CBFOoaaTWZ7aRfGZPm4y1D3L9NCA33-9FD5zGBmZ6hLfZ9X_nfus8KtQSQ4cWoJRPaeivrE0mc8EcWjPKoz1GxUEv0RZqLLVgRv4bKz

[aravindavk]

@anon314159

That's cute but I can list on one hand how many FOSS based SDS stacks that leveraged the GPL but where acquired by a private company and then immediately made proprietary in direct violation of the FSF's GPL. Rules do not matter, when they are not enforced. When is the last time the FSF, copyright or patent holders pushed back against this shady practice? Essentially it's free software development that private companies can exploit with little cost.

There may be many examples where the companies violate the Open Source but I would like to clarify a few things wrt Gluster.

• Architects of Gluster project are from multiple companies (Ref: https://github.com/gluster/glusterfs/blob/devel/MAINTAINERS#L52). Red Hat alone can't change the direction of the Gluster project (Same for other companies)
• Parts of the Gluster project can't be made proprietary as per the license.
• Since Red Hat/IBM owns the brand they can decide to close the project, but nothing stops the community to fork it and maintain with a new name.
• Many companies are already selling Gluster packages, Gluster based deployments to their customers. This is much much before the Gluster project slowdown, why is this a problem now? The Gluster license doesn't restrict the use of Gluster for commercial applications. But all the source code modifications should be given back to the Gluster project or make it public.
• Many features can't be supported by the companies who sell Gluster to their customers only because they don't have expertise to support a feature. But it is not possible to hide a feature to charge extra money. The feature is anyway available in upstream main/devel branch.
• There are many projects that are dependent on Gluster but do not modify the source. These projects need not be in the same license as of Gluster. It can be private as well. For example, one can build a Web dashboard/S3 gateway/Alternative Geo-replication/Monitoring for Gluster and sell it as proprietary, it may use the Gluster commands installed on the machine but do not modify the source code. As per my understanding, these are all not the violation of GPL.

I completely understand that free doesn't imply cost or monetary gain, only the distribution, as there's no such thing as a fee lunch but it doesn't mean the abuse of FOSS isn't happening at all. Essentially it's the stripping of previous features in favor of support subscriptions based on capacity or the aforementioned previous features but at a price.

The Gluster source will always be available and the packages/releases will be available based on the release/package maintainers availability. If all the release/package maintainers stop working on Gluster and no releases are available, even then the packages can be built from the source.

On other hand, if any company builds the Gluster packages, spends time on testing for real world use cases and assures to provide the updates for the long term, then the interested users may buy the packages from the company. It has nothing to do with the features, people are interested to buy because of the assurance given by the supporting company.

Private Gluster packages offerings have been available for a long time from many companies even before Red Hat's decision to make their Gluster product EOL. The point is nobody can make Gluster private. The Open Source project source is always accessible to users.

Providing support services are one thing, but forking from a main branch, privatizing it, removing features and charging money favors the notification of all patent/copyright holders. I farely certain they wouldn't be too happy about their code being straight misused in such a manner. This process is usually a slow burn against FOSS in favor of socialized software but privatized profiteering/closed sourcing it over a certain period of time. In essence companies can claim immunity while simultaneously taking but not actually contributing.

As per my understanding, all the source code changes must be given back to the main branch as per the license. Please write here if you know any companies violating the license wrt Gluster source.

GlusterFS's Future (ex. Kadalu Storage)

I wish :)

Kadalu Storage is also a Open Source project and it uses the core Gluster components and a new management layer as alternative to Glusterd. Kadalu Storage contributes all the fixes that belong to Gluster core back to the Gluster project. Any changes done to the management layers are only with the Kadalu Storage project since it is not applicable to Gluster (https://github.com/kadalu/moana).

• Can we make an existing Gluster feature proprietary in Kadalu Storage? No. Unless we reimplement it in our own way without changing the Gluster source. For example, Erasure coding is not yet available in Kadalu Storage. Once it is integrated and available with Kadalu Storage, can we make it proprietary? No.
• Can we make a new feature (Not available in Gluster) as proprietary? Yes. We can make it proprietary only if the new feature is not part of Gluster source code. For example, Kadalu Storage provides template based Volfile generation, Persistent Brick ports, or Storage console and many other new features compared to Gluster. None of these new features modify the Gluster source code. We can make these features proprietary if we want to. But we decided to not hide any features and make all the features available with the same license as of Gluster.
• Can we make a publicly contributed feature proprietary? No. If anyone contributes to Kadalu Storage, we can't make that feature private.

[bitchecker]

Indirectly related to this because the news comes from TrueNas system (after the main news about the completely moving from CORE to SCALE system)…but I want to share with you this, updating the thread.

TrueNas SCALE was born to have storage system GlusterFS scaleup system with ZFS backend, but as we can read on the documentation :

TrueCommand 3.0 has not passed validation for Clustering and that feature is expected to be highly unstable in this release. With the current unmaintained state of the upstream Gluster project, consider this functionality deprecated. The clustering feature is scheduled for removal in a future TrueCommand revision.

Further, TrueNAS SCALE 24.04 has removed the deprecated gluster backend. Systems installed with SCALE 24.04 (Dragonfish) are unable to use this deprecated TrueCommand feature.

...deprecated gluster backend...

[anon314159]

Indirectly related to this because the news comes from TrueNas system (after the main news about the completely moving from CORE to SCALE system)…but I want to share with you this, updating the thread.

TrueNas SCALE was born to have storage system GlusterFS scaleup system with ZFS backend, but as we can read on the documentation :

TrueCommand 3.0 has not passed validation for Clustering and that feature is expected to be highly unstable in this release. With the current unmaintained state of the upstream Gluster project, consider this functionality deprecated. The clustering feature is scheduled for removal in a future TrueCommand revision.

Further, TrueNAS SCALE 24.04 has removed the deprecated gluster backend. Systems installed with SCALE 24.04 (Dragonfish) are unable to use this deprecated TrueCommand feature.

...deprecated gluster backend...

This is just further confirmation and fuel to the fire. Meanwhile, all of the core devs persist with gaslighting the community.

[aravindavk]

Indirectly related to this because the news comes from TrueNas system (after the main news about the completely moving from CORE to SCALE system)…but I want to share with you this, updating the thread.

TrueNas SCALE was born to have storage system GlusterFS scaleup system with ZFS backend, but as we can read on the documentation :

TrueCommand 3.0 has not passed validation for Clustering and that feature is expected to be highly unstable in this release. With the current unmaintained state of the upstream Gluster project, consider this functionality deprecated. The clustering feature is scheduled for removal in a future TrueCommand revision.

Further, TrueNAS SCALE 24.04 has removed the deprecated gluster backend. Systems installed with SCALE 24.04 (Dragonfish) are unable to use this deprecated TrueCommand feature.

...deprecated gluster backend...

We are aware of this decision by iXsystems and I responded to the similar queries in the Reddit.

In the past, iXsystems sponsored Kadalu Technologies to develop Gluster FS features (https://github.com/gluster/glusterfs/pulls?q=is%3Apr+is%3Aclosed+iXsystems+review%3Aapproved). It is sad to see that the Gluster FS is not fitting there use cases and they are deprecating the Gluster FS integration.

Red Hat ending their Gluster based product or some other company deprecating the products will not make the Gluster Open Source project deprecated. I see the community is still active with a few maintainers from multiple companies. Patches are reviewed, maintainers are participating in mailing list, Slack and Github issues.

We can't say anything about the future, but I see Gluster community is healthy and still solving many real world use cases.

This is just further confirmation and fuel to the fire. Meanwhile, all of the core devs persist with gaslighting the community.

@anon314159 Why are you so interested in calling the Gluster FS project deprecated? Is this project not responding to your queries or is it missing any features that you needed the most or you are using it in production and worried about the future?
or you want to promote any other project? Let us know if you need help with Gluster FS.

[anon314159]

Indirectly related to this because the news comes from TrueNas system (after the main news about the completely moving from CORE to SCALE system)…but I want to share with you this, updating the thread.
TrueNas SCALE was born to have storage system GlusterFS scaleup system with ZFS backend, but as we can read on the documentation :

TrueCommand 3.0 has not passed validation for Clustering and that feature is expected to be highly unstable in this release. With the current unmaintained state of the upstream Gluster project, consider this functionality deprecated. The clustering feature is scheduled for removal in a future TrueCommand revision.

Further, TrueNAS SCALE 24.04 has removed the deprecated gluster backend. Systems installed with SCALE 24.04 (Dragonfish) are unable to use this deprecated TrueCommand feature.

...deprecated gluster backend...

We are aware of this decision by iXsystems and I responded to the similar queries in the Reddit.

In the past, iXsystems sponsored Kadalu Technologies to develop Gluster FS features (https://github.com/gluster/glusterfs/pulls?q=is%3Apr+is%3Aclosed+iXsystems+review%3Aapproved). It is sad to see that the Gluster FS is not fitting there use cases and they are deprecating the Gluster FS integration.

Red Hat ending their Gluster based product or some other company deprecating the products will not make the Gluster Open Source project deprecated. I see the community is still active with a few maintainers from multiple companies. Patches are reviewed, maintainers are participating in mailing list, Slack and Github issues.

We can't say anything about the future, but I see Gluster community is healthy and still solving many real world use cases.

This is just further confirmation and fuel to the fire. Meanwhile, all of the core devs persist with gaslighting the community.

@anon314159 Why are you so interested in calling the Gluster FS project deprecated? Is this project not responding to your queries or is it missing any features that you needed the most or you are using it in production and worried about the future? or you want to promote any other project? Let us know if you need help with Gluster FS.

This has nothing to do with applying false labels to GlusterFS, it's about pragmatism and viability. For additional context my customers and I have a pretty large stake with this SDS project. I have deployed multiple clusters ranging from a few TB's all the way up to 200PB's and have championed this software since 2014. However, there's several very acute yet pervasive issues leading me to believe it's a sinking ship and they are directly related to its stability, usability, and viability. Like everything else in life, there's no such thing as solutions but compromise. I've been using GlusterFS for a decade and determined it to be highly unstable and practically unusable in terms of direct user interaction with the file system. Spurious brick failures when under heavy load and incredibly slow directory traversals when there's a large number of files within them have plagued this project since its inception. For backend purposes GlusterFS is pretty sweet but anything requiring direct user action has been the bane of this project's existence. No product is perfect obviously and there's going to be bugs but some of the pretty severe bugs in this product have persisted for the better part of a decade. Mainly those pertaining to stability and usability. Again, I want GlusterFS to succeed but don't want to place my organization and it's customers at an elevated risk doing so. I intend on using this product for as long as I can even though there's usability issues because of the reed-solomon erasure code implementation and the space efficiency offered by it. Sadly, I really want Gluster to succeed but wanting doesn't make it so in the real world.

[aravindavk]

Great points and I agree that Gluster FS has problems and many things to be improved. But all the issues are existing issues, are you saying Gluster FS is not usable after Red Hat doing EOL of their product?

We don't gain anything with false advertising. People will only use it if it solves their problem. It is Open Source project, companies sponsor feature development or packages only if they see value in the project and not for false advertisement.

Please share if you have better ideas to improve the usability and stability. Or let us know the bugs/issues(Please open the Github issue), I will surely give it a try to fix them (Except big design changes).

[anon314159]

Practically speaking, implementing any major architectural changes that enhance stability and resolve the slow directoy traversal issues are far less likely to every get up streamed since most of the devs pretty much went there own ways. I haven't seen a any roadmaps for GlusterFS 12 and the normal six month minor version updates been pushed to the mainline. Generally speaking, supported projects have continued velocity in terms of updates, addressing bugs, and feature additions. I realize Glusterfs is mature product but there's an awful lot of indicators similar to what happened to the LizardFS project. Money talks and if the core developers aren't getting paid from their parent companies incentive pretty much declines and so does the projects velocity. I know conversation such as these makes folks like ourselves uncomfortable but it doesn't mean they shouldn't happen and the developer silence pretty much speaks for itself.

[bitchecker]

It seems that latest QEMU release has marked as "deprecated" glusterfs backend support!

Here is the link: https://wiki.qemu.org/ChangeLog/9.2

[dlbogdan]

yeah, was looking for reasons to not deploy new gluster setups and now I got it. sad.

[anon314159]

yeah, was looking for reasons to not deploy new gluster setups and now I got it. sad.

Technically speaking, GlusterFS version 12 should have been released already but I haven't seen any active development or feature updates. I am kind of getting that LizardFS vibe right now. Either way, depending on your scalability requirements MooseFS 4.x has feature parody with the current version of GlusterFS and has a pretty active development team. I have been encouraging everyone to conduct research and an in place migration if feasible.

[dbray925]

Unless I'm missing something, only MooseFS Pro has "feature parody" with current versions of GlusterFS. Meaning, you need the paid for Pro version in order to get true master-master availability, right? Otherwise, it is similar to all the others, primary-secondary setup, and no live VM migrations. I'm getting that from the line "New MooseFS Pro version introduces many Master Servers working together in multiple roles." from https://moosefs.com/Content/Downloads/moosefs-installation.pdf

GlusterFS, so far, has been the only easy solution to have this functionality, available "out of the box" and ready to go within minutes. A quick:

sudo dnf -y install glusterfs glusterfs-fuse glusterfs-server glusterfs-ganesha
sudo gluster peer probe $NODE2
sudo gluster peer probe $NODE3
udo gluster volume create $VOL replica 3 arbiter 1 $NODE1:/gluster/$VOL/brick $NODE2:/gluster/$VOL/brick $NODE3:/gluster/$VOL/brick force
sudo gluster volume start $VOL

...and you're done. With that, you have a true multi-master setup, with a "locally" mounted /data volume, ready for your VMs, and ready to live migrate from one host/node to another. Plus, Gluster has libvirtd's built in "protocol='gluster'", which really makes it easy to integrate with KVM. None of the other solutions, Ceph, DRBD, nor MooseFS have this simplicity (nor functionality). Right?

If not, please provide a link to something as straight forward and simple as what GlusterFS has done, because I'm struggling to find a solution as simple as to what GlusterFS provides for KVM live migrations. I'm actively testing: DRBD, Ceph, Moose, and few others from https://en.wikipedia.org/wiki/Comparison_of_distributed_file_systems

[bitchecker]

[...]
GlusterFS, so far, has been the only easy solution to have this functionality, available "out of the box" and ready to go within minutes. A quick:

You're right.

...and you're done. With that, you have a true multi-master setup, with a "locally" mounted /data volume, ready for your VMs, and ready to live migrate from one host/node to another. Plus, Gluster has libvirtd's built in "protocol='gluster'", which really makes it easy to integrate with KVM. None of the other solutions, Ceph, DRBD, nor MooseFS have this simplicity (nor functionality). Right?

As I posted, latest QEMU release, has marked as DEPRECATED the gluster backend.

As I reported at the starting of the discussion, Red Hat has officially marked all the GlusterFS products in EOL. So it seems that GlusterFS will be an orphaned project if nobody will catch the repo and continue to maintain and support officially.

[anon314159]

Unless I'm missing something, only MooseFS Pro has "feature parody" with current versions of GlusterFS. Meaning, you need the paid for Pro version in order to get true master-master availability, right? Otherwise, it is similar to all the others, primary-secondary setup, and no live VM migrations. I'm getting that from the line "New MooseFS Pro version introduces many Master Servers working together in multiple roles." from https://moosefs.com/Content/Downloads/moosefs-installation.pdf

GlusterFS, so far, has been the only easy solution to have this functionality, available "out of the box" and ready to go within minutes. A quick:

sudo dnf -y install glusterfs glusterfs-fuse glusterfs-server glusterfs-ganesha
sudo gluster peer probe $NODE2
sudo gluster peer probe $NODE3
udo gluster volume create $VOL replica 3 arbiter 1 $NODE1:/gluster/$VOL/brick $NODE2:/gluster/$VOL/brick $NODE3:/gluster/$VOL/brick force
sudo gluster volume start $VOL

...and you're done. With that, you have a true multi-master setup, with a "locally" mounted /data volume, ready for your VMs, and ready to live migrate from one host/node to another. Plus, Gluster has libvirtd's built in "protocol='gluster'", which really makes it easy to integrate with KVM. None of the other solutions, Ceph, DRBD, nor MooseFS have this simplicity (nor functionality). Right?

If not, please provide a link to something as straight forward and simple as what GlusterFS has done, because I'm struggling to find a solution as simple as to what GlusterFS provides for KVM live migrations. I'm actively testing: DRBD, Ceph, Moose, and few others from https://en.wikipedia.org/wiki/Comparison_of_distributed_file_systems

Correct regarding the requirement to have an active-active Master setup. However, I have been using MooseFS as a distributed storage backend with xcp-ng and xen orchestra for over a year and have no issues migrating live VM's from host to host. The only caveat is if the master dies, things can stop working fairly quickly and requires intervention from an administrator to migrate the roles to another server. However, migrating live virtual machines is not an issue with MooseFS as it supports locking and sparse files. From my understanding, MooseFS can be used as a hyper-converged distributed file system by which Zen, KVM, Proxmox and other hypervisors. It's a matter of installing and configuring all the required services on each respective virtualization host, mounting those exports again on each host, and then adding that mount point as a datastore using the virtadm console.

While GlusterFs works relatively well as a storage backing for KVM , it has a whole host of issues with metadata performance as a generalized distributed file system. Another interesting thing to consider MooseFS has a native service for hosting block devices called mfsbdev and is relatively easy to set up and tear down. From an experience perspective, I've been using GlusterFS since 2016 and it has been incredibly reliable but also incredibly unstable and fairly underwhelming in terms of performance. The community edition of MooseFS 4.x now supports 4+1 and 8+1 Erasure Codes as an alternative to replicas and I submitted a feature request for the addition of an active-active distributed-replicated Master setup that would overcome some of the single point of failure/performance issues.

[Rush]

It's best to wait and see. GlusterFS still works and it's unlikely to stop working any time soon.

[dbray925]

That is exactly what I'm falling back on. GlusterFS is by far the most simple to setup, and you instantly get master-master live VM migrations. A little extra tweaks for a few MariaDB VMs (aka /var/lib/glusterd/groups/db-workload )...but other than that, rock solid performance, and no complaints.

[anon314159]

That is exactly what I'm falling back on. GlusterFS is by far the most simple to setup, and you instantly get master-master live VM migrations. A little extra tweaks for a few MariaDB VMs (aka /var/lib/glusterd/groups/db-workload )...but other than that, rock solid performance, and no complaints.

I guess it depends on your security posture as once core/system libraries associated with GlusterFS start to diverge, traditional system updates may actually break or introduce bugs with GlusterFS impacting both stability and security. Or worse, maintaining outdated libraries poses a cyber security risk for any systems that are forward-facing or exposed to the internet. But yes, I tend to agree with the philosophy if it ain't broke, don't fix it. In the end it comes down you and your organization's requirements and workloads. GlusterFS is perfectly acceptable and performant for virtualization and database workloads when compared to MooseFS,but if you have to use GlusterFS for general purpose file storage leveraging the native fuse client, NFS, or SMB, MFS absolutely dominates in terms of metadata performance.