votequorum: make site aware - #801

Can one of the admins check and authorise this run please: https://ci.kronosnet.org/job/corosync/job/corosync-pipeline/job/PR-801/1/input

Thank you for the patch. Quorum related changes are always very risky so it definitively needs approval from more people than just me, ideally at least @chrissie-c (she knows everything about quorum) and @fabbione (who is author or votequorum.c implementation).

Honestly, patch is a bit different than what I've expected from conversation we had on Slack (and just to be clear, that's not wrong) because I was expecting worst case is 50:50. With this patch, situation like 4-node cluster where each node is having different site can happen, so basically there is never inquorate partition - I'm not sure if this is really intended?

From a brief look to source code, it looks like num_sites depends directly on fact site numbers are increasing by 1 and never skipped - if it is so, it should be enforced. Also it looks like runtime reconfiguration (reload of config file) is not yet taken into account (what happens/should happen if user changes site?).

Anyway, I will convert this to draft so CI is not running all the time, because I feel this will need quite some work before merge.

Yes that is basically intentional that even a site with let's say 2 nodes can stay quorate when loosing connection to a 2nd site that has e.g. 3 nodes. I think that we would not need to limit to a symmetric split of the nodes over sites. And as long as a site is complete it is useful and healthy to some extent so it should be enabled to fence the invisible site. Of course we need similar measures as with two-node that this fencing race doesn't result in all nodes fenced (priority fencing, different wait, random wait or combinations of those).

And as already stated above this isn't meant to be merged as is. I was just a little bit surprised to have something simple working quite quickly and wanted to share for further discussion and to have something to have a hand on to see how suitable it would be for the applications in mind.
Not taking care of all the nasty details like dynamic config changes (explicitly that I was aware - sry for not mentioning) was intentional as well to keep it as simple as possible for the first for visibility of the basic principle and quick modifications.

Guess it is fine to not have further and recurring CI runs on that. Just wanted to have CI run on it once to see if it breaks anything basic. What I had tested so far was just the new feature and not a regression test on the existing functionality.

The idea that each site can have separate quorum breaks all the rules around quorum. There is ONLY ONE quorum in any given cluster/partition of nodes within a cluster (other than 2 nodes, protected by other methods).

If two separate quorums are needed, then there should be two separate clusters and services managed via booth or similar.

The idea that each site can have separate quorum breaks all the rules around quorum. There is ONLY ONE quorum in any given cluster/partition of nodes within a cluster (other than 2 nodes, protected by other methods).

If two separate quorums are needed, then there should be two separate clusters and services managed via booth or similar.

Compare it to two-node. Regaining quorum is limited > 1/2 of the nodes as usually.
I guess 2 separate quorums would unnecessarily limit to 2 sites where I don't actually see a need for.
One point I have forgotten to make above is that SBD needs rethinking - at least watchdog-fencing - well as we have it with two-node quorum isn't usable anymore.

The idea that each site can have separate quorum breaks all the rules around quorum. There is ONLY ONE quorum in any given cluster/partition of nodes within a cluster (other than 2 nodes, protected by other methods).
If two separate quorums are needed, then there should be two separate clusters and services managed via booth or similar.

Compare it to two-node. Regaining quorum is limited > 1/2 of the nodes as usually. I guess 2 separate quorums would unnecessarily limit to 2 sites where I don't actually see a need for. One point I have forgotten to make above is that SBD needs rethinking - at least watchdog-fencing - well as we have it with two-node quorum isn't usable anymore.

Thinking over it I take back that going for more than 2 sites is a good idea.
It is probably dangerous and the config should limit to just 2 sites.
And on top for more than 2 sites the concept isn't needed anyway as main idea is to be able to cope with outage/separation of a site while fencing is still working (SBD disk for instance).
Forgotten to mention fence-heuristics (e.g. fence_heuristics_ping) as an additional means to favor the site that is still connected to the rest of the world.

ad 2nd quorum)

Don't know if you meant it that way but kind of a 2nd quorum or flag that quorum is retain state would be useful.

I don't think it would actually be needed for pacemaker decisions - record of resource state and fencing should take care of that - but for SBD in poison-pill mode. Currently SBD is counting the number of nodes registered for the cpg-protocol in case of two-node when it is about surviving the loss of a single disk (otherwise the disk becomes a spof or if we use quorum instead we create a split-brain). This is rather clumsy and it would be nicer if SBD wouldn't have to care about this kind of corosync internal stuff but have a flag it can use in all cases - no checking for two-node, no switching between quorum and counting cpg.

Given the above the feature should probably be rather called something like 'two-site'.
And maybe we could spawn out a discussion about something like an interface extension of corosync to be able to know if we have retained quorum state - could be an additional flag or a 3rd state for quorum.

Quorum is not a site property, quorum is a CLUSTER property. If you split a cluster - you have two clusters, not two parts of one cluster. Two parts of one cluster is a split brain. This a very common misconception of quorum, you're not the first one to make it.

You're using the old quorum state to decide whether a 'site' is quorate or not, so if you have a 4 node quorate cluster that gets split into two equal parts, are they both quorate? whoops, yes they are! I tested this - it is what happens.

The problem is that neither site knows the state of the other site, that's why quorum is a whole cluster property and why qdevice is needed to resolve these things. qdevice can see all of the parts of the cluster so it knows which side has which nodes active and which are dead (That's not strictly true but the combination of totem/knet/fencing and qdevice makes it reliable).

It's an interesting idea, but fundamentally flawed I'm afraid.

ad 2nd quorum)

Don't know if you meant it that way but kind of a 2nd quorum or flag that quorum is retain state would be useful.

I don't think it would actually be needed for pacemaker decisions - record of resource state and fencing should take care of that - but for SBD in poison-pill mode. Currently SBD is counting the number of nodes registered for the cpg-protocol in case of two-node when it is about surviving the loss of a single disk (otherwise the disk becomes a spof or if we use quorum instead we create a split-brain). This is rather clumsy and it would be nicer if SBD wouldn't have to care about this kind of corosync internal stuff but have a flag it can use in all cases - no checking for two-node, no switching between quorum and counting cpg.

Reacting just on this. If you are already counting cpg/quorum, wouldn't it be way to use qdevice api and make sbd act as qdevice provider? This would mean sbd can choose what partition is quorate, handle sites and keeping corosync quorum safe (no chance to have two quorate partitions) same time?