How to configure Clustering in DXP Liferay 7.4

                                                                  

Liferay DXP Cluster Communication 

1. Overview

JGroups is a reliable group communication toolkit for Java applications. In Liferay DXP, JGroups manages cluster membership, node discovery, and data replication across cluster nodes. The configuration file shown in this document defines a TCP-based cluster stack, which is the recommended transport for Liferay clusters running on a single machine or across a LAN where UDP multicast is restricted.

The configuration is stored in a file typically named tcp.xml and referenced from Liferay's portal-ext.properties:

• cluster.link.channel.properties.control=tcp.xml
• cluster.link.channel.properties.transport.0=tcp.xml

2. Complete Configuration File

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:org:jgroups"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP bind_addr="IP" <!--update your IP here -->
bind_port="7800"
port_range="20"
recv_buf_size="20m"
send_buf_size="640k"/>
"IP" <!--update your IP here -->

<TCPPING initial_hosts="IP[7800],IP[7900]"
ergonomics="false"/>
<MERGE3 min_interval="10000" max_interval="30000"/>
<FD_ALL timeout="3000" interval="500"/>
<VERIFY_SUSPECT timeout="1500"/>
<BARRIER/>
<pbcast.NAKACK2 use_mcast_xmit="false" discard_delivered_msgs="true"/>
<UNICAST3/>
<pbcast.STABLE desired_avg_gossip="50000" max_bytes="4m"/>
<pbcast.GMS print_local_addr="true" join_timeout="10000"/>
<MFC max_credits="2m" min_threshold="0.4"/>
<FRAG2 frag_size="60k"/>
</config>

3. JGroups Protocol Stack Architecture

JGroups uses a layered protocol stack where each protocol handles a specific concern. Protocols are stacked from bottom (transport) to top (group membership), with each layer adding reliability, ordering, or flow control. The order in the XML file defines the stack

Layer	Protocol	Category	Responsibility
1 — Transport	TCP	Transport	Actual byte transfer over TCP sockets
2 — Discovery	TCPPING	Discovery	Find and list cluster members at startup
3 — Merge	MERGE3	Cluster Mgmt	Reunite split cluster partitions
4 — Failure Detection	FD_ALL	Reliability	Detect crashed/unreachable nodes
5 — Suspect Verification	VERIFY_SUSPECT	Reliability	Double-check suspected failures
6 — State Transfer	BARRIER	Reliability	Pause traffic during state transfer
7 — Reliable Multicast	pbcast.NAKACK2	Reliability	Reliable ordered multicast delivery
8 — Reliable Unicast	UNICAST3	Reliability	Reliable ordered unicast delivery
9 — Stability	pbcast.STABLE	Reliability	Garbage-collect delivered messages
10 — Membership	pbcast.GMS	Cluster Mgmt	Manage join/leave/merge of nodes
11 — Flow Control	MFC	Flow Control	Prevent fast sender from flooding receivers
12 — Fragmentation	FRAG2	Optimization	Split large messages into fragments

4. TCP — Transport Layer

//update IP here

• <TCP bind_addr="IP"
• bind_port="7800"
• port_range="20"
• recv_buf_size="20m"
• send_buf_size="640k"/>

TCP is the foundation of the entire stack. It establishes persistent TCP connections between cluster nodes and handles raw data transmission. Unlike UDP multicast, TCP requires knowing peer addresses in advance (handled by TCPPING).

4.1 bind_addr="IP" //update IP here

• Binds JGroups to the specific network interface with IP .
• Without this, JGroups may bind to localhost (127.0.0.1) or the wrong interface.
• Critical in multi-NIC machines: must point to the LAN interface that other nodes can reach.
• In production, this value is typically set per node and differs between Node 1 and Node 2.

4.2 bind_port="7800"

• The primary TCP port this node listens on for cluster communication.
• Node 1 uses 7800. If this is the same machine (same IP), Node 2 must use a different port — hence 7900.
• Ensure this port is open in the host firewall (e.g., Windows Firewall, iptables).

4.3 port_range="20"

• If port 7800 is unavailable, JGroups tries ports 7800 through 7820 (7800 + 20).
• Provides resilience if another process has temporarily occupied the configured port.
• Important: TCPPING must list all possible ports (or the actual port that binds successfully).

4.4 recv_buf_size="20m"

• Sets the OS-level TCP receive socket buffer to 20 MB.
• A large receive buffer allows the OS to buffer incoming messages even if the application thread is busy.
• Reduces dropped messages under high cluster replication load (e.g., large document uploads, cache invalidations).
• The OS may cap this value — check with: sysctl net.core.rmem_max on Linux.

4.5 send_buf_size="640k"

• Sets the OS-level TCP send socket buffer to 640 KB per connection.
• Controls how much unsent data can be queued before backpressure is applied.
• 640k is a balanced default for a 2-node cluster on a local network.
• Increase for high-throughput clusters or high-latency WAN links.

Attribute	Value	Default	Purpose
bind_addr	Give your computer IP	Any available	Network interface to bind to
bind_port	7800	7800	Primary listen port for cluster traffic
port_range	20	0	Fallback port range if primary is busy
recv_buf_size	20m	OS default	OS TCP receive buffer size
send_buf_size	640k	OS default	OS TCP send buffer size per connection

5. TCPPING — Initial Member Discovery

//update IP=your computer internal IP here with value

• <TCPPING initial_hosts="IP[7800],IP[7900]"
• ergonomics="false"/>

TCPPING is the discovery protocol for TCP-based stacks. Unlike UDP's multicast discovery, TCP cannot auto-discover peers, so you must explicitly list all potential cluster members.

5.1 initial_hosts

Value: IP[7800],IP[7900]

• Lists all nodes that could be cluster members, formatted as IP[port].
• At startup, JGroups sends a PING request to every address in this list.
• IP[7800] → Node 1 (same machine, port 7800).
• IP[7900] → Node 2 (same machine, port 7900).
• In a real multi-machine cluster, these would be different IPs:

• <!-- Multi-machine example update IP value -->
• initial_hosts="IP[7800],IP[7800]"
  
  
• 
  
• 
  
• If a host is unreachable, JGroups skips it and continues — this is not a fatal error.
• The list must include ALL nodes (including the node itself).

5.2 ergonomics="false"

• Disables JGroups' automatic protocol tuning based on detected environment.
• With ergonomics=true, JGroups might auto-adjust timeouts and buffer sizes, overriding your manual settings.
• false ensures your explicit configuration is used exactly as specified — essential for deterministic training/demo environments.

6. MERGE3 — Network Partition Recovery

• <MERGE3 min_interval="10000" max_interval="30000"/>

MERGE3 handles the scenario where a network partition splits the cluster into two sub-groups (split-brain). When connectivity is restored, MERGE3 detects that there are multiple cluster coordinators and merges the sub-groups back into one unified cluster.

6.1 min_interval="10000"

• Minimum time (10 seconds) MERGE3 waits before attempting a merge check.
• Prevents constant merge attempts during transient connectivity issues.

6.2 max_nterval="30000"

• Maximum time (30 seconds) between merge checks.
• MERGE3 picks a random interval between min and max to stagger merge attempts across nodes.
• This jitter prevents all nodes from attempting merge simultaneously (thundering herd problem).

Split-brain scenario example:

• Node 1 and Node 2 lose network connectivity → each forms its own cluster.
• Users on Node 1 see only Node 1's state; users on Node 2 see only Node 2's state.
• Network comes back → MERGE3 detects two coordinators → triggers merge → unified cluster restored.

7. FD_ALL — Failure Detection

<FD_ALL timeout="3000" interval="500"/>

FD_ALL (Failure Detector for All) monitors all cluster members simultaneously. Every node sends a heartbeat to all other nodes. If a node misses heartbeats beyond the timeout, it is suspected of having failed.

7.1 timeout="3000"

• If a node has not sent a heartbeat for 3000 ms (3 seconds), it is considered suspected.
• Too low → false positives (healthy but slow nodes marked as failed).
• Too high → delayed detection of genuinely crashed nodes.
• 3 seconds is appropriate for a local LAN or same-machine cluster.
• 
  

7.2 interval="500"

• FD_ALL sends a heartbeat every 500 ms (0.5 seconds).
• With timeout=3000 and interval=500, a node must miss 6 consecutive heartbeats to be suspected.
• Smaller interval = faster detection but more heartbeat overhead.

timeout	interval	Max missed beats	Detection speed
3000 ms (current)	500 ms (current)	6 beats	3 seconds — good for local LAN
5000 ms	1000 ms	5 beats	5 seconds — relaxed for WAN
1000 ms	200 ms	5 beats	1 second — fast, high overhead

8. VERIFY_SUSPECT — Suspect Confirmation

<VERIFY_SUSPECT timeout="1500"/>

Before FD_ALL's suspicion triggers node removal, VERIFY_SUSPECT makes one final confirmation attempt. This prevents false positives caused by momentary network blips.

• When FD_ALL suspects a node, VERIFY_SUSPECT sends a direct PING to that node.
• Timeout 1500 ms: if no response within 1.5 seconds, the suspicion is confirmed.
• If the node responds, it is cleared from suspicion — FD_ALL was a false positive.
• This two-step approach (FD_ALL → VERIFY_SUSPECT) greatly reduces ghost failures.

Flow: FD_ALL detects no heartbeat → suspects node → VERIFY_SUSPECT pings directly → confirmed? → GMS removes node from cluster.

9. BARRIER — State Transfer Coordination

<BARRIER/>

BARRIER pauses all application-level message traffic during state transfer operations. When a new node joins the cluster, it needs to receive the current state (e.g., Liferay cache contents). During this transfer, incoming messages must be held to ensure state consistency.

• No configuration attributes in this case — uses all defaults.
• Ensures that the new node receives a consistent point-in-time snapshot of cluster state.
• Messages that arrive during state transfer are queued and delivered after transfer completes.
• Essential for Liferay's distributed cache (Infinispan/Ehcache) to work correctly at join time.

10. pbcast.NAKACK2 — Reliable Multicast

• <pbcast.NAKACK2 use_mcast_xmit="false"
• discard_delivered_msgs="true"/>

NAKACK2 (Negative Acknowledgement 2) provides reliable, ordered delivery of multicast (one-to-many) messages. It uses a negative-acknowledgement (NAK) scheme: receivers send NAKs only when they detect gaps, rather than sending ACKs for every message.

10.1 use_mcast_xmit="false"

• Controls whether retransmissions are sent via multicast or unicast.
• false (our setting): Retransmissions are unicasted directly to the requester — correct for TCP-only stacks.
• true: Retransmissions go to all nodes via multicast — useful only for UDP multicast stacks.
• Must be false in a TCP stack because TCP has no native multicast.

10.2 discard_delivered_msgs="true"

• Once a message has been delivered to the application layer, it is removed from the retransmission buffer.
• true: Reduces memory usage — delivered messages are not kept for potential retransmission.
• false: Messages are kept in buffer for possible retransmission to late-joining nodes.
• true is appropriate here since UNICAST3 handles point-to-point reliability and STABLE handles garbage collection.

11. UNICAST3 — Reliable Unicast

• <UNICAST3/>

UNICAST3 ensures reliable, ordered, point-to-point (one-to-one) message delivery between any two nodes. While NAKACK2 handles group (multicast) messages, UNICAST3 handles direct node-to-node communication.

• Uses sliding-window acknowledgements to confirm delivery.
• Automatically retransmits lost messages until confirmed.
• Guarantees FIFO ordering per sender: messages from Node A to Node B always arrive in send order.
• No explicit configuration attributes needed — defaults are well-tuned for LAN clusters.
• In Liferay: used for direct cache invalidation messages between specific nodes.

Protocol	Message Type	Ordering	Use Case
pbcast.NAKACK2	One-to-Many (Multicast)	Total order	Broadcast to all cluster members
UNICAST3	One-to-One (Unicast)	FIFO per sender	Direct node-to-node messages

12. pbcast.STABLE — Message Garbage Collection

• <pbcast.STABLE desired_avg_gossip="50000" max_bytes="4m"/>

STABLE coordinates when old messages can be safely discarded from NAKACK2's retransmission buffer. It ensures all nodes have received a message before it is removed from the buffer.

12.1 desired_avg_gossip="50000"

• Target interval (50 seconds) between STABLE gossip rounds.
• Nodes periodically broadcast their highest received sequence numbers.
• When all nodes agree on the minimum received seqno, messages up to that point are garbage-collected.
• Lower value = more frequent cleanup = less memory, but more gossip traffic.

12.2 max_bytes="4m"

• If the total size of un-garbage-collected messages exceeds 4 MB, triggers an immediate STABLE round.
• Acts as a safety valve to prevent unbounded memory growth during high message throughput.
• Typical for Liferay portals doing heavy cache replication (bulk document uploads, user sessions).

13. pbcast.GMS — Group Membership Service

• <pbcast.GMS print_local_addr="true"
• join_timeout="10000"/>

GMS is the heart of the cluster membership protocol. It manages who is in the cluster, elects a coordinator, processes join/leave requests, and triggers merge operations after network partitions.

13.1 print_local_addr="true"

• When enabled, GMS prints the node's physical address to the NGINX/JGroups log at startup.
• Extremely useful for debugging: confirms which IP and port this node bound to.
• Sample log output:

-------------------------------------------------------------------
GMS: address=IP:7800, cluster=liferay-channel-control
-------------------------------------------------------------------

13.2 join_timeout="10000"

• Maximum time (10 seconds) a node waits for a JOIN response from the cluster coordinator.
• If no response is received within 10 seconds, the node retries or becomes a coordinator itself.
• Too low → nodes give up joining prematurely during coordinator election.
• Too high → cluster startup appears hung when the coordinator is temporarily slow.
• 10 seconds is conservative and appropriate for same-machine development clusters.

GMS Coordinator Role:

• The first node to start becomes the coordinator.
• The coordinator processes all JOIN and LEAVE requests sequentially.
• If the coordinator crashes, GMS automatically elects the next oldest member as the new coordinator.

14. MFC — Multicast Flow Control

<MFC max_credits="2m" min_threshold="0.4"/>

MFC (Multicast Flow Control) prevents a fast sender from overwhelming slower receivers. It uses a credit-based system: senders have a credit balance, and sending messages consumes credits. When credits run out, the sender blocks until receivers replenish them.

14.1 max_credits="2m"

• Each sender starts with 2 MB of send credits.
• Every multicast message sent reduces the credit by its size.
• When credits reach 0, the sender blocks and waits for receivers to grant new credits.
• 2 MB is a reasonable buffer for typical Liferay cluster replication messages.

14.2 min_threshold="0.4"

• Receivers send credit replenishment when remaining credits drop below 40% of max (0.4 x 2m = 0.8 MB).
• Proactive replenishment at 40% prevents the sender from reaching 0 and blocking.
• Lower threshold = more frequent replenishment messages (overhead).
• Higher threshold = replenishment earlier, less chance of sender blocking.

Scenario	Credits Available	Receiver Action	Sender Behavior
Normal	2 MB → 0.8+ MB	No action yet	Sends freely
Threshold reached	0.8 MB (40%)	Sends credit replenishment	Continues sending
Credits exhausted	0 MB	Sending credits	Blocks until credits received

15. FRAG2 — Message Fragmentation

• <FRAG2 frag_size="60k"/>

FRAG2 splits large messages into smaller fragments for transmission and reassembles them at the receiver. This prevents any single large message from monopolizing the network and improves throughput for mixed message sizes.

15.1 frag_size="60k"

• Messages larger than 60 KB are split into 60 KB fragments.
• Each fragment is sent independently over the TCP connection.
• Fragments can be interleaved with other messages, preventing head-of-line blocking.
• FRAG2 reassembles fragments at the receiver before delivering to upper layers.
• 60 KB is a well-tested default; smaller values increase fragmentation overhead, larger values reduce interleaving benefits.

Example: A 180 KB Liferay cache invalidation message is split into 3 fragments of 60 KB each, transmitted, and reassembled transparently.

16. Complete Protocol Reference Table

Protocol	Type	Key Attributes	Purpose for Liferay
TCP	Transport	bind_addr, bind_port, port_range, recv/send_buf_size	Raw TCP socket communication between nodes
TCPPING	Discovery	initial_hosts, ergonomics	Static node list; no UDP multicast needed
MERGE3	Cluster Mgmt	min_interval, max_interval	Recover from network split-brain partitions
FD_ALL	Failure Detection	timeout, interval	Heartbeat monitoring of all cluster members
VERIFY_SUSPECT	Reliability	timeout	Confirm failures before removing from cluster
BARRIER	Coordination	(none)	Pause traffic during new node state transfer
pbcast.NAKACK2	Reliability	use_mcast_xmit, discard_delivered_msgs	Reliable ordered multicast (broadcast) messages
UNICAST3	Reliability	(none)	Reliable ordered unicast (direct) messages
pbcast.STABLE	Reliability	desired_avg_gossip, max_bytes	Garbage-collect delivered messages from buffers
pbcast.GMS	Membership	print_local_addr, join_timeout	Join/leave/merge + coordinator election
MFC	Flow Control	max_credits, min_threshold	Prevent fast sender flooding slow receivers
FRAG2	Optimization	frag_size	Fragment large messages for efficient transfer

17. Two-Node Single-Machine Configuration

This configuration runs both Liferay nodes on a single Windows laptop for training/demo purposes. Node 1 and Node 2 share the same IP (11.168.1.133) but differ by port.

Setting	Node 1	Node 2
bind_addr	IP	IP
bind_port	7800	7900
initial_hosts (both)	IP[7800],IP[7900]	IP[7800],IP[7900]
Liferay HTTP Port	8080	9090
Liferay AJP Port	8009	8010
Liferay Shutdown Port	8005	8006

Note: Both nodes use identical tcp.xml files in this setup because they share the same IP. In a real multi-machine cluster, each node would have a different bind_addr in its tcp.xml.

18. Common Issues & Troubleshooting

18.1 Nodes Not Forming Cluster

• Check: firewall blocking ports 7800 and 7900.
• Check: initial_hosts lists all nodes with correct IPs and ports.
• Check: bind_addr is not 127.0.0.1 — must be the LAN IP.
• Verify GMS log line shows correct address at startup.

18.2 Session Loss After Node Failover

• Check NGINX ip_hash is configured — without it, users get re-routed to a node without their session.
• Check Liferay's distributed session replication is enabled in portal-ext.properties.

18.3 Slow Cluster Join

• join_timeout in GMS may be too low — increase to 15000 or 20000.
• TCPPING initial_hosts may include unreachable nodes — each causes a 10s wait.

18.4 OutOfMemoryError in JGroups

• Reduce max_bytes in STABLE to trigger more frequent garbage collection.
• Increase send_buf_size and recv_buf_size if large messages are being dropped.
• Check MFC credits — sender may be blocked waiting for credit replenishment.