Proxmox Cluster Network: Properly Configure Corosync

A stable Proxmox® cluster requires precise Corosync configuration for reliable cluster communication. This guide shows you how to optimally set up your Proxmox® cluster network and correctly configure Corosync. The usual process for system administrators with basic Proxmox® VE knowledge is correctly set up by the automatic installer or in the Proxmox VE frontend, requiring no manual work.

We explicitly advise against working on the configuration without deep knowledge of Corosync’s intricacies and the consequences of custom settings. This should be left to specialists with expertise. However, this overview may be helpful for experimentation and further education.

You need at least two Proxmox® nodes and a Q-Device, or three Proxmox nodes, administrative access to all cluster nodes, redundant network connections, and basic Linux® command-line knowledge. After successful configuration, you will have a highly available Proxmox® cluster with optimal failover functionality.

Why Corosync is Critical for Proxmox® Clusters

Corosync forms the core of cluster communication in Proxmox® VE environments. This software ensures that all nodes in the cluster continuously communicate with each other and are aware of the current status of the entire infrastructure.

Split-brain prevention is one of Corosync’s most important functions. Without functioning cluster communication, individual nodes could mistakenly assume they are the only active systems. This leads to data inconsistencies and failures of critical services. In the worst case, data loss is imminent.

Corosync continuously monitors the accessibility of all cluster members. In the event of network problems or node failures, the system initiates automatic failover processes. Virtual machines and containers are seamlessly migrated to functioning nodes.

The reliability of your entire Proxmox® cluster directly depends on the Corosync configuration. Incorrect settings can lead to unstable clusters, unpredictable failovers, or complete system failures.

Check Proxmox® Cluster Network Requirements

First, check your hardware requirements for stable cluster communication. Each node requires at least two physical network interfaces for redundant connections.

Validate network infrastructure:

1. Test latency between all nodes using the ping command.
2. Check available bandwidth with iperf3.
3. Verify switch configurations for VLAN settings.
4. Document all IP addresses for cluster communication.

IP addressing must be carefully planned. Use separate subnets for primary and secondary Corosync communication, for example, 10.1.1.0/24 for Ring 0 and 10.1.2.0/24 for Ring 1.

Bandwidth requirements vary depending on cluster size. For small clusters with 3–5 nodes, 100 Mbit/s per ring is sufficient. Larger installations with frequent VM migrations require Gigabit connections.

Warning: Never use the same network segments for Corosync and productive VM communication. This can lead to performance problems and cluster instability. Corosync should always have its own network segment and interfaces.

Set Up Corosync Ring Configuration

The Corosync rings are configured via the file /etc/pve/corosync.conf. Normally, this file is created and maintained by the frontend or the installer. As already mentioned, manual editing is not necessary. First, create a backup copy of the existing configuration.

Configure Primary Ring (Ring 0):

1. Open the Corosync configuration: vi /etc/pve/corosync.conf
2. Define the nodelist with all cluster members.
3. Assign a unique ID to each node.
4. Configure the IP addresses for Ring 0.

Add a secondary ring (Ring 1) for redundancy. Edit the nodelist and add ring1_addr parameters for each node. These addresses must be in a separate network segment.

The Totem section controls the behavior of cluster communication. Adjust the token timeout values to your network latency. Low values improve failover speed but can lead to false alarms with slow connections.

Success Check: After configuration, restart the Corosync service: systemctl restart corosync. Check the status with corosync-cmapctl and review the logs for error messages.

Multicast vs. Unicast: Making the Right Choice

The choice between Multicast and Unicast used to significantly influence the performance and compatibility of your Corosync cluster. Meanwhile, with Corosync 3, KNET has become prevalent, and both UDPU as a Unicast connection and Multicast with UDP are classified as “Deprecated.” Multicast is now explicitly discouraged.

Test and Validate Cluster Communication

Comprehensive testing of Corosync functionality is crucial for cluster stability. Start with basic connectivity checks between all nodes.

Systematically use diagnostic tools:

1. Check cluster status: pvecm status
2. Analyze Corosync membership: corosync-quorumtool -l
3. Check ring status: corosync-cfgtool -s
4. Check cluster logs: journalctl -u corosync

Failover tests simulate real failure scenarios. Disconnect individual nodes from the network one by one and observe cluster behavior. Virtual machines should automatically migrate to functioning nodes.

Test split-brain scenarios by simultaneously disconnecting multiple nodes. The cluster must correctly decide which partition remains active. Nodes without quorum must not perform critical operations.

Performance monitoring over longer periods reveals intermittent problems. Monitor latency spikes, packet loss, and timeout events. These metrics indicate potential weaknesses in the network infrastructure.

How credativ® Helps with Proxmox® Cluster Configuration

credativ® offers comprehensive support for the professional implementation and optimization of Proxmox® clusters. Our specialists have many years of experience with complex high-availability infrastructures.

Our Proxmox services include:

• Detailed analysis of your existing network infrastructure
• Tailored Corosync configuration for your requirements, as needed.
• Proactive checking and maintenance of your cluster environment
• 24/7 support for critical cluster issues
• Regular performance optimization and updates

Our Open Source Support Center ensures that your Proxmox® installation always functions optimally. We offer direct access to Linux® specialists without an intermediary call center.

Contact us for individual consultation regarding your Proxmox® cluster configuration. Our experts will work with you to develop a customized solution for maximum reliability and performance.

Transparency notice

Proxmox® is a registered trademark of Proxmox Server Solutions GmbH. credativ® is an authorized reseller of Proxmox®. Linux® is a registered trademark of Linus Torvalds.

The mention of trademarks serves solely for the factual description of migration scenarios and services provided by credativ®. There is no business relationship with the mentioned trademark owners.

About the author

Peter Dreuw

Head of Sales & Marketing

about the person

Peter Dreuw has been working for credativ GmbH since 2016 and has been a team lead since 2017. Since 2021, he has been part of Instaclustr’s management team as VP Services. Following the acquisition by NetApp, his new role became “Senior Manager Open Source Professional Services”. As part of the spin-off, he became a member of the executive management as an authorized signatory. His responsibilities include leading sales and marketing. He has been a Linux user from the very beginning and has been running Linux systems since kernel 0.97. Despite extensive experience in operations, he is a passionate software developer and is also well versed in hardware-near systems.

View posts