Proxmox 8.1 released

Proxmox Virtual Environment, die führende Open-Source-Plattform für Servervirtualisierung und Enterprise-Cloud, hat die Version 8.1 veröffentlicht. Diese Version bringt viele neue Funktionen und Verbesserungen für die Verwaltung von virtuellen Maschinen, Containern, Speicher, Netzwerk und Cluster.

• Kompatibilität mit Secure Boot: Proxmox VE ist jetzt mit Secure Boot kompatibel, einer Sicherheitsfunktion, die den Boot-Vorgang eines Computers schützt. Secure Boot stellt sicher, dass nur Software mit einer gültigen digitalen Signatur gestartet wird. Proxmox VE beinhaltet jetzt einen signierten Shim-Bootloader, der von den meisten UEFI-Implementierungen der Hardware akzeptiert wird. Das ermöglicht die Installation von Proxmox VE in Umgebungen mit aktiviertem Secure Boot.

• Software-Defined Networking (SDN): Proxmox VE enthält jetzt standardmäßig die Kernpakete für Software-Defined Networking (SDN), einer Technologie, die eine dynamische und automatisierte Konfiguration von Netzwerkressourcen ermöglicht. Mit SDN können Netzadministratoren virtuelle Zonen und Netzwerke (VNets) erstellen und verwalten, und damit komplexe Netzwerkkonfigurationen und Multitenancy-Setups direkt über das Webinterface auf „Datacenter“-Ebene steuern. Die Anwendungsmöglichkeiten für SDN sind vielfältig, von einem isolierten privaten Netzwerk auf jedem einzelnen Knoten bis hin zu komplexen Overlay-Netzwerken über mehrere Proxmox VE-Cluster an verschiedenen Standorten. Die Vorteile sind eine schnellere und anpassungsfähigere Netzwerkinfrastruktur, die nach den Geschäftsanforderungen skaliert werden kann.

• Neues flexibles Benachrichtigungssystem: Proxmox VE bietet jetzt ein neues Framework, das die Weiterleitung von Benachrichtigungen steuert. Es erlaubt die Auswahl verschiedener Ziele als Empfänger von Benachrichtigungen. Neben dem bisher üblichen lokalen Postfix MTA können jetzt auch Benachrichtigungen an einen Gotify-Server oder eine E-Mail direkt über einen SMTP-Server, inkl. SMTP-Authentifizierung gesendet werden. Mit vorher definierten Regeln bestimmen die Matcher, welche Ziele welche Benachrichtigungen für bestimmte Ereignisse erhalten. Das neue Benachrichtigungssystem ist sehr flexibel und ermöglicht eine genaue Konfiguration, wann, wo und wie die Benachrichtigungen gesendet werden.

• Unterstützung für Ceph Reef und Ceph Quincy: Proxmox VE unterstützt Ceph Reef 18.2.0 sowie auch weiterhin Ceph Quincy 17.2.7. Die bevorzugte Ceph-Version kann während des Installationsprozesses ausgewählt werden. Ceph Reef bietet bessere Standard-Einstellungen, die die Performance verbessern und die Lesegeschwindigkeit erhöhen.
•  

Bei der Umsetzung Ihres Proxmox Projekts sind wir Ihnen gerne behilflich.

Das ausführliche Changelog finden Sie hier.

Enhancements in the web interface (GUI)

• Improvements to bulk actions:
  • Add a new "Bulk Suspend" action to suspend a selection of guests in one action.
  • Add a new section above the guest list for configuring guest filters and add a button for clearing filters.
  • Allow to filter guests by their tags.
  • Reorder fields and drop obvious warning about local storage to improve on screen-space utilization.
  • Reword the message displayed for bulk actions in the task log to "Bulk Start/Stop/Migrate". The message shown previously was "Start/Stop/Migrate all" and could be misleading in case not all guests were affected (issue 2336).
  • The "Bulk Migrate" action is now hidden on standalone nodes, as there is no valid migration target in that case.
• Improvements to the node summary panel:
  • The summary now indicates whether the node was booted in legacy (BIOS) mode, EFI mode, or EFI mode with Secure Boot enabled.
  • The currently running kernel is now reported more compactly by indicating only the version and the build date.
• Allow to automatically decompress an ISO file when downloading from a URL to a storage. The decompression algorithm can be set in the GUI (issue 4849).
• Allow moving VMs and containers from one pool to a different pool in one operation.
• Avoid needlessly reloading the GUI after ordering a certificate via ACME for a different cluster node.
• The permission editor now also shows the ACL paths for notifications and PCI/USB mappings.
• The resource tree now displays the usage in percent when hovering over a storage.
• If the configured tree shape for tags is not "Full", the resource tree now displays a tooltip with the tag name when hovering over the configured shape.
• Ensure the SPICE config is downloaded with the correct file extension on Safari to avoid problems with macOS application association (issue 4947)
• Fix an issue where the "Migrate" button stayed disabled even if selecting a valid target node.
• Fix a bug where the backup job editor window would add an invalid entry for VMID 0 when using selection mode "all".
• Improve error message when creating a VM with an invalid hostname: Clarify that a valid hostname, not a valid DNS name, is expected (issue 4874).
• When uploading a subscription, ignore surrounding whitespace in the subscription key to avoid confusing errors.
• Improve the focus handling when editing tags to allow tabbing though editable tag fields.
• Allow adding tags already when creating VMs and containers.
• Increase height of the VM and container creation wizard to obtain a 4:3 ratio.
• When creating an IP/CIDR inside an IPSet, the GUI now requires that an IP/CIDR is provided. Previously, the GUI accepted an empty field, but the API threw an error.
• Update external links to proxmox.com that changed during the website redesign.
• Fix an issue where the OK button would stay disabled when editing an ACME DNS challenge plugin (issue 4531).
• Fix an issue where clicking "Reset" in the zpool creation window could cause an error when re-opening the window (issue 4951).
• Fix an issue where users could write notes with links containing JavaScript code. This JavaScript code would be executed when a (different) user clicked on such a link.
• HTML-encode API results before rendering as additional hardening against XSS.
• Improved translations, among others:
  • Croatian (NEW!)
  • Georgian (NEW!)
  • Arabic
  • Catalan
  • German
  • Italian
  • Polish
  • Simplified Chinese
  • Traditional Chinese
  • Ukrainian
  • Several remaining occurrences of the GiB unit in the GUI can now be translated (issue 4551).

Virtual machines (KVM/QEMU)

• New QEMU version 8.1.

  See the upstream changelog for details.

• Add clipboard support to the VNC console in the GUI. For now the feature cannot be enabled in the GUI and has to be manually enabled via API or CLI. After installing the SPICE guest tools, text can be copied from/to the guest clipboard using the noVNC clipboard button.
• When creating a Windows VM, allow adding a second CD/DVD drive directly in the creation wizard.

  This makes it easier to add the Windows VirtIO drivers ISO before starting the VM.

• Remove the 10-minute timeout for allocating VM disks when restoring from backup, as this timeout may be exceeded if disks are large or network storage is involved (issue 2817).
• Log a warning when starting a VM with a deprecated machine version.
• Fix issues where shutdown and reboot commands would time out (instead of failing immediately) on ACPI-suspended VMs.
• Enabling or disabling CPU hotplug for a running VM did not work reliably and is not allowed anymore.
• Avoid leaving potentially large amounts of memory assigned to the QEMU process after backup.
• Fix an issue where heavy network traffic or connection issues during a backup to Proxmox Backup Server could cause an unsolicited write to the first sector of a backed-up SATA disk, which usually contains the boot-sector (issue 2874).
• Fix an issue where a race condition could cause a VM crash during backup if iothread is enabled.
• Fix an issue where each pause and resume operation (for example when taking a snapshot) would increase the number of open file descriptors of the QEMU process, which could eventually lead to crashes.
• Fix an issue where starting a VM with machine type q35 and multiple IDE drives would fail.
• cloud-init: Fix issues where non-root users could not regenerate the cloud-init drive or set the ciupgrade option.
• Start VMs using PCI passthrough with a higher timeout that is calculated from the configured memory. Previously, the timeout was reported to be too short when using PCI passthrough.
• Fix an issue where qmeventd failed to retrieve VMIDs from processes on hybrid cgroup systems and logged errors to the journal.
• Fix an issue where remote migration would fail for certain combinations of source/target storage, for example from qcow2 on directory to LVM-thin.
• Fix an issue where backup of a VM template with a TPM would fail (issue 3963).
• Fix an issue where the VNC proxy would fail if the LC_PVE_TICKET was not set (issue 4522).
• Backports of several upstream kernel patches:
  • Fix an issue where VMs with a restricted CPU type could get stuck after live-migration from a host with kernel 5.15 to a host with kernel 6.2.
  • Fix an issue where VMs could get stuck after several days of uptime if KSM, ballooning, or both, were enabled.
  • The FLUSHBYASID flag is now exposed to nested VMs when running on an AMD CPU. This fixes an issue where some hypervisors running in a VM would fail to start nested VMs.
  • Fix an issue with recovering potential NX huge pages that resulted in a warning logged to the journal (issue 4833).
  • Fix an issue where only one NVMe device would be recognized even though multiple are present (issue 4770).

Containers (LXC)

• Support device passthrough for containers. The new dev0/dev1/... options take the path of host device. Optionally, owner and permission settings for the device node inside the container can be given. For now, the option cannot be set in the GUI and has to be manually set via API or CLI.
• Allow specifying multiple SSH keys in the container creation wizard (issue 4758).
• Show privileged status as a separate row in the guest status view in the GUI.
• Show distribution logo and name in the guest status view in the GUI.
• Fix an issue where network would fail to come up for Fedora containers.
• Add an API endpoint /nodes/{node}/lxc/{vmid}/interfaces for querying network interfaces of a running container.
• Improve architecture detection for NixOS containers, which would previously produce a warning and default to x86_64 in case /bin/sh did not (yet) exist in the container.
• The pct status command does not report guest CPU usage anymore, as there is currently no fast way to measure it (issue 4765).
• Restoring a container from a PBS backup now honors the ignore-unpack-errors flag (issue 3460).
• Fix an issue where Fedora containers would not have a container-getty on first boot.

General improvements for virtual guests

• Show progress of offline disk migration in the migration task log by use of dd's status=progress argument (issue 3004).
• Proxmox VE now has an officially assigned OUI from the IEEE BC:24:11 to be used as product-specific MAC prefix. This is now used by default instead of assigning purely random MACs. (issue 4764).

HA Manager

• Notification for HA events, like fencing, are now configurable via the new modular notification system.
• An issue with the target selection during service recovery, where a fenced node was selected as target, was fixed (issue 4984).

Improved management for Proxmox VE clusters

• New flexible notification system.

  Allows sending notifications to different targets. The local Postfix MTA, previously the sole notification option, is now one of several target types available.

  Two new target types include: smtp allowing direct notification emails via authenticated SMTP, and gotify, which sends notifications to a Gotify instance.

  Flexible notification routing is possible through matcher-based rules that determine which targets receive notifications for specific events.

  Match rules can select events based on their severity, time of occurrence, or event-specific metadata fields (such as the event type). Multiple rules can be combined to implement more complex routing scenarios.

• Name resolution to find an IP for a node's name now looks at all IPs associated with the name, only excluding loopback addresses. Additionally, a better warning is logged in case of a failed lookup.
• pvecm updatecerts, which is used to ensure a consistent state of the certificates in a cluster, was reworked for increased robustness:

  Files not being stored in the cluster filesystem are now created earlier.

  The call now waits for the node to join the quorate partition of the cluster instead of failing. This is especially helpful during node-bootup, when running before starting pveproxy.service.

  The error message in case the call fails due to missing quorum was reworded for better clarity.

• The MAC addresses of the guests under SDN are now cached in the cluster filesystem for improved lookup speed in a cluster.

Backup/Restore

• Backups and backup jobs can now be configured with a notification mode for a smooth migration to the new notification system.

  The legacy-sendmail mode replicates the previous behavior of sending an email via the local Postfix MTA if an email is configured.

  The notification-system mode sends notifications exclusively using the new notification system.

  The default mode auto behaves like legacy-sendmail if an email address is configured, and like notification-system if no email address is configured.

  Thus, existing backup jobs without a configured email address will default to sending notification emails to the root user after the upgrade to Proxmox VE 8.1.

• Allow setting the pbs-entries-max parameter. In order to prevent failing container backups with a huge number of files in a directory, it can help to set it to a higher value than the default (issue 3069).
• Improvements to the vma CLI tool that handles VMA backup files:
  • The vma extract command now optionally takes a filter to only extract specific disks from the backup (issue 1534).
  • Fix an issue where the vma create command could not write to tmpfs (issue 4710).
• Improvements to file restore:
  • Fix an issue where the settings for ZFC ARC minimum and maximum were not properly set for the temporary file-restore VM.
  • Fix an issue where debug log messages were not printed even though the PBS_QEMU_DEBUG environment variable was set.
• Fix an issue with backups of diskless VMs to Proxmox Backup Server: Even though encryption was enabled, such backups would not be encrypted. Since the backup contained no disks, this did not reveal any VM data, but the VM configuration was stored in plaintext (issue 4822).
• File restore now allows downloading .tar.zst archives as an alternative to .zip archives.
• Improved handling of backups with master key:
  • Abort the backup if the the running QEMU binary does not support master keys, instead of just printing a warning. Master keys are supported in QEMU builds of Proxmox VE since version 6.4.
  • If no encryption key is configured, the backup task will explicitly warn that the backup will be unencrypted.
  • The backup log now prints only one message that encryption is enabled, instead of previously two messages.

Storage

• Improvements to the iSCSI storage backend:
  • Try to log into all discovered portals for a target, instead of just the single portal initially configured for the storage. This way, the storage can now become available in a multipath setup if at least one portal is online, even if the single configured portal is offline.
  • The backend is now usable immediately after installing Open-iSCSI. Previously, some services needed to be restarted first.
• Fix an issue where a replication job could not be run or deleted if it referred to a storage that does not exist anymore.
• SMB/CIFS: Fix connection check in case an empty domain is provided.
• The BTRFS plugin received a fix for creating base templates when falling back to the standard directory variant.

Ceph

• Support installing Ceph 18.2 Reef and make it the default release for new setups.
• Allow creating multiple OSDs per physical device via API and CLI, and display such setups properly in the GUI. Multiple OSDs on one device can be useful when using fast NVMe drives that would be bottle-necked by a single OSD service (issue 4631).
• When creating a pool, read the default values for size/min_size from the Ceph configuration instead of using hard-coded default values 3/2 (issue 2515).

  There are use cases where different values for size/min_size make sense, for example 4/2 if a cluster spans two rooms.

• The pveceph install commands now asks the user to confirm the Ceph version to be installed (issue 4364).
• Improve discoverability of Ceph warnings by providing a tabular view and a button to copy warning details.
• Report OSD memory usage more accurately by using the Proportional Set Size (PSS) of the OSD process. Previously, memory usage was read from the OSD service and thus included the page cache, leading to extremely high values shown in the GUI.
• Use snake_case when setting options in Ceph config files to ensure consistency within that file (issue 4808).
• Mark global pg_bits setting as deprecated and make it a no-op. The setting has been deprecated since Ceph 13.
• Improve reporting of cluster health:
  • Replace "Error" category for PG states with "Warning" and "Critical" categories to allow more fine-grained assessment of the cluster state.
  • Rename "Working" state to "Busy" state to better convey its meaning.

Access control

• Support nested pools up to a nesting depth of 3 levels for greater flexibility in structuring VMs and containers (issue 1148).

  Pool names can now contain at most two slashes (allowing to structure them as parent/child/grandchild).

  Permissions are inherited along the path according to the usual inheritance rules.

• Improvements to LDAP/AD realms:
  • When adding/updating an LDAP/AD realm, there is now the option to directly check if the bind works, instead of having to wait for the first sync. This check is enabled by default in the GUI and can be disabled in the advanced options if needed.
  • Forbid specifying a Bind DN without a password in the GUI, which is already forbidden by the API.
  • Expose the mode option in the GUI that allows switching between LDAP, LDAPS and LDAP via STARTTLS. This option was already supported by the backend and succeeds the secure option which allowed switching between LDAP and LDAPS only.
  • Fix an issue with enforced TFA where certain sync settings would cause the TFA restriction to not be enforced.
  • It is now possible to update only the password field for the bind-user of an LDAP realm, this failed previously.
  • Allow setting the case-sensitive option of AD realms, which was previously only editable via CLI, in the GUI.
• Existing-but-disabled TFA factors can no longer circumvent realm-mandated TFA.
• The list of SDN-related ACL paths now reflects all SDN objects, ensuring that there are no settings which remain root@pam only.
• A mistyped entry of pools instead of pool in the default set of ACL paths was fixed.

Firewall & Software-Defined Networking

• The core of Proxmox VE's Software-Defined Network stack has been lifted from experimental to supported.
• New DHCP plugin for SDN (tech preview).

  Enabling DHCP for a zone will start a DHCP server that can automatically assign IP addresses to associated virtual guests (VMs and containers).

  Currently, only Simple zones are supported, and dnsmasq is the only supported DHCP server.

  Each subnet of a Simple zone can now be configured with DHCP ranges.

  When a virtual guest associated to the zone starts, the DHCP plugin queries the zone's IPAM for an IP address and offers it to the virtual guest.

  If the built-in Proxmox VE IPAM is used, active DHCP leases can be viewed and edited conveniently on the web UI.

• IS-IS was added as a further SDN controller, next to EVPN and BGP.
• The interfaces section of the frr is now parsed in order to support multiple underlay networks (like IS-IS).
• MAC learning on SDN bridges can now selectively be disabled for individual plugins. This is implemented for the EVP plugin.
• A warning is logged if the main network configuration (/etc/network/interfaces) does not source the SDN controlled configuration (/etc/network/interfaces.d/sdn), because the SDN configuration would be ignored in that case.
• The error reporting for problems with vnet generation was improved, by pointing to the relevant task log.
• The firewall log can now be also displayed for a specific timespan instead of showing the live-view (issue 4442).
• Fix an issue where scoped alias resolution would fail with an error.
• Enabling VLAN-awareness for an EVPN zone is unsupported and now fails instead of just printing a warning (issue 4917).
• Fix an issue where an empty subnet could not be deleted if it has a gateway defined.
• The IPAM selector, which is a required choice, is not hidden behind the Advanced checkbox in the UI anymore.
• The identifying CIDR for a vnet is now named Subnet to improve clarity.
• A systemd.link(5) configuration is now shipped to both keep bridges up even if there's no port connected, and to prevent that a random MAC-address is assigned to bridges or bond interfaces.
• A ethtool is now a hard dependency of ifupdown2, matching the common need of disabling offloading features of certain NICs.
• Prevent a crash in ifupdown2 caused by an error in a third-party plugin in /etc/network/ifup.d/.
• The accept_ra and autoconf sysctl settings are now also applied for bridge interfaces.
• ifupdown2 now correctly recognizes when remote IPs for vxlan are configured by external sources and does not remove them on reconfiguration.

Improved management of Proxmox VE nodes

• Secure Boot support.

  Proxmox VE now ships a shim bootloader signed by a CA trusted by most hardware's UEFI implementation. In addition, it ships variants of the GRUB bootloader, MOK utilities and kernel images signed by Proxmox and trusted by the shim bootloader.

  New installation will support Secure Boot out of the box if it is enabled.

  Existing installations can be adapted to Secure Boot by installing optional packages, and possibly reformatting and re-initializing the ESP(s), without the need for a complete reinstallation. See the wiki article for more details.

• The kernel shipped by Proxmox is shared for all products. This is now reflected in the renaming from pve-kernel and pve-headers to proxmox-kernel and proxmox-headers respectively in all relevant packages.
• The new proxmox-default-kernel and proxmox-default-headers meta-packages will depend on the currently recommended kernel-series.
  • Avoid logging benign but confusing warnings about a segfault in pverados.
• Many edge-cases encountered during the upgrade from PVE 7.4 to 8 by our user-base are now detected and warned about in the improved pve7to8 checks:
  • Warn if DKMS modules are detected, as many of them do not upgrade smoothly to the newer kernel versions in PVE 8.
  • Warn if the PVE 7 system does not have the correct meta-package of grub installed ensures to actually upgrade the installed bootloader to the newest version.
  • The check for old cgroupv1 containers was adapted to not cause false positives on current containers (for example Fedora 38).
• Support for adding custom ACME enabled CA's which require authentication through External Account Binding (EAB) on the commandline (issue 4497).
• Using the Console/Shell on a PVE node is now possible for all users with the appropriate permissions (Sys.Console). The restriction to the pam was removed. The users will still need to login as a system user on the shell though.
• With the Proxmox repositories having support for fetching them directly the changelogs for new package versions shown in the UI are now all gathered with apt changelog.
• The pvesh debug tool now also supports yielding output for streaming API calls, like for example the syslog.
• The documentation on firmware updates provided by the operating system has been extended and revised, helping administrators to identify if their setup is optimal.

Installation ISO

• The ISO is able to run on Secure Boot enabled machines.
• The text-based UI got significant improvement based on the feedback received from the first release in PVE 8.0.
• The current link-state of each network interface is now displayed in the network configuration view, helping in identifying the correct NIC for the management interface (issue 4869).
• If provided by the DHCP server, the hostname field is already filled out with the information from the lease.
• The arc_max parameter for installations on ZFS can now be set in the Advanced Options. If not explicitly set by the user, it is set to a value targeting 10% of system memory instead of 50%, which is a better fit for a virtualization workload (issue 4829).
• The correct meta-package of grub is now installed based on the boot mode (grub-pc or grub-efi-amd64). This ensures that the bootloader on disk gets updated when there is an upgrade for the grub package.
• The text-based UI is now also available over a serial console, for headless systems with a serial port.
• /var/lib/vz backing the local storage is now created as separate dataset for installations on ZFS (issue 1410).
• The root dataset on ZFS installations now uses acltype=posixacl in line with upstream's recommendation.
• Kernel parameters passed on the commandline during install are now also set in the target system (issue 4747).
• Fix the warning that is shown in case the address family (IPv4, IPv6) of the host IP and DNS server do not match.
• The text-based UI now sets the correct disk-size for the selected disk, instead of limiting the installation to the size of the first disk in the list (issue 4856).
• For better UX, the text-based UI now also displays a count-down before automatically rebooting.
• The screensaver in the graphical installer is now disabled.
• The graphical installer now displays the units used for disk-based options.
• The kernel commandline parameter vga788 is now set for both the graphical debug and all text-based UI installation options. This improves compatibility of the installer with certain hardware combinations.

Other Notable changes

• Existing backup jobs without a configured email address did not send email notifications before the upgrade, but will default to sending email notifications to the root user via the new notification system after the upgrade to Proxmox VE 8.1.

  In order to disable notification emails, either change the job's notification mode to legacy-sendmail or configure the notification system to ignore backup job notifications.

Known Issues & Breaking Changes

Kernel

• With Kernel 6.5 and ZFS it can happen that the host hits a kernel bug when starting a VM with hugepages, and the host must be rebooted.

  More information can be found in the forum and in the bug reports for ZFS and Linux kernel.

• Some users with Intel Wi-Fi cards, like the AX201 model, reported that initialization of the card failed with Linux kernel 6.5.

  This is still being investigated. You should avoid booting into the new kernel if you have no physical access to your server and an Intel Wi-Fi device is used as its only connection. See the documentation for how to pin a kernel version.

• Some SAS2008 controllers need a workaround to get detected since kernel 6.2, see the forum thread for details.