RHV 4.4 SP1 released
Red Hat has released RHV 4.4 SP1, the latest version based on upstream oVirt 4.5.x series. Major changes include support for RHEL 8.6 hypervisors, and a new workflow to renew hypervisor certificates. Internal certificates changed validity from 5 years to 13 months during the 4.4 series, and this version rolls back these changes to allow a more convenient way of managing the platform.
Previous to performing an upgrade, the following documents are relevant:
Upgrading RHV-M to the latest version
First I enabled the right repositories for RHV 4.4, which now include some Ceph repositories:
subscription-manager repos \ --disable='*' \ --enable=rhel-8-for-x86_64-baseos-rpms \ --enable=rhel-8-for-x86_64-appstream-rpms \ --enable=rhv-4.4-manager-for-rhel-8-x86_64-rpms \ --enable=fast-datapath-for-rhel-8-x86_64-rpms \ --enable=jb-eap-7.4-for-rhel-8-x86_64-rpms \ --enable=openstack-16.2-cinderlib-for-rhel-8-x86_64-rpms \ --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms
In my lab environment, I found the following snags while upgrading:
Unsupported package manager
Prior to launching engine-setup to upgrade the Manager, I manually upgraded
the yum and rpm packages to avoid an issue with the RHV-M installer
(yum upgrade 'yum*' 'rpm*') .
I was originally runing RHV-M 4.4.5 based on RHEL 8.3, so quite an old release. After upgrading those packages, the upgrade progressed until I found the following issue:
2022-05-27 09:20:35,463+0200 DEBUG otopi.context context._executeMethod:127 Stage setup METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine_common.distro-rpm.packages.Plugin._setup 2022-05-27 09:20:35,465+0200 DEBUG otopi.context context._executeMethod:145 method exception Traceback (most recent call last): File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/util.py", line 305, in getPackageManager from otopi import minidnf File "/usr/lib/python3.6/site-packages/otopi/minidnf.py", line 25, in <module> import dnf.transaction_sr ModuleNotFoundError: No module named 'dnf.transaction_sr' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/util.py", line 312, in getPackageManager from otopi import miniyum File "/usr/lib/python3.6/site-packages/otopi/miniyum.py", line 17, in <module> import rpmUtils.miscutils ModuleNotFoundError: No module named 'rpmUtils' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine-common/distro-rpm/packages.py", line 293, in _setup osetuputil.getPackageManager(self.logger) File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/util.py", line 322, in getPackageManager 'No supported package manager found in your system' RuntimeError: No supported package manager found in your system 2022-05-27 09:20:35,467+0200 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Environment setup': No supported package manager found in your system
The installation was automatically rolled-back, so no issues there, so by just updating the yum and rpm packages, the issue was solved.
Unable to upgrade database schema
Another of the issues I found is that the upgrade process wasn't working due to
engine-setup being unable to refresh the database schema.
# view /var/log/ovirt-engine/setup/ovirt-engine-setup-20220527092805-eci7jy.log 255732 CONTEXT: SQL statement "ALTER TABLE vdc_options ALTER COLUMN default_value SET NOT NULL" 255733 PL/pgSQL function fn_db_change_column_null(character varying,character varying,boolean) line 10 at EXECUTE 255734 FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/pre_upgrade/0000_config.sql 255735 255736 2022-05-27 09:36:22,230+0200 ERROR otopi.plugins.ovirt_engine_setup.ovirt_engine.db.schema schema._misc:530 schema.sh: FATAL: Cannot execute sql command: --file=/usr /share/ovirt-engine/dbscripts/upgrade/pre_upgrade/0000_config.sql 255737 2022-05-27 09:36:22,231+0200 DEBUG otopi.context context._executeMethod:145 method exception 255738 Traceback (most recent call last): 255739 File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod 255740 method['method']() 255741 File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/db/schema.py", line 532, in _misc 255742 raise RuntimeError(_('Engine schema refresh failed')) 255743 RuntimeError: Engine schema refresh failed 255744 2022-05-27 09:36:22,232+0200 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Misc configuration': Engine schema refresh failed
This is covered in Bugzilla 2077387#c4, and is easily fixed by updating the database schema
root@rhevm ~ # /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select * from vdc_options where default_value is null ;" option_id | option_name | option_value | version | default_value -----------+-------------------------------+-------------------------------------------------------------+---------+--------------- 472 | ConfigDir | /etc/ovirt-engine | general | 473 | AdminDomain | internal | general | 474 | AllowDuplicateMacAddresses | false | general | 475 | DefaultWorkgroup | WORKGROUP | general | 476 | KeystoneAuthUrl | | general | 477 | LicenseCertificateFingerPrint | 5f 38 41 89 b1 33 49 0c 24 13 6b b3 e5 ba 9e c7 fd 83 80 3b | general | 478 | MacPoolRanges | 00:1A:4A:16:01:51-00:1A:4A:16:01:e6 | general | 479 | MaxMacsCountInPool | 100000 | general | 482 | VdsFenceOptions | | general | 483 | GlusterTunedProfile | rhs-high-throughput,rhs-virtualization | 3.0 | 484 | GlusterTunedProfile | rhs-high-throughput,rhs-virtualization | 3.1 | 485 | GlusterTunedProfile | rhs-high-throughput,rhs-virtualization | 3.2 | 486 | GlusterTunedProfile | rhs-high-throughput,rhs-virtualization | 3.3 | 487 | GlusterTunedProfile | rhs-high-throughput,rhs-virtualization | 3.4 | 488 | GlusterTunedProfile | rhs-high-throughput,rhs-virtualization | 3.5 | 462 | SupportBridgesReportByVDSM | true | 3.1 | 716 | GlusterTunedProfile | virtual-host,rhgs-sequential-io,rhgs-random-io | 4.2 | (17 rows) root@rhevm ~ # /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "UPDATE vdc_options SET default_value=option_value WHERE default_value IS NULL AND option_value IS NOT NULL;" UPDATE 17 root@rhevm ~ # /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "UPDATE vdc_options SET default_value='' WHERE default_value IS NULL AND option_value IS NULL;" UPDATE 0
Finally, the engine-setup process finishes OK and after running a yum upgrade -y && systemctl restart ovirt-engine the Web UI is available again.
Upgrading RHEL hypervisors
My hypervisors where also running RHEL 8.3, and some minor RPM problems were found. It is expected that RHV-H insallations (RHV host) do not find such issues.
After enabling the repositories:
subscription-manager repos \ --disable='*' \ --enable=rhel-8-for-x86_64-baseos-rpms \ --enable=rhel-8-for-x86_64-appstream-rpms \ --enable=rhv-4-mgmt-agent-for-rhel-8-x86_64-rpms \ --enable=fast-datapath-for-rhel-8-x86_64-rpms \ --enable=advanced-virt-for-rhel-8-x86_64-rpms \ --enable=openstack-16.2-cinderlib-for-rhel-8-x86_64-rpms \ --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms
When using the integrated Cluster Upgrade assistant in the WebUI, package
resolution problems were found, and could be trivially fixed by removing
the rpm -e network-scripts-openvswitch2.11.
Certificate validation
KCS 6865861 provides a detail
explanation on how the process to renew certificates works at the moment, and
provides a nifty script to check overall certificate validity of both RHVM and
hypervisors (cert_date.sh).
A sample run shows:
root@rhevm ~ # ./cert_date_0.sh This script will check certificate expiration dates Checking RHV-M Certificates... ================================================= /etc/pki/ovirt-engine/ca.pem: Feb 27 07:27:16 2028 GMT /etc/pki/ovirt-engine/certs/apache.cer: Jun 11 11:38:13 2023 GMT /etc/pki/ovirt-engine/certs/engine.cer: Jun 11 11:38:12 2023 GMT /etc/pki/ovirt-engine/qemu-ca.pem Aug 5 19:07:11 2030 GMT /etc/pki/ovirt-engine/certs/websocket-proxy.cer Jun 11 11:38:13 2023 GMT /etc/pki/ovirt-engine/certs/jboss.cer Jun 11 11:38:12 2023 GMT /etc/pki/ovirt-engine/certs/ovirt-provider-ovn May 18 16:01:35 2023 GMT /etc/pki/ovirt-engine/certs/ovn-ndb.cer May 18 16:01:35 2023 GMT /etc/pki/ovirt-engine/certs/ovn-sdb.cer May 18 16:01:35 2023 GMT /etc/pki/ovirt-engine/certs/vmconsole-proxy-helper.cer Feb 3 07:28:10 2023 GMT /etc/pki/ovirt-engine/certs/vmconsole-proxy-host.cer Feb 3 07:28:10 2023 GMT /etc/pki/ovirt-engine/certs/vmconsole-proxy-user.cer Feb 3 07:28:10 2023 GMT Checking Host Certificates... Host: rhevh1 ================================================= /etc/pki/vdsm/certs/vdsmcert.pem: May 30 02:55:03 2027 GMT /etc/pki/vdsm/libvirt-spice/server-cert.pem: May 30 02:55:03 2027 GMT /etc/pki/vdsm/libvirt-vnc/server-cert.pem: May 30 02:55:03 2027 GMT /etc/pki/libvirt/clientcert.pem: May 30 02:55:03 2027 GMT /etc/pki/vdsm/libvirt-migrate/server-cert.pem: May 30 02:55:04 2027 GMT Host: rhevh2 ================================================= /etc/pki/vdsm/certs/vdsmcert.pem: May 30 03:19:59 2027 GMT /etc/pki/vdsm/libvirt-spice/server-cert.pem: May 30 03:19:59 2027 GMT /etc/pki/vdsm/libvirt-vnc/server-cert.pem: May 30 03:19:59 2027 GMT /etc/pki/libvirt/clientcert.pem: May 30 03:19:59 2027 GMT /etc/pki/vdsm/libvirt-migrate/server-cert.pem: May 30 03:19:59 2027 GMT
Wrap-up
All in all, some minor snags during the upgrade that should be fixed in newer releases to having a smoother experience.
Happy hacking!