FranGarcia.me (Posts about rhv)

Notes on upgrading RHV 4.3 to RHV 4.4

Fran Garcia — Sat, 08 Aug 2020 10:04:11 GMT

Reciently Red Hat has published the latest RHV 4.4 version. This introduces some major changes in the underlying operating system (migration from RHEL7 to RHEL8 in both hypervisors and Engine / Self Hosted Engine), and a bunch of new features.

There are extensive notes on how to perform the upgrade, especially for the Self-hosted Engine-type of deployments.

I upgraded a small 2-node lab environment and besides the notes already mentioned in the docs above, I also found relevant:

Before you start

Understand the NIC naming differences between RHEL7 and RHEL8.
- Your hypervisor NICs will probably be renamed.
Jot down your hypervisors' NIC to MAC-addresses mappings prior to attempting an upgrade.
- This will ease understanding what NIC is what after installing RHEL8.
When using shared storage (FC), beware of unmapping it while you reinstall each host, or ensure your kickstart does NOT clear the shared disks.
- Otherwise this might lead into data loss!!

Prerequistes

One spare hypervisor, feshly installed with RHEL8/RHVH8 and NOT added to the manager.
One additional LUN / NFS share for the new SHE 4.4 deployment.
- The installer does not upgrade the old SHE in-place, so a new lun is required.
- This eases the rollback, as the original SHE LUN is untouched.
Ensure the new hypervisor has all the configuration to access all required networks prior to starting the upgrade.
- IP configuration for the ovirtmgmt network (obvious).
- IP configuration for any NFS/iSCSI networks, if required.
- Shared FC storage, if required.
- This is critical as the restore process does not prompt to configure/fix network settings when deploying the upgraded manager.
Extra steps
- Collect your RHV-M details:
  - IP address and netmask
  - FQDN
  - Mac-address if using DHCP.
  - Extra software and additional RPMs (eg: AD/IDM/ldap integration, etc)
  - Existing /etc/hosts details in case you use hosts instead of DNS (bad bad bad!!!).
  - Same for hypervisors!
- Optionally: Mark your networks within the cluster as non-Required . This might be useful until BZ #1867198 is addressed.

Deploying and registering the hypervisors.

The RHEL8/RHVH8 can be deployed as usual with Foreman / Red Hat Satellite.

Ensure the hypervisors are registered and have access to the repositories as below:

RHEL8 Host repositories

POOLID=`subscription-manager list --available --matches "Red Hat Virtualization"  --pool-only | head -n 1`
subscription-manager attach --pool=$POOLID
subscription-manager repos \
    --disable='*' \
    --enable=rhel-8-for-x86_64-baseos-rpms \
    --enable=rhel-8-for-x86_64-appstream-rpms \
    --enable=rhv-4-mgmt-agent-for-rhel-8-x86_64-rpms \
    --enable=fast-datapath-for-rhel-8-x86_64-rpms \
    --enable=ansible-2.9-for-rhel-8-x86_64-rpms \
    --enable=advanced-virt-for-rhel-8-x86_64-rpms

yum module reset -y virt
yum module enable -y virt:8.2
systemctl enable --now firewalld
yum install -y rhevm-appliance ovirt-hosted-engine-setup

RHVH8 Host repositories

POOLID=$(subscription-manager list --available --matches "Red Hat Virtualization"  --pool-only | head -n 1)
subscription-manager attach --pool=$POOLID
subscription-manager repos \
    --disable='*' \
    --enable=rhvh-4-for-rhel-8-x86_64-rpms
systemctl enable --now firewalld
yum install -y rhevm-appliance

Powering off RHV 4.3 manager

Set the Manager in global maintenance mode.
OPTIONAL: Mark your networks within the cluster as non-Required . This might be useful until BZ #1867198 is addressed.
Stop the ovirt-engine service.
Backup the RHV 4.3 database and save in a shared space.

Performing the RHV-M upgrade

Copy the database backup into the RHEL8 hypervisor.
Launch the restore process with hosted-engine --deploy --restore-from-file=backup.tar.bz2

The process has changed significantly in the last RHV releases and it now performs the new SHE rollout or restore in two phases :

Phase 1: it tries to roll it out in the hypervisor local storage.
- Gather FQDN, IP details of the Manager.
- Gather other configuration.
Phase 2: migrate to shared storage.
- If Phase1 is successful, this takes care of gathering shared storage details (LUN ID or NFS defails).
- Copy the bootstrap manager into the shared storage.
- Configure the ovirt-ha-broker and ovirt-ha-agent in the hypervisor to monitor and ensure the SHE is started.

Phase 1 details

[root@rhevh2 rhev]# time  hosted-engine --deploy --restore-from-file=engine-backup-rhevm-20200807_1536.tar.bz2
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
          Continuing will configure this host for serving as hypervisor and will create a local VM with a running engine.
          The provided engine backup file will be restored there,
          it's strongly recommended to run this tool on an host that wasn't part of the environment going to be restored.
          If a reference to this host is already contained in the backup file, it will be filtered out at restore time.
          The locally running engine will be used to configure a new storage domain and create a VM there.
          At the end the disk of the local VM will be moved to the shared storage.
          The old hosted-engine storage domain will be renamed, after checking that everything is correctly working you can manually remove it.
          Other hosted-engine hosts have to be reinstalled from the engine to update their hosted-engine configuration.
          Are you sure you want to continue? (Yes, No)[Yes]: yes
          It has been detected that this program is executed through an SSH connection without using tmux.
          Continuing with the installation may lead to broken installation if the network connection fails.
          It is highly recommended to abort the installation and run it inside a tmux session using command "tmux".
          Do you want to continue anyway? (Yes, No)[No]: yes
          Configuration files: 
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20200807155111-5blcva.log
          Version: otopi-1.9.2 (otopi-1.9.2-1.el8ev)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup (late)
[ INFO  ] Stage: Environment customization

          --== STORAGE CONFIGURATION ==--


          --== HOST NETWORK CONFIGURATION ==--

          Please indicate the gateway IP address [10.48.0.100]: 
[ INFO  ] TASK [ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Force facts gathering]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Detecting interface on existing management bridge]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Get all active network interfaces]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Filter bonds with bad naming]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Generate output list]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Collect interface types]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Check for Team devices]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Get list of Team devices]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Filter unsupported interface types]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Failed if only teaming devices are availible]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Validate selected bridge interface if management bridge does not exist]
[ INFO  ] skipping: [localhost]
         Please indicate a nic to set ovirtmgmt bridge on: (eth4.100, ens15.200) [ens15.200]: eth4.100
          Please specify which way the network connectivity should be checked (ping, dns, tcp, none) [dns]: 

          --== VM CONFIGURATION ==--

          Please enter the name of the datacenter where you want to deploy this hosted-engine host. Please note that if you are restoring a backup that contains info about other hosted-engine hosts,
          this value should exactly match the value used in the environment you are going to restore. [Default]: 
          Please enter the name of the cluster where you want to deploy this hosted-engine host. Please note that if you are restoring a backup that contains info about other hosted-engine hosts,
          this value should exactly match the value used in the environment you are going to restore. [Default]: 
          Renew engine CA on restore if needed? Please notice that if you choose Yes, all hosts will have to be later manually reinstalled from the engine. (Yes, No)[No]: 
          Pause the execution after adding this host to the engine?
          You will be able to iteratively connect to the restored engine in order to manually review and remediate its configuration before proceeding with the deployment:
          please ensure that all the datacenter hosts and storage domain are listed as up or in maintenance mode before proceeding.
          This is normally not required when restoring an up to date and coherent backup. (Yes, No)[No]: 
          If you want to deploy with a custom engine appliance image,
          please specify the path to the OVA archive you would like to use
          (leave it empty to skip, the setup will use rhvm-appliance rpm installing it if missing): 
          Please specify the number of virtual CPUs for the VM (Defaults to appliance OVF value): [4]: 
          Please specify the memory size of the VM in MB (Defaults to appliance OVF value): [16384]: 
[ INFO  ] Detecting host timezone.
          Please provide the FQDN you would like to use for the engine.
          Note: This will be the FQDN of the engine VM you are now going to launch,
          it should not point to the base host or to any other existing machine.
         Engine VM FQDN:  []: rhevm.example.org
          Please provide the domain name you would like to use for the engine appliance.
          Engine VM domain: [example.org]
          Enter root password that will be used for the engine appliance: 
          Confirm appliance root password: 
          Enter ssh public key for the root user that will be used for the engine appliance (leave it empty to skip): 
          Do you want to enable ssh access for the root user (yes, no, without-password) [yes]: 
          Do you want to apply a default OpenSCAP security profile (Yes, No) [No]: 
          You may specify a unicast MAC address for the VM or accept a randomly generated default [00:16:3e:03:ec:35]: 
          How should the engine VM network be configured (DHCP, Static)[DHCP]? static
          Please enter the IP address to be used for the engine VM []: 10.48.0.4
[ INFO  ] The engine VM will be configured to use 10.48.0.4/24
          Please provide a comma-separated list (max 3) of IP addresses of domain name servers for the engine VM
          Engine VM DNS (leave it empty to skip) [10.48.0.100]: 
          Add lines for the appliance itself and for this host to /etc/hosts on the engine VM?
          Note: ensuring that this host could resolve the engine VM hostname is still up to you
          (Yes, No)[No] 

          --== HOSTED ENGINE CONFIGURATION ==--

          Please provide the name of the SMTP server through which we will send notifications [localhost]: 
          Please provide the TCP port number of the SMTP server [25]: 
          Please provide the email address from which notifications will be sent [root@localhost]: 
          Please provide a comma-separated list of email addresses which will get notifications [root@localhost]: 
          Enter engine admin password: 
          Confirm engine admin password: 
[ INFO  ] Stage: Setup validation
          Please provide the hostname of this host on the management network [rhevh2]: rhevh2.example.org
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stage: Misc configuration (early)
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Cleaning previous attempts
[ INFO  ] TASK [ovirt.hosted_engine_setup : Execute just a specific set of steps]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Force facts gathering]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Install oVirt Hosted Engine packages]
[ INFO  ] ok: [localhost]

[... snip ...]

The manager is now being deployed and made available via the hypervisor at a later stage:

[ INFO  ] TASK [ovirt.hosted_engine_setup : Adding new SSO_ALTERNATE_ENGINE_FQDNS line]
[ INFO  ] changed: [localhost -> rhevm.example.org]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Restart ovirt-engine service for changed OVF Update configuration and LibgfApi support]
[ INFO  ] changed: [localhost -> rhevm.example.org]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Mask cloud-init services to speed up future boot]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Wait for ovirt-engine service to start]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Open a port on firewalld]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Expose engine VM webui over a local port via ssh port forwarding]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Evaluate temporary bootstrap engine URL]
[ INFO  ] ok: [localhost]
[ INFO  ] The bootstrap engine is temporary accessible over https://rhevh2.example.org:6900/ovirt-engine/ 
[ INFO  ] TASK [ovirt.hosted_engine_setup : Detect VLAN ID]
[ INFO  ] changed: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Set Engine public key as authorized key without validating the TLS/SSL certificates]
[ INFO  ] changed: [localhost]
[...]
[ INFO  ] TASK [ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] skipping: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Always revoke the SSO token]
[ INFO  ] TASK [ovirt.hosted_engine_setup : include_tasks]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials]
[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Wait for the host to be up]

The bootstrap manager is available at https://hypervisor.example.org:6900/ovirt-engine/ and the installer tries to add the current host under the Manager management. (It waits for the host to be in the 'Up' state. This is why is important to have all the storage and network prerequisites prepared/available).

And to finish up :

[ INFO  ] ok: [localhost]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool localvm7imrhb7u]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool localvm7imrhb7u]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool {{ local_vm_disk_path.split('/')[5] }}]
[ INFO  ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool {{ local_vm_disk_path.split('/')[5] }}]
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20200807193709.conf'
[ INFO  ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ INFO  ] Hosted Engine successfully deployed
[ INFO  ] Other hosted-engine hosts have to be reinstalled in order to update their storage configuration. From the engine, host by host, please set maintenance mode and then click on reinstall button ensuring you choose DEPLOY in hosted engine tab.
[ INFO  ] Please note that the engine VM ssh keys have changed. Please remove the engine VM entry in ssh known_hosts on your clients.

real    45m1,768s
user    18m4,639s
sys     1m9,271s

After finishing the upgrade it is also recommended to register the RHV-Manager virtual machine and upgrade to the latest RPMs available in the Red Hat CDN.

Set the Hosted Engine in Global Maintenance mode and:

POOLID=`subscription-manager list --available --matches "Red Hat Virtualization"  --pool-only | head -n 1`
subscription-manager attach --pool=$POOLID

subscription-manager repos \
    --disable='*' \
    --enable=rhel-8-for-x86_64-baseos-rpms \
    --enable=rhel-8-for-x86_64-appstream-rpms \
    --enable=rhv-4.4-manager-for-rhel-8-x86_64-rpms \
    --enable=fast-datapath-for-rhel-8-x86_64-rpms \
    --enable=ansible-2.9-for-rhel-8-x86_64-rpms \
    --enable=jb-eap-7.3-for-rhel-8-x86_64-rpms

yum module -y enable pki-deps
yum module -y enable postgresql:12
yum module reset -y virt
yum module enable -y virt:8.2

Performing the upgrade :

systemctl stop ovirt-engine
yum upgrade -y
engine-setup --accept-defaults

Rolling back a failed upgrade

A rollback can be performed if the following applies:

The deployment or upgrade to the new RHV 4.4 Manager was not successfull.
No new instances have been created and/or VMs have not been altered (eg, added disks or nics, etc). If a rollback occurs those changes will be inconsistent with the old manager DB status and potentially imposible to reconciliate.

If so, the rollback can be performed by:

Powering off the new RHEL8/RHVH hypervisor and manager.
Powering on the old Manager in a RHEL7 hosts. They should be pointed to the old SHE LUN and storage.

Finalising the upgrade

At this point you should have a working manager under the regular https://FQDN/ovirt-engine/ address. Don't forget to clear cookies and browser cache as this might lead into strange WebUI issues.

At this point you can continue reinstalling your hypervisors. I'd suggest:

Starting with your SHE hypervisors first. This will ensure you have SHE HA asap.
Then the non-SHE hypervisors.
Then finalise with the rest of the task such as upgrading Cluster and DC compatibility, rebooting the guest VMs, etc.

Happy hacking!

Provisioning RHV 4.1 hypervisors using Satellite 6.2

Fran Garcia — Wed, 17 Jul 2019 08:15:39 GMT

Overview

The RHV-H 4.1 installation documentation describes a method to provision RHV-H hypervisors using PXE and/or Satellite. This document covers all steps required to archieve such configuration in a repeatable manner.

Prerequisites

A RHV-H installation ISO file such as RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso downloaded into Satellite/Capsules.
rhviso2media.sh script (see here)

Creating the installation media in Satellite and Capsules

Deploy the RHV-H iso in /var/lib/pulp/tmp
Run the rhviso2media.sh to populate installation media directories. It will make the following files available:
- kernel and initrd files in tftp://host/ISOFILE/vmlinuz and tftp://host/ISOFILE/initrd
- DVD installation media in /var/www/html/pub/ISOFILE directory
- squashfs.img file in /var/www/html/pub/ISOFILE directory

Example:

# ./rhviso2media.sh RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso
Mounting /var/lib/pulp/tmp/RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso in /mnt/RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso ...
mount: /dev/loop0 is write-protected, mounting read-only
Copying ISO contents to /var/www/html/pub/RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso ...
Extracting redhat-virtualization-host-image-update ...
./usr/share/redhat-virtualization-host/image
./usr/share/redhat-virtualization-host/image/redhat-virtualization-host-4.0-20170307.1.el7_3.squashfs.img
./usr/share/redhat-virtualization-host/image/redhat-virtualization-host-4.0-20170307.1.el7_3.squashfs.img.meta
1169874 blocks
OK
Copying squash.img to public directory . Available as http://sat62.lab.local/pub/RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso/squashfs.img ...
Copying /mnt/RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso/images/pxeboot/vmlinuz /mnt/RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso/images/pxeboot/initrd.img to /varlib/tftpboot/RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso ...
OK
Unmounting /mnt/RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso

Create Installation media

RHVH Installation media

http://fake.url

This is not used as it the kickstart will use a liveimg url, not a media url; however Satellite is stubborn and still requires it.

Create partition table

name : RHVH Kickstart default

<%#
kind: ptable
name: Kickstart default
%>
zerombr
clearpart --all --initlabel
autopart --type=thinp

Create pxelinux for RHV

Based on Kickstart default PXELinux

```erb
<%#
kind: PXELinux
name: RHVH Kickstart default PXELinux
%>
#
# This file was deployed via '<%= @template_name %>' template
#
# Supported host/hostgroup parameters:
#
# blacklist = module1, module2
#   Blacklisted kernel modules
#
<%
options = []
if @host.params['blacklist']
    options << "modprobe.blacklist=" + @host.params['blacklist'].gsub(' ', '')
end
options = options.join(' ')
-%>

DEFAULT rhvh

LABEL rhvh
    KERNEL <%= @host.params["rhvh_image"] %>/vmlinuz <%= @kernel %>
    APPEND initrd=<%= @host.params["rhvh_image"] %>/initrd.img inst.stage2=http://<%= @host.hostgroup.subnet.tftp.name %>/pub/<%= @host.params["rhvh_image"] %>/ ks=<%= foreman_url('provision') %> intel_iommu=on ssh_pwauth=1 local_boot_trigger=<%= foreman_url("built") %> <%= options %>
    IPAPPEND 2
```

Create Kickstart for RHV

File under Satellite Kickstart Default for RHVH .

Note that the @host.hostgroup.subnet.tftp.name variable is used to point to the capsule associated to this host, rather than the Satellite server itself.

<%#
kind: provision
name: Satellite Kickstart default
%>
<%
rhel_compatible = @host.operatingsystem.family == 'Redhat' && @host.operatingsystem.name != 'Fedora'
os_major = @host.operatingsystem.major.to_i
# safemode renderer does not support unary negation
pm_set = @host.puppetmaster.empty? ? false : true
puppet_enabled = pm_set || @host.params['force-puppet']
salt_enabled = @host.params['salt_master'] ? true : false
section_end = (rhel_compatible && os_major <= 5) ? '' : '%end'
%>
install
# not required # url --url=http://<%= @host.hostgroup.subnet.tftp.name %>/pub/<%= @host.params["rhvh_image"] %>
lang en_US.UTF-8
selinux --enforcing
keyboard es
skipx

<% subnet = @host.subnet -%>
<% if subnet.respond_to?(:dhcp_boot_mode?) -%>
<% dhcp = subnet.dhcp_boot_mode? && !@static -%>
<% else -%>
<% dhcp = !@static -%>
<% end -%>

network --bootproto <%= dhcp ? 'dhcp' : "static --ip=#{@host.ip} --netmask=#{subnet.mask} --gateway=#{subnet.gateway} --nameserver=#{[subnet.dns_primary, subnet.dns_secondary].select(&:present?).join(',')}" %> --hostname <%= @host %><%= os_major >= 6 ? " --device=#{@host.mac}" : '' -%>

rootpw --iscrypted <%= root_pass %>
firewall --<%= os_major >= 6 ? 'service=' : '' %>ssh
authconfig --useshadow --passalgo=sha256 --kickstart
timezone --utc <%= @host.params['time-zone'] || 'UTC' %>

<% if @host.operatingsystem.name == 'Fedora' and os_major <= 16 -%>
# Bootloader exception for Fedora 16:
bootloader --append="nofb quiet splash=quiet <%=ks_console%>" <%= grub_pass %>
part biosboot --fstype=biosboot --size=1
<% else -%>
bootloader --location=mbr --append="nofb quiet splash=quiet" <%= grub_pass %>
<% end -%>

<% if @dynamic -%>
%include /tmp/diskpart.cfg
<% else -%>
<%= @host.diskLayout %>
<% end -%>

text
reboot

liveimg --url=http://<%= foreman_server_fqdn %>/pub/<%= @host.params["rhvh_image"] %>/squashfs.img


%post --nochroot
exec < /dev/tty3 > /dev/tty3
#changing to VT 3 so that we can see whats going on....
/usr/bin/chvt 3
(
cp -va /etc/resolv.conf /mnt/sysimage/etc/resolv.conf
/usr/bin/chvt 1
) 2>&1 | tee /mnt/sysimage/root/install.postnochroot.log
<%= section_end -%>


%post
logger "Starting anaconda <%= @host %> postinstall"

nodectl init

exec < /dev/tty3 > /dev/tty3
#changing to VT 3 so that we can see whats going on....
/usr/bin/chvt 3
(
<% if subnet.respond_to?(:dhcp_boot_mode?) -%>
<%= snippet 'kickstart_networking_setup' %>
<% end -%>

#update local time
echo "updating system time"
/usr/sbin/ntpdate -sub <%= @host.params['ntp-server'] || '0.fedora.pool.ntp.org' %>
/usr/sbin/hwclock --systohc

<%= snippet "subscription_manager_registration" %>

<% if @host.info['parameters']['realm'] && @host.realm && @host.realm.realm_type == 'Red Hat Identity Management' -%>
<%= snippet "idm_register" %>
<% end -%>

# update all the base packages from the updates repository
#yum -t -y -e 0 update

<%= snippet('remote_execution_ssh_keys') %>

sync

<% if @provisioning_type == nil || @provisioning_type == 'host' -%>
# Inform the build system that we are done.
echo "Informing Foreman that we are built"
wget -q -O /dev/null --no-check-certificate <%= foreman_url %>
<% end -%>
) 2>&1 | tee /root/install.post.log
exit 0

<%= section_end -%>

Create new Operating system

Name: RHVH
Major Version: 7
Partition table: RHVH Kickstart default
Installation media: RHVH Installation media
Templates: "Kickstart default PXELinux for RHVH" and "Satellite kickstart default for RHVH"

Associate the previously-created provisioning templates with this OS.

Create a new hostgroup

Create a new host-group with a Global Parameter called rhvh_image . This parameter is used by the provisioning templates to generate the installation media paths as required.

eg:

rhvh_image = RHVH-4.0-20170308.0-RHVH-x86_64-dvd1.iso

rhviso2media

Final thoughts

Future Satellite versions of satellite might include better integration of RHV-H provisioning; however the method described above can be used in the meantime.

Happy hacking!

RHV 4.2: Using rhv-log-collector-analyzer to assess your virtualization environment

Fran Garcia — Mon, 19 Nov 2018 21:15:39 GMT

RHV 4.2 includes a tool that allows to quickly analyze your RHV environment. It bases its analysis in either a logcollector report (sosreport and others), or it can connect live to you environment and generate some nice JSON or HTML output.

NOTE : This article is deprecated and is only left for historical reasons.

rhv-log-collector-analyzer now only supports live reporting, so use the method rhv-log-collector-analyzer --live to grab a snapshot of your deployment and verify its status.

You'll find it already installed in RHV 4.2 , and gathering a report is as easy as:

# rhv-log-collector-analyzer --live
Generating reports:
===================
Generated analyzer_report.html

If you need to assess an existing logcollector report on a new system that never had a running RHV-Manager, things get a bit more complicated:

root@localhost # yum install -y ovirt-engine
root@localhost # su - postgres
postgres@localhost ~ # source scl_source enable rh-postgresql95
postgres@localhost ~ # cd /tmp
postgres@localhost /tmp # time rhv-log-collector-analyzer  /tmp/sosreport-LogCollector-20181106134555.tar.xz

Preparing environment:
======================
Temporary working directory is /tmp/tmp.do6qohRDhN
Unpacking postgres data. This can take up to several minutes.
sos-report extracted into: /tmp/tmp.do6qohRDhN/unpacked_sosreport
pgdump extracted into: /tmp/tmp.do6qohRDhN/pg_dump_dir
Welcome to unpackHostsSosReports script!
Extracting sosreport from hypervisor HYPERVISOR1 in /tmp/ovirt-log-collector-analyzer-hosts/HYPERVISOR1
Extracting sosreport from hypervisor HYPERVISOR2 in /tmp/ovirt-log-collector-analyzer-hosts/HYPERVISOR2
Extracting sosreport from hypervisor HYPERVISOR3 in /tmp/ovirt-log-collector-analyzer-hosts/HYPERVISOR3
Extracting sosreport from hypervisor HYPERVISOR4 in /tmp/ovirt-log-collector-analyzer-hosts/HYPERVISOR4

Creating a temporary database in /tmp/tmp.do6qohRDhN/postgresDb/pgdata. Log of initdb is in /tmp/tmp.do6qohRDhN/initdb.log

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
LOG:  redirecting log output to logging collector process
HINT:  Future log output will appear in directory "pg_log".
Importing the dump into a temporary database. Log of the restore process is in /tmp/tmp.do6qohRDhN/db-restore.log

Generating reports:
===================
Generated analyzer_report.html

Cleaning up:
============
Stopping temporary database
Removing temporary directory /tmp/tmp.do6qohRDhN

You'll find a analyzer_report.html file in your current working directory. It can be reviews with a text-only browser such as lynx/links , or opened with a proper full-blown browser.

Bonus track

Sometimes it can also be helpful to check the database dump that is included in the logcollector report. In order to do that, you can do something like:

Review pg_dump_dir in the log above: /tmp/tmp.do6qohRDhN/pg_dump_dir .

Initiate a new postgres instance as follows :

postgres@localhost $ source scl_source enable rh-postgresql95
postgres@localhost $ export PGDATA=/tmp/foo
postgres@localhost $ initdb -D ${PGDATA} 
postgres@localhost $ /opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl start -D ${PGDATA} -s -w -t 30 &
postgres@localhost $ psql -c "create database testengine"
postgres@localhost $ psql -c "create schema testengine"
postgres@localhost $ psql testengine < /tmp/tmp.*/pg_dump_dir/restore.sql

Happy hacking!

Configuring Cisco UCS fencing in RHV Manager

Fran Garcia — Wed, 21 Feb 2018 16:31:13 GMT

Just a quick postit on how to configure fencing of Cisco UCS servers in RHV. Since the fencing agent requires the full path to the Service Profile in UCS, it needs to be carefully specified as:

Slot: MyServer
Options: suborg=org-CLOUD/org-CLOUD-HYPERVISORS/org-AZ1,ssl_insecure=1

The path to the Service Profile can be checked by inspecting the UCS 'Events' tab, which will show entries similar to:

<eventRecord
affected="org-root/org-CLOUD/org-CLOUD-HYPERVISORS/org-AZ1/ls-MyServer"
cause="transition"
changeSet=""
code="E4196007"
created="2018-02-21T12:44:16Z"
descr="[FSM:BEGIN]: Configuring Service Profile CFS1PHY03(FSM:sam:dme:LsServerConfigure)"
id="2586840"
ind="state-transition"
sessionId="internal"
severity="info"
trig="special"
txId="1209150"
user="ucsCentral"
dn="event-log/2586840"
status="created"
sacl="addchild,del,mod">
</eventRecord>

Happy hacking!

Configuring an iscsi volume for RHV usage

Fran Garcia — Tue, 29 Aug 2017 06:23:07 GMT

This is a cheatsheet to quickly configure storage and export it as an iSCSI volume using RHEL7 targetcli and have it configured under RHV. This is by no means a production configuration as your RHEL7 system might become a single point of failure, but it convers nicely building a home lab or test environment.

Just for clarity, a quick reminder of iSCSI concepts:

An iSCSI target provides some storage (here called server),
An iSCSI initiator uses this available storage (here called client).

Prerequisites

Configure server's storage

You can configure several types of backends, and for me the most versatile is using LVM's Logical Volumes. You'll need to create your volumes in advance, for example:

lvcreate yourVG -n yourLV1 -L 50G

Install software

Install the targetcli RPM:

yum install -y targetcli

Enable the target daemon (NOT targetd)

systemctl enable --now target.service

Gather RHV configuration

You'll need to gather the following information from RHV:

IQN (iSCSI identifier)

Configure and enable iSCSI

targetcli provides a very simple way to create iscsi targets once you understand how it works. Namely what needs to be done is:

Add your backend devices. This is where you add into targetcli's control the LVM devices created in previous steps
Create an IQN target. This is a collection of luns shared to the same system(s) under the same group. It is used later to apply ACLs so only certain hosts can use certain LUNs.
Add LUNs into your IQN target. After creating your IQN target, you need to add the backstore devices so they're shared via iSCSI.
Add ACLs into your IQN target. Unless configured otherwise, LUNs are not visible to systems unless they're added into the right ACL.

Here is a dump on how all this can be accomplished with targetcli:

Add storage into RHV

foo

Tales from the Field: Updating RHEV 3.5 to RHV 4.1

Fran Garcia — Tue, 11 Jul 2017 20:06:41 GMT

Welcome to the first post in the Tales from the Field series. In this section I indent to collect the most useful stories about different products and how they are being used in every day situations.

For a starter, I'll cover the new RHV suite, now on version 4.1, and the upgrade path to have an older environment migrated to the latest version.

RHV fundamentals

RHEV is now simply known as Red Hat Virtualization and it has several components:

The RHEV Manager, a system that acts as a central configuration and orchestration point for all hypervisors.
Several types of hypervisors:
- RHEL-6 hypervisors. Used primarily in the RHEV 3.x suite.
- RHEL-7 hypervisors. Used in the latter RHEV 3.x and 4.x suites.
- RHEV-H 6 (vintage) hypervisors. Used primarily in the RHEV 3.x suite.
- RHEV-H 7 (vintage) hypervisors. Used in the latter RHEV 3.x suite.
- RHEV-H 7 (ngn) hypervisors. The new hypervisor in RHEV 4.x suite.

Standard RHEV to RHV upgrade path

Red Hat has put together some documentation and labs that should ease the upgrade process:

The 'RHV upgrade helper' lab
Extensive documentation:
- Upgrading 3.5 to 3.6 :
- Upgrading 3.6 to 4.0 :
  - RHV 4.0 UPGRADE GUIDE
  - How to upgrade RHEV 3.6 Hosted Engine to RHV 4.0
- Upgrading 4.0 to 4.1 :
  - RHV 4.1 UPGRADE GUIDE

The 3.6 to 4.0 step is probably the one that requires most planning, as it involves rebuilding the manager with a new version of the underlying operating system (RHEL6 to RHEL7).

Shortcutting the upgrade

If I'm running a 3.5 cluster I surely do not need to install/upgrade every version of RHEV to end up in the latest version, do I?

Unfortunately, yes. But there are some tweaks that can be done in the process to ease out the migration.

If you are running a 3.5 cluster with either EL6 or EL7 3.5 hypervisors, you'll need to reinstall them. RHV4 uses the next generation node (NGN) hypervisors which are not upgradable from previous versions of RHEV-H. The good news is it's now available a 3.6 NGN version of the hypervisor. So you can reinstall some of your existing 3.5 hosts with 3.6NGN, and that's the only reinstallation you will need.
Once you have updated your manager to 3.5.latest, then to 3.6 you are in a position to start reinstalling your hypervisors with 3.6 NGN. Once that part is done, it is required you change your cluster compatibility settings to 3.6 and recommended you reboot all your VMs. There is an ongoing buzilla BZ#1474870 that should address documenting what is the official recommendation regarding VM reboot on an upgrade scenario.
Once you are in RHEV 3.6 Manager with 3.6 compat level, you need to backup your engine database and keep in a safe place. If you're migrating to a Hosted Engine from a standalone Engine, you'll need it to restore the data. If your plan is to reinstall your standalone Engine with RHEL7, it's the moment to do so.
Once your Engine is available, you'll run a restore then an upgrade so it's upgraded to the latest 4.0 version .
If your 3.6 hypervisors have a configured RHSM/Satellite subscription, you'll be able to launch a 3.6 to 4.0 upgrade from the Manager itself. This greatly eases the upgrade as no manual provision is required.
After all hypervisors have been migrated to 4.0, change cluster compatibility to 4.0 .
Rinse and repeat for Manager & host upgrades to 4.1 .

Happy hacking!

Satellite 6.2 : Adding RHV4 compute resources

Fran Garcia — Thu, 29 Jun 2017 10:12:30 GMT

Satellite server allows to manage compute resources such as Vmware and Red Hat Virtualization (RHEV/RHV). In the later RHV versions there's a caveat on how they should be added into Satellite.

The RHV certificate file has changed location:

http://rhv4-fqdn//ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA

The API endpoint should be specified with a 'v3' tag, as follows :

https://rhv4-fqdn/ovirt-engine/api/v3

Happy hacking!

Injecting proxy configuration into remote-viewer (RHEV/oVirt console)

Fran Garcia — Tue, 06 Jun 2017 13:09:01 GMT

One of the things that bother me is that I have not been able to find an easy way to change proxy settings when using the remote-viewer tool to connect to RHEV/oVirt virtual machines.

Finally I managed to put some thought into how to handle this situation -- and the easiest way I found is just to puto a wrapper into remote-viewer so whenver it's invoked via the browser, it can do all required mangling of the *.vv file we just downloaded.

The virt-viwer *.vv files are just INI files so updating them is usually quite easy via :

All in all, my wrapper looks like this :

#!/bin/bash

# Wrapper around remote-viewer to inject proxy parameters.
# Needs ansible, but can be trivially amended to use crudini.

export PATH=$PATH:/usr/bin:/usr/sbin

path=$1
chmod 600 $path

logger $0 $path


# Add your magic about detecting whether we need to use a proxy in the 'if' below ;-)
if [ $(netstat -putan 2> /dev/null | grep -P ':3128.*LISTEN' -c) -ne 0 ]; then
    logger $0 enabling proxy
    ansible -m ini_file localhost -a "state=present section=virt-viewer option=proxy value=\"http://localhost:3128\" path=\"$path\""
    ansible -m ini_file localhost -a "state=present section=ovirt       option=proxy value=\"http://localhost:3128\" path=\"$path\""
    sed -i 's# = #=#g' $path
fi

#logger < $path
/usr/bin/remote-viewer $path

exit $?

And then you just need to instruct your browser to launch remote-viewer.sh rather than remote-viewer to have your proxy settings automatically added.

Happy hacking!

Identifying VirtIO disks in RHEV

Fran Garcia — Mon, 17 Apr 2017 14:03:11 GMT

When you have a lot of disks attached to a single VM, it can be cumbersome to identify which VM disk is what inside the VM. For example, you might have several 100 GB disks named vde, vdf, vdg and have no obvious way to properly detect which one is the right one.

Say you want to remove your vdd disk, and have it removed from your RHEV environment.

You can check their VirtIO identifier with:

# find /dev/disk/by-id "(" -name "virtio*" -and -not -name "*part*" ")"  -exec ls -l "{}" +
lrwxrwxrwx. 1 root root 9 Mar  8 09:18 /dev/disk/by-id/virtio-38ebb33e-db1c-4048-b -> ../../vdb
lrwxrwxrwx. 1 root root 9 Mar  8 09:18 /dev/disk/by-id/virtio-4d18901c-1dea-414d-b -> ../../vda
lrwxrwxrwx. 1 root root 9 Mar  8 09:18 /dev/disk/by-id/virtio-69c3688d-f8b3-464c-8 -> ../../vdc
lrwxrwxrwx. 1 root root 9 Mar  8 09:18 /dev/disk/by-id/virtio-b1666304-fba1-44a2-a -> ../../vdd

We can confirm the VirtIO identifier for vdd is b1666304-fba1-44a2 .

Now we need to map this to the human-readable names available in the RHEV Web UI. This can be done by leveraging the URL as follows :

# curl --silent -k -u "admin@internal:password" https://server:port/api/vms/UUID/disks | grep -P 'disk href|alias'

    <disk href="/api/vms/VM-UUID/disks/38ebb33e-db1c-4048-a345-e381abc7f8cc" id="c8aad46c-9373-4670-a345-e381abc7f8cc">
        <alias>MyVMName_Disk3</alias>
    <disk href="/api/vms/VM-UUID/disks/4d18901c-1dea-414d-9cca-811dac616443" id="b7840cff-8f55-4dbb-9cca-811dac616443">
        <alias>MyVMName_Disk2</alias>
    <disk href="/api/vms/VM-UUID/disks/69c3688d-f8b3-464c-87e1-000e19d85c79" id="e9938a29-b792-4f6e-87e1-000e19d85c79">
        <alias>MyVMName_Disk1</alias>
    <disk href="/api/vms/VM-UUID/disks/b1666304-fba1-44a2-9cb1-fb4cfcbc19dd" id="9c2df047-343e-484a-9cb1-fb4cfcbc19dd">
        <alias>MyVMName_Disk4</alias>

The UUID for the instance can be checked in the RHEV Web UI by opening the VMs tab and clicking in the VM name.

Happy hacking! :-)

Red Hat Virtualization to get Hyperconvergence (HCI) support

Fran Garcia — Fri, 31 Mar 2017 22:11:54 GMT

It's been a while since I last touched Red Hat Virtualization (now without the 'E'!). Once thing that picked my eye in this review is the fact that Hyperconvergence is now a thing and will be supported very soon.

The idea behind this is providing ROBOs (remote offices / branch offices) a low-complexity solution to support their computing requirements without investing in difficult to manage technology. But enough with the buzzwords, the technical details are:

Virtualization technology provided by Red Hat Virtualization (RHV) 4.1 .
Software-defined Storage technology provided by GlusterFS .
Automathed installation via the Cockpit software.

The plan is to support up to 3 pods (3, 6 and 9 hosts) configurations (remember this is a ROBO product). As with all beta software, this is subject to change in the final product so keep an eye on the Release Notes.

Here are a few links that will bring further info if this sounds interesting to you:

Happy hacking!