Fixing yum incomplete transactions

I just found an issue with a server that wouldn't let me install any additional software with yum because it complained of an incomplete transaction. Plus checking the list of installed RPMs with rpm -qa showed that many of them where duplicated (old and new versions at the same time).

I used the following commands to get out that mess, and worked beautifully :

package-cleanup --dupes | grep -v Loaded | awk 'NR % 2 == 0' | xargs -n1 rpm -e --nodeps --justdb
yum update
yum-complete-transaction
yum -y reinstall kernel

The first command is the meaty part; get a list of duplicate RPMs and, with the awk command get one out of two lines. Then pipe that list to rpm so we can mark the RPM as removed without running their removal scripts, or actually removing the files from disk !

After that, running yum update will take care of ensuring we didn't remove any dependency and finally yum-complete-transaction will replay the last failed command.

The last command is optional but apparently is a common problem to end up with a borked kernel and/or initrd, so reinstalling it for good measure.

As a sample of what yum and yum-complete-transaction where complaining about, here's a small excerpt of what was going on at the time of the issue:

--> Restarting Dependency Resolution with new changes. 
--> Running transaction check 
--> Finished Dependency Resolution 
Error: Trying to remove "systemd", which is protected 
Error: Trying to remove "yum", which is protected  
 You could try using --skip-broken to work around the problem   
** Found 43 pre-existing rpmdb problem(s), 'yum check' output follows: 
audit-libs-2.7.6-3.el7.i686 is a duplicate with audit-libs-2.6.5-3.el7.x86_64   
32:bind-license-9.9.4-50.el7.noarch is a duplicate with 32:bind-license-9.9.4-37.el7.noarch
7:device-mapper-1.02.140-8.el7.x86_64 is a duplicate with 7:device-mapper-1.02.135-1.el7.x86_64
7:device-mapper-libs-1.02.140-8.el7.x86_64 is a duplicate with 7:device-mapper-libs-1.02.135-1.el7.x86_64  
dracut-033-502.el7.x86_64 is a duplicate with dracut-033-463.el7.x86_64
e2fsprogs-libs-1.42.9-10.el7.x86_64 is a duplicate with e2fsprogs-libs-1.42.9-9.el7.x86_64  
fipscheck-lib-1.4.1-6.el7.x86_64 is a duplicate with fipscheck-lib-1.4.1-5.el7.x86_64  
glib2-2.50.3-3.el7.x86_64 is a duplicate with glib2-2.46.2-4.el7.x86_64
glibc-2.17-196.el7.i686 is a duplicate with glibc-2.17-157.el7.x86_64  
glibc-common-2.17-196.el7.x86_64 is a duplicate with glibc-common-2.17-157.el7.x86_64  
gobject-introspection-1.50.0-1.el7.x86_64 is a duplicate with gobject-introspection-1.42.0-1.el7.x86_64
kbd-misc-1.15.5-13.el7.noarch is a duplicate with kbd-misc-1.15.5-12.el7.noarch
kernel-tools-libs-3.10.0-693.el7.x86_64 is a duplicate with kernel-tools-libs-3.10.0-514.el7.x86_64
krb5-libs-1.15.1-8.el7.x86_64 is a duplicate with krb5-libs-1.14.1-26.el7.x86_64   
libblkid-2.23.2-43.el7.x86_64 is a duplicate with libblkid-2.23.2-33.el7.x86_64
libcap-2.22-9.el7.i686 is a duplicate with libcap-2.22-8.el7.x86_64
libcom_err-1.42.9-10.el7.x86_64 is a duplicate with libcom_err-1.42.9-9.el7.x86_64  
libcurl-7.29.0-42.el7.x86_64 is a duplicate with libcurl-7.29.0-35.el7.x86_64  
libdb-5.3.21-20.el7.i686 is a duplicate with libdb-5.3.21-19.el7.x86_64
libgcc-4.8.5-16.el7.i686 is a duplicate with libgcc-4.8.5-11.el7.x86_64   

Happy hacking!

Configuring proxy settings in lftp

I sometimes need to upload data to Red Hat's dropbox service, and most of the times I need to go through a proxy of some sort. Here's a quick note on how to configure lftp to use such a proxy.

# lftp
lftp :~> set ftp:proxy http://USER:password@yourproxy:8080
lftp :~> open 209.132.183.100
lftp 209.132.183.100:~> user anonymous
Password: 
lftp anonymous@209.132.183.100:~> cd /incoming
cd ok, cwd=/incoming
lftp anonymous@209.132.183.100:/incoming> put yourfile.tar.gz -o YOURCASENUMBER-yourfile.tar.gz

As you can see:

  • I used the set ftp:proxy command to configure the proxy.
  • Then I used 209.132.183.100 rather than dropbox.redhat.com . Some proxies do not work if using the DNS hostname, for some reason.
  • After that, I blindy changed into /incoming . It's the only directory with allowed wrting permissions
  • Finally I used put -o to specify the destination filename.

Happy hacking!

Satellite 6 Maintenance tools

Almost missed this release. A few days ago, the Satellite 6 Maintenance tools were released. These are tools that will ease the migration to Satellite 6.3 once is released, as well as one of my favourite BZs for Satellite: BZ 1480346 Tool to change Satellite hostname

They can be installed with :

subscription-manager repos --enable rhel-7-server-satellite-maintenance-6-rpms
yum install rubygem-foreman_maintain satellite-clone

Note the repository above is part of RHEL 7, not Satellite product.

Also bear in mind this is a very first iteration of these tools; I'd expect they will be further enhanced before the Satellite 6.3 GA release. At the moment they are in a BETA state :-)

You can read the official notes here:

Happy hacking!

Gerrit for dummies and Software Factory

Last Thursday I attended a great presentation by Javier Peña, an introduction to Gerrit. In no particular order, this are the things that catched my eye:

  • Gerrit is a code review system. It sits between your source code repository (Git) and your developers, so any proposed change can be better discussed and reviewed. In parallel, Gerrit can be instructed to launch integration tests to ensure the proposed change doesn't break anything. If the CI integration tests are successfull, the patch can be merged with the code. Otherwise, further changes can be done by the submitter with git commit --amend until the patch is succesfully merged (or discarded).
  • It is better than Github's review workflow because:
    • It's not propietary ;-)
    • Can work in a disconnected fashion with Gertty
    • Reviewing dozens of commits for a single PR can get confusing fast, specially with large patches.
    • As mentioned earlier, Gerrit can be easily integrated with several CI systems.
  • Important opensource projects such as Openstack and oVirt are using it, although it started at Google.

There is project that is integrating Git, Gerrit, Zuul, Nodepool and a bunch of other tools to make development easier. It's called Software Factory and you can find additional info in their documentation: https://softwarefactory-project.io/docs/.

Software Factory logo

Happy hacking!

Satellite 6: Configuring vlan tagging and bonding deployments

One of the features provided by Satellite 6 is both provisioning of hosts (via PXE or other methods), and automated network configuration. We can use Satellite to automatically configure our bonds and vlan tagging for us.

Let's review how this all is done using the Satellite web UI. My suggestion is to configure these details one by one, and examine how the generated kickstart changes in each step. This will help us identifies issues and mistakes before starting a lenghty trial and error process provisioning physical servers :-)

The kickstart rendered template is available under https://SATELLITE/unattended/provision?hostname=your.host.name .

VLAN tagging

This is the simplest scenario; we just want to configure vlan tagging in an existing interface.

Imagine we want to configure the following interfaces:

eth0: PXE (no specific configuration mentioned here)
eth1: Base NIC for vlan tagging
eth1.666 : 192.168.0.10/24 , using vlan 666.

We need to ensure that:

  • We have configured a domain.
  • We have configured a subnet, and is attached to that domain.
  • The network is configured to use Static boot mode (this is a personal preference of mine -- I'd prefer all my interfaces to become up regardless the availabity of a DHCP capsule).

Once we perform that, we can perform a server discovery and start editing the network interfaces with the relevant info.

So we'd configure the following interfaces in Satellite :

  • eth0:
    • Mac address: <autodiscovered>
    • Device identifier: eth0
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: <blank>
    • Managed: true
  • eth0.666:
    • Mac address: <blank>
    • Device identifier: eth0.666
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: 192.168.0.10
    • Managed: true
    • Virtual: true
    • Attached device: eth0
    • Tag: 666

It's important that you configure the eth0 interface itself; otherwise when eth0.666 is enabled, it'll fail because the parent device isn't ready.

All in all, your generated configuration should look like :

####### parent device #######
# eth0 interface
real=`ip -o link | grep 00:50:56:04:1a:8a | awk '{print $2;}' | sed s/:$//`

# ifcfg files are ignored by NM if their name contains colons so we convert colons to underscore
sanitized_real=$real

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-$sanitized_real
BOOTPROTO="none"
IPADDR=""
NETMASK="255.255.255.0"
GATEWAY="172.16.16.1"
DEVICE=$real
HWADDR="00:50:56:04:1a:8a"
ONBOOT=yes
PEERDNS=no
PEERROUTES=no
EOF


###### vlan tagging #######
# eth0.666 interface
real=`ip -o link | grep 00:50:56:04:1a:8a | awk '{print $2;}' | sed s/:$//`
  real=`echo eth0.666 | sed s/eth0/$real/`

# ifcfg files are ignored by NM if their name contains colons so we convert colons to underscore
sanitized_real=$real

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-$sanitized_real
BOOTPROTO="none"
IPADDR="192.168.0.10"
NETMASK="255.255.255.0"
GATEWAY="192.168.0.1"
DEVICE=$real
ONBOOT=yes
PEERDNS=no
PEERROUTES=no
VLAN=yes
EOF

Bonding

In the same way as before, we need to configure the underlying interfaces before we configure the bonded one.

eth0: PXE (no specific configuration mentioned here)
eth1: Bond slave
eth2: Bond slave
bond0: Active-Passive bond enslaving eth1 and eth2

For this example we'll be configuring eth1 and eth2 as a slaves of bond0, that will have an IP assigned to it. It is very important you configure both bond slaves first, then the bond interface. Otherwise the bond won't be properly linked to the slaves and the template won't properly generate the kickstart.

  • eth1:
    • Mac address: <autodiscovered>
    • Device identifier: eth1
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: <blank>
    • Managed: true
  • eth2:
    • Mac address: <autodiscovered>
    • Device identifier: eth2
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: <blank>
    • Managed: true
  • bond0:
    • Type: bond0
    • Mac address: <none>
    • Device identifier: bond0
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: 192.168.0.11
    • Managed: true
    • Bond configuration:
      • Mode: Active-Backup
      • Attached devices: eth0,eth1
      • Bond options: ""

The generated config looks like :

# bond0 interface
real="bond0"
cat << EOF > /etc/sysconfig/network-scripts/ifcfg-$real
BOOTPROTO="none"
IPADDR="172.16.16.230"
NETMASK="255.255.255.0"
GATEWAY="172.16.16.1"
DEVICE=$real
ONBOOT=yes
PEERDNS=no
PEERROUTES=no
DEFROUTE="no"
TYPE=Bond
BONDING_OPTS=" mode=active-backup"
BONDING_MASTER=yes
NM_CONTROLLED=no
EOF



# eth1 interface
real=`ip -o link | grep 00:50:56:04:1a:8f | awk '{print $2;}' | sed s/:$//`

# ifcfg files are ignored by NM if their name contains colons so we convert colons to underscore
sanitized_real=$real

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-$sanitized_real
BOOTPROTO="none"
DEVICE=$real
HWADDR="00:50:56:04:1a:8f"
ONBOOT=yes
PEERDNS=no
PEERROUTES=no
NM_CONTROLLED=no
MASTER=bond0
SLAVE=yes
EOF



# eth2 interface
real=`ip -o link | grep 00:50:56:04:1a:90 | awk '{print $2;}' | sed s/:$//`

# ifcfg files are ignored by NM if their name contains colons so we convert colons to underscore
sanitized_real=$real

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-$sanitized_real
BOOTPROTO="none"
DEVICE=$real
HWADDR="00:50:56:04:1a:90"
ONBOOT=yes
PEERDNS=no
PEERROUTES=no
NM_CONTROLLED=no
MASTER=bond0
SLAVE=yes
EOF

Bonding + VLAN tagging

In this final example we want to configure a bond and add different vlan-tagged interfaces to it :

eth0: PXE (no specific configuration mentioned here)
eth1: Bond slave
eth2: Bond slave
bond0: Active-Passive bond enslaving eth1 and eth2
bond0.666: Interface in vlan 666 (192.168.6.6/24)
bond0.777: Interface in vlan 777 (192.168.7.7/24)
  • eth1:
    • Mac address: <autodiscovered>
    • Device identifier: eth1
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: <blank>
    • Managed: true
  • eth2:
    • Mac address: <autodiscovered>
    • Device identifier: eth2
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: <blank>
    • Managed: true
  • bond0:
    • Type: Bond
    • Mac address: <none>
    • Device identifier: bond0
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: 192.168.0.11
    • Managed: true
    • Bond configuration:
      • Mode: Active-Backup
      • Attached devices: eth0,eth1
      • Bond options: ""
  • bond0.666:
    • Type: interface
    • Mac address: <blank>
    • Device identifier: bond0.666
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: 192.168.6.6
    • Managed: true
    • Virtual: true
    • Attached device: eth0
    • Tag: 666
  • bond0.777:
    • Type: interface
    • Mac address: <blank>
    • Device identifier: bond0.777
    • DNS name: <none>
    • Domain: your.domain
    • Subnet: your-subnet-with-Static-bootproto
    • IP Address: 192.168.7.7
    • Managed: true
    • Virtual: true
    • Attached device: eth0
    • Tag: 777

Happy hacking!

First steps with Infoblox

Infoblox produces some appliances that do DNS/DHCP management, full network IPAM management and so on. Since I needed to so some usage of their APIs I've had to set up an infoblox appliance and here I'm jotting down some of the steps I took for easier reference.

The overall steps are:

  • Download the appliance from www.infoblox.com
  • Deploy on your favourite virtualization system, ej KVM.
  • Start the VM, and ensure the cpu and memory prerequisites are set.

If you use Vmware/vCloud and the OVA you'll probably have most network and password details prompted upon when deploying the appliance so in that regard is a bit more straight forward to deploy.

Once your VM is started, you can log in with :

user: admin
pass: infoblox

The first step is configure the network, which can be done with:

Infoblox > set network
Enter IPv4 address [Default: 172.16.16.102]: 
Enter netmask [Default: 255.255.255.0]: 
Enter gateway address [Default: 172.16.16.1]: 
NOTICE: Additional IPv6 interface can be configured only via GUI.
Become grid member? (y or n): n

 New Network Settings:
  IPv4 address:         172.16.16.102
  IPv4 Netmask:         255.255.255.0
  IPv4 Gateway address: 172.16.16.1

        Is this correct? (y or n): y

And now the most confusing thing for new starters is getting your licenses right. If you're using an evaluation license, you don't need to register in the infoblox website, but rather have the appliance generate some 60-day evaluation licenses.

You'll need more than one. They can be checked as below :

Infoblox > show license all
Public IP       License Type                Kind      Exp Date   Replaced Hardware ID             License String

To generate the evaluation licenses:

Infoblox > set temp_license

  1. DNSone (DNS, DHCP)
  2. DNSone with Grid (DNS, DHCP, Grid)
  3. Network Services for Voice (DHCP, Grid)
  4. Add DNS Server license
  5. Add DHCP Server license
  6. Add Grid license
  7. Add Microsoft management license
  8. Add vNIOS license
  9. Add Multi-Grid Management license
 10. Add Query Redirection license
 11. Add Response Policy Zones license
 12. Add FireEye license
 13. Add DNS Traffic Control license
 14. Add Cloud Network Automation license
 15. Add Security Ecosystem license
 16. Add Flex Grid Activation license

Select license (1-16) or q to quit: 1

This action will generate a temporary 60-day DNSone license.
Are you sure you want to do this? (y or n): y
DNS temporary license installed.
DHCP temporary license installed.

Temporary license is installed.


The UI needs to be restarted in order to reflect license changes.
Restart UI now, this will log out all UI users? (y or n):y

You will need to repeat this process a bunch of times until all required licenses are in place. As a guideline, this are the licneses I needed to build a working appliance :

Infoblox > show license
Version         : 8.1.2-356916
Hardware ID     : 42140354685089f1cdccff04ff9cec5d

License Type    : DNS
Expiration Date : 11/27/2017
License String  : EwAAAEdPGOmqWmv1aFgZbs+JuxsU6WM=

License Type    : DHCP
Expiration Date : 11/27/2017
License String  : FAAAAEdJCOXkWyS7bRNXbM6P8ARA7mYv

License Type    : vNIOS (model IB-VM-820)
Expiration Date : 11/27/2017
License String  : GgAAAFVPAvrrFSX0IxYcIsyO9k8K7nUvmR1TaVew

vNIOS: CPU cores detected: 2 - [License allows: 2]
vNIOS: System memory detected: 4096MB - [License allows: 7168MB]

License Type    : Grid
Expiration Date : 11/27/2017
License String  : GgAAAEZPH/DqGWWuLEFXbM3C9U8K7WEsnEUFal64

Only once your subscriptions are properly attached your web interface will become available under https://your.appliance.ip .

For the Cisco-oriented people, the appliance CLI is somewhat similar to some Cisco devices. Is specially useful the show tech-support command that will show all low-level configuration and status.

Happy hacking!

Configuring an iscsi volume for RHV usage

This is a cheatsheet to quickly configure storage and export it as an iSCSI volume using RHEL7 targetcli and have it configured under RHV. This is by no means a production configuration as your RHEL7 system might become a single point of failure, but it convers nicely building a home lab or test environment.

Just for clarity, a quick reminder of iSCSI concepts:

  • An iSCSI target provides some storage (here called server),
  • An iSCSI initiator uses this available storage (here called client).

Prerequisites

Configure server's storage

You can configure several types of backends, and for me the most versatile is using LVM's Logical Volumes. You'll need to create your volumes in advance, for example:

lvcreate yourVG -n yourLV1 -L 50G

Install software

Install the targetcli RPM:

yum install -y targetcli targetd

Enable the targetd daemon

systemctl enable --now targetd.service

Gather RHV configuration

You'll need to gather the following information from RHV:

  • IQN (iSCSI identifier)

Configure and enable iSCSI

targetcli provides a very simple way to create iscsi targets once you understand how it works. Namely what needs to be done is:

  • Add your backend devices. This is where you add into targetcli's control the LVM devices created in previous steps
  • Create an IQN target. This is a collection of luns shared to the same system(s) under the same group. It is used later to apply ACLs so only certain hosts can use certain LUNs.
  • Add LUNs into your IQN target. After creating your IQN target, you need to add the backstore devices so they're shared via iSCSI.
  • Add ACLs into your IQN target. Unless configured otherwise, LUNs are not visible to systems unless they're added into the right ACL.

Here is a dump on how all this can be accomplished with targetcli:

Add storage into RHV

foo

Satellite 6.2 : Pruning mongodb database

Satellite 6 uses a MongoDB database to keep track, among other things, which RPMs compose a certain content view, etc. As the Satellite gets used and CVs are created and deleted, it can leave some old data around.

A way to trim that data can be achieved with the MongoDB built-in tools, mongod. Note that you will need as much free space as your database uses at the moment, plus a couple of GB. You can check the MongoDB documentation for more details:

https://docs.mongodb.com/v2.6/faq/storage/#how-do-i-reclaim-disk-space

This can help improve content view promotion time among other tasks.

The process would be:

  • Stop Satellite services with katello-service stop .
  • Backup your MongoDB ; a tar would suffice or use katello-backup .
  • Launch the repair: mongod --dbpath {{ mongodb_path }} --repair
  • Start services with katello-service start .

This can reduce MongoDB database size up to 50% and halve publication time :-)

Happy hacking!

Tales from the Field: Updating RHEV 3.5 to RHV 4.1

Welcome to the first post in the Tales from the Field series. In this section I indent to collect the most useful stories about different products and how they are being used in every day situations.

For a starter, I'll cover the new RHV suite, now on version 4.1, and the upgrade path to have an older environment migrated to the latest version.

RHV fundamentals

RHEV is now simply known as Red Hat Virtualization and it has several components:

  • The RHEV Manager, a system that acts as a central configuration and orchestration point for all hypervisors.

  • Several types of hypervisors:

    • RHEL-6 hypervisors. Used primarily in the RHEV 3.x suite.
    • RHEL-7 hypervisors. Used in the latter RHEV 3.x and 4.x suites.
    • RHEV-H 6 (vintage) hypervisors. Used primarily in the RHEV 3.x suite.
    • RHEV-H 7 (vintage) hypervisors. Used in the latter RHEV 3.x suite.
    • RHEV-H 7 (ngn) hypervisors. The new hypervisor in RHEV 4.x suite.

Standard RHEV to RHV upgrade path

Red Hat has put together some documentation and labs that should ease the upgrade process:

The 3.6 to 4.0 step is probably the one that requires most planning, as it involves rebuilding the manager with a new version of the underlying operating system (RHEL6 to RHEL7).

Shortcutting the upgrade

If I'm running a 3.5 cluster I surely do not need to install/upgrade every version of RHEV to end up in the latest version, do I?

Unfortunately, yes. But there are some tweaks that can be done in the process to ease out the migration.

  • If you are running a 3.5 cluster with either EL6 or EL7 3.5 hypervisors, you'll need to reinstall them. RHV4 uses the next generation node (NGN) hypervisors which are not upgradable from previous versions of RHEV-H. The good news is it's now available a 3.6 NGN version of the hypervisor. So you can reinstall some of your existing 3.5 hosts with 3.6NGN, and that's the only reinstallation you will need.
  • Once you have updated your manager to 3.5.latest, then to 3.6 you are in a position to start reinstalling your hypervisors with 3.6 NGN. Once that part is done, it is required you change your cluster compatibility settings to 3.6 and recommended you reboot all your VMs. There is an ongoing buzilla BZ#1474870 that should address documenting what is the official recommendation regarding VM reboot on an upgrade scenario.
  • Once you are in RHEV 3.6 Manager with 3.6 compat level, you need to backup your engine database and keep in a safe place. If you're migrating to a Hosted Engine from a standalone Engine, you'll need it to restore the data. If your plan is to reinstall your standalone Engine with RHEL7, it's the moment to do so.
  • Once your Engine is available, you'll run a restore then an upgrade so it's upgraded to the latest 4.0 version .
  • If your 3.6 hypervisors have a configured RHSM/Satellite subscription, you'll be able to launch a 3.6 to 4.0 upgrade from the Manager itself. This greatly eases the upgrade as no manual provision is required.
  • After all hypervisors have been migrated to 4.0, change cluster compatibility to 4.0 .
  • Rinse and repeat for Manager & host upgrades to 4.1 .

Happy hacking!

Satellite 6.2 : Adding RHV4 compute resources

Satellite server allows to manage compute resources such as Vmware and Red Hat Virtualization (RHEV/RHV). In the later RHV versions there's a caveat on how they should be added into Satellite.

  • The RHV certificate file has changed location:

http://rhv4-fqdn//ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA

  • The API endpoint should be specified with a 'v3' tag, as follows :

https://rhv4-fqdn/ovirt-engine/api/v3

Happy hacking!