Satellite 6.10 released
Red Hat Satellite version 6.10 has been released! This is a preparatory release for the upcoming Satellite 7.0, where major migrations take place for the new software version.
The official information available in:
Preparing an update
The following steps needs to be taken before upgrading to Satellite 6.10:
-
Ensure you are in the latest Satellite 6.9.z release (6.9.7). This is important as this release relies on having the latest packages to make pulp2 to pulp3 migration feasible.
-
Ensure you have plenty of space in
/var/lib/pulp/published. This is where metadata of each content view is kept (namely, repository metadata). This information needs to be renegated by pulp3 so at some point both versions of the information exists at the same time. If you keep lots of content view versions, it is recommended to purge them prior to starting the process in order to save space (and to generally speed up Satellite operations). -
You can review the pulp migration summary once you are in Satellite 6.9.7 with the following command
foreman-maintain content migration-stats:
# foreman-maintain content migration-stats Running Retrieve Pulp 2 to Pulp 3 migration statistics ================================================================================ Retrieve Pulp 2 to Pulp 3 migration statistics: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache. ============Migration Summary================ Migrated/Total RPMs: 111437/111456 Migrated/Total errata: 41998/41998 Migrated/Total repositories: 115/115 Estimated migration time based on yum content: fewer than 5 minutes Note: ensure there is sufficient storage space for /var/lib/pulp/published to double in size before starting the migration process. Check the size of /var/lib/pulp/published with 'du -sh /var/lib/pulp/published/' Note: ensure there is sufficient storage space for postgresql. You will need additional space for your postgresql database. The partition holding '/var/opt/rh/rh-postgresql12/lib/pgsql/data/' will need additional free space equivalent to the size of your Mongo db database (/var/lib/mongodb/).
In case of problems with missing or broken RPMs, they will be detected as well:
============Missing/Corrupted Content Summary================ WARNING: MISSING OR CORRUPTED CONTENT DETECTED Corrupted or Missing Rpm: 19/111456 Corrupted or missing content has been detected, you can examine the list of content in /tmp/unmigratable_content-20211117-32242-1m0sghx and take action by either: 1. Performing a 'Verify Checksum' sync under Advanced Sync Options, let it complete, and re-running the migration 2. Deleting/disabling the affected repositories and running orphan cleanup (foreman-rake katello:delete_orphaned_content) and re-running the migration 3. Manually correcting files on the filesystem in /var/lib/pulp/content/ and re-running the migration 4. Mark currently corrupted or missing content as skipped (foreman-rake katello:approve_corrupted_migration_content). This will skip migration of missing or corrupted content. [OK] --------------------------------------------------------------------------------
In my test lab, I just ingored those errors as they were some minor issues with some kernel packages.
It is also good to review the sizes of the current MongoDB and PostgreSQL databases. As MongoDB is finally removed, data will be migrated to Postgres and its filesystem should have enough space.
# du -scm /var/lib/mongodb/ 7196 /var/lib/mongodb/ 7196 total du -scm /var/opt/rh/rh-postgresql12/lib/pgsql/data/ 10205 /var/opt/rh/rh-postgresql12/lib/pgsql/data/ 10205 total
Note that you might also need to remove the following legacy RPMs prior to upgrading to Satellite 6.10 . My Satellite was installed in the 6.3 timeframe and for some reason the packages have been lingering around since then. If the packages are present, the installer will issue a message regarding yum being unable to properly resolve dependencies.
yum erase tfm-rubygem-ethon tfm-rubygem-qpid_messaging tfm-rubygem-typhoeus tfm-rubygem-zest tfm-rubygem-typhoeus tfm-rubygem-fog-xenserver tfm-rubygem-pulp_docker_client tfm-ruby
gem-awesome_print tfm-rubygem-trollop
Upgrading the Satellite version
The upgrade process itself doesn't change much from earlier. It will just take more time to accomodate the data migration.
# time foreman-maintain upgrade run --target-version=6.10 -y Checking for new version of satellite-maintain... Security: kernel-3.10.0-1160.45.1.el7.x86_64 is an installed security update Security: kernel-3.10.0-1160.42.2.el7.x86_64 is the currently running version Loaded plugins: foreman-protector, product-id, subscription-manager Unable to upload Enabled Repositories Report Nothing to update, can't find new version of satellite-maintain. Running preparation steps required to run the next scenarios ================================================================================ Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: | Checking repositories enabled on the systemUnable to upload Enabled Repositories Report | Checking repositories enabled on the system [OK] -------------------------------------------------------------------------------- Running Checks before upgrading to Satellite 6.10 ================================================================================ Warn about Puppet content removal prior to 6.10 upgrade: [OK] -------------------------------------------------------------------------------- Check for newer packages and optionally ask for confirmation if not found.: Confirm that you are running the latest minor release of Satellite 6.9 (assuming yes) [OK] -------------------------------------------------------------------------------- Check for HTTPS proxies from the database: [OK] -------------------------------------------------------------------------------- Clean old Kernel and initramfs files from tftp-boot: [OK] -------------------------------------------------------------------------------- Check number of fact names in database: [OK] -------------------------------------------------------------------------------- Check for verifying syntax for ISP DHCP configurations: [OK] -------------------------------------------------------------------------------- Check whether all services are running: [OK] -------------------------------------------------------------------------------- Check whether all services are running using the ping call: [OK] -------------------------------------------------------------------------------- Check for paused tasks: [OK] -------------------------------------------------------------------------------- Check to verify no empty CA cert requests exist: [OK] -------------------------------------------------------------------------------- Check whether system is self-registered or not: [OK] -------------------------------------------------------------------------------- Check to make sure root(/) partition has enough space: [OK] -------------------------------------------------------------------------------- Check to make sure /var/lib/candlepin has enough space: [OK] -------------------------------------------------------------------------------- Check to validate candlepin database: [OK] -------------------------------------------------------------------------------- Check for running tasks: [OK] -------------------------------------------------------------------------------- Check for old tasks in paused/stopped state: [OK] -------------------------------------------------------------------------------- Check for pending tasks which are safe to delete: [OK] -------------------------------------------------------------------------------- Check for tasks in planning state: [OK] -------------------------------------------------------------------------------- Check to verify if any hotfix installed on system: - Checking for presence of hotfix(es). It may take some time to verify. [OK] -------------------------------------------------------------------------------- Check whether system has any non Red Hat repositories (e.g.: EPEL) enabled: / Checking repositories enabled on the systemUnable to upload Enabled Repositories Report / Checking repositories enabled on the system [OK] -------------------------------------------------------------------------------- Check if TMOUT environment variable is set: [OK] -------------------------------------------------------------------------------- Check if any upstream repositories are enabled on system: \ Checking for presence of upstream repositories [OK] -------------------------------------------------------------------------------- Check for roles that have filters with multiple resources attached: [OK] -------------------------------------------------------------------------------- Check for duplicate permissions from database: [OK] -------------------------------------------------------------------------------- Check if system has any non Red Hat RPMs installed (e.g.: Fedora): [OK] -------------------------------------------------------------------------------- Check whether reports have correct associations: [OK] -------------------------------------------------------------------------------- Check to validate yum configuration before upgrade: [OK] -------------------------------------------------------------------------------- Check if checkpoint_segments configuration exists on the system: [OK] -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Validate availability of repositories: / Validating availability of repositories for 6.10 [OK] -------------------------------------------------------------------------------- The pre-upgrade checks indicate that the system is ready for upgrade. It's recommended to perform a backup at this stage. Confirm to continue with the modification part of the upgrade (assuming yes) Running Procedures before migrating to Satellite 6.10 ================================================================================ disable active sync plans: \ Total 0 sync plans are now disabled. [OK] -------------------------------------------------------------------------------- Add maintenance_mode chain to iptables: [OK] -------------------------------------------------------------------------------- Stop applicable services: Stopping the following service(s): rh-mongodb34-mongod, rh-redis5-redis, postgresql, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, foreman.socket, dynflow-sidekiq@worker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy \ All services stopped [OK] -------------------------------------------------------------------------------- Running preparation steps required to run the next scenarios ================================================================================ Check if tooling for package locking is installed: [OK] -------------------------------------------------------------------------------- Running Migration scripts to Satellite 6.10 ================================================================================ Enable applicable services: Enabling the following service(s): pulpcore-api, pulpcore-content, pulpcore-resource-manager, pulpcore-worker@1, pulpcore-worker@2, pulpcore-worker@3, pulpcore-worker@4 | enabling pulpcore-resource-manager Created symlink from /etc/systemd/system/multi-user.target.wants/pulpcore-api.service to /etc/systemd/system/pulpcore-api.service. Created symlink from /etc/systemd/system/multi-user.target.wants/pulpcore-content.service to /etc/systemd/system/pulpcore-content.service. Created symlink from /etc/systemd/system/multi-user.target.wants/pulpcore-resource-manager.service to /etc/systemd/system/pulpcore-resource-manager.service. \ enabling pulpcore-worker@4 Created symlink from /etc/systemd/system/multi-user.target.wants/pulpcore-worker@1.service to /etc/systemd/system/pulpcore-worker@.service. Created symlink from /etc/systemd/system/multi-user.target.wants/pulpcore-worker@2.service to /etc/systemd/system/pulpcore-worker@.service. Created symlink from /etc/systemd/system/multi-user.target.wants/pulpcore-worker@3.service to /etc/systemd/system/pulpcore-worker@.service. Created symlink from /etc/systemd/system/multi-user.target.wants/pulpcore-worker@4.service to /etc/systemd/system/pulpcore-worker@.service. | All services enabled [OK] -------------------------------------------------------------------------------- Start applicable services: Starting the following service(s): rh-mongodb34-mongod, rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-resource-manager, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, pulpcore -worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, smart_proxy_dynflow_core, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@w orker, dynflow-sidekiq@worker-hosts-queue, foreman-proxy \ All services started [OK] -------------------------------------------------------------------------------- Switch support for certain content from Pulp 2 to Pulp 3: Performing final content migration before switching content [OK] Print pulp 2 removal instructions: ====================================================== Migration of content from Pulp 2 to Pulp3 is complete After verifying accessibility of content from lients, it is strongly recommend to run "foreman-maintain content remove-pulp2" This will remove Pulp 2, MongoDB, and all pulp2 content in /var/lib/pulp/ontent/ ====================================================== [OK] -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Upgrade finished.
The whole upgrade process took about 2.5h for a Satellite system with RHEL7 and RHEL8 main repos and about 10 content view versions. Note that this migration time is severly affected by amount of RAM, CPU and storage performance.
Cleaning up
Once Satellite 6.10 has been fully migrated and verified, the old pulp2 content should be removed with the following command:
# time foreman-maintain content remove-pulp2 ; time foreman-maintain upgrade run --target-version=6.10.z -y Running Remove Pulp2 and mongodb packages and data ================================================================================ Remove pulp2: WARNING: All pulp2 packages will be removed with the following commands: # rpm -e pulp-docker-plugins pulp-ostree-plugins pulp-puppet-plugins pulp-puppet-tools pulp-rpm-plugins pulp-selinux pulp-server python-bson python-mongoengine python-nectar python-pulp-common python-pulp-docker-common python-pulp-integrity python-pulp-oid_validation python-pulp-ostree-common python-pulp-puppet-common python-pulp-repoauth python-pulp-rpm-common python-pulp-streamer python-pymongo python-pymongo-gridfs python2-amqp python2-billiard python2-celery python2-django python2-kombu python2-solv python2-vine pulp-katello pulp-maintenance python3-pulp-2to3-migration # yum remove rh-mongodb34-* # yum remove squid mod_wsgi All pulp2 data will be removed. # rm -rf /var/lib/pulp/published # rm -rf /var/lib/pulp/content # rm -rf /var/lib/pulp/importers # rm -rf /var/lib/pulp/uploads # rm -rf /var/lib/mongodb/ # rm -rf /var/cache/pulp Do you want to proceed?, [y(yes), q(quit)] y - Removing pulp2 packages - Removing mongo packages | Removing additional packages - Dropping migration tables | Dropping migrations \ Done deleting pulp2 data directories [OK] -------------------------------------------------------------------------------- real 2m46.147s user 1m32.814s sys 0m17.502s
Happy upgrading!