Upgrading “in-place” from PMC 3.x or 4.0.0 to 4.0.2 (deployments in Azure Cloud)

Based on the reports from customers using PMC in Azure Cloud, we have noticed that enabling the Microsoft Defender for Cloud option for the Azure Subscription covering the PMC virtual machine can cause serious problems with PMC appliances. The problems arise because the Microsoft Azure Linux Agent (waagent) software component running in PMC virtual machines in Azure Cloud attempts to install certain VM extensions when the Microsoft Defender for Cloud option is enabled. The extensions occupy several hundred megabytes of storage space on one of PMC storage volumes, where only 2 GB are available by default. Due to the incompatibility of some of the Microsoft packages installed by the extensions with Debian 11 Linux (the base OS of PMC), the system services installed by the incompatible packages write massive amounts of messages to system journal, quickly consuming the remaining storage space. A shortage of storage space can lead to serious system problems.

Following preparation steps are necessary on Azure-hosted PMC VMs prior to in-place update:

·       If PMC VM was not deployed to a dedicated Resource Group, it is advisable to move the VM into a Resource Group dedicated to PMC appliance resources. To move the PMC to a dedicated Resource Group:

o   In the Azure Portal, open the current Resource Group of the PMC appliance and select the resources belonging to your current PMC appliance (at least the Virtual Machine and the four Disks).

o   In the toolbar at the top of the page, click the -> Move button, and select Move to another resource group option.

o   On the Move resources / Source + target page, select the target (or create a new) Resource Group and click Next.

o   On the Move resources / Resources to move page, confirm that the necessary resources are selected and, once the validation completes, click Next.

o   On the Move resources / Review page, select the checkbox to confirm you understand the consequences of the move operation and click the Move button to start the process.

·       Connect to the PMC VM with SSH or open the Serial Console (Azure Portal > PMC VM > Help options > Serial console) and logon to the admin user account which was created during PMC VM deployment.

·       In the Linux shell of the PMC admin user, execute the ‘sudo journalctl --vacuum-size=100M’ command to clean-up the system journal (only leaving 100 MB of the events data, if the size was bigger).

·       Execute the ‘sudo du -ah /var/log/ | grep -v "/var/log/journal" | sort -h -r | head -n 11 | tail -n 10’ command to find the top ten largest files (excluding system journal files) in the /var/log directory. Its output will be like the following:

On PMC appliances which have been running for several months, especially if Azure Defender for Cloud components have been installed, there might be log files with sizes of several hundred megabytes. To truncate such files, execute the following command on each of them (replace <log_file_name> with actual log file name):

echo | sudo tee /var/log/<log_file_name>

·       In the Linux shell of the PMC admin user, execute the ‘df -h’ command to determine the storage device of the 2 GB overlay volume. Its output will be like the following:

In the above example, the 2 GB overlay volume (mounted on /run/live/overlay) resides on the /dev/sdc device. Write this information down for further use. The filesystem from the overlay volume stores the changes made to the underlying root filesystem (which is read-only). PMC 4.0.2 appliances newly deployed to Azure cloud will have the overlay volume size already set to 5 GB. Since increasing the overlay volume size cannot be done from within the VM during the in-place update process, the volume size will need to be manually increased before beginning the in-place update.

·       (Optional but recommended) To allow reverting the changes which will be done in next steps, create a Restore Point of your current PMC VM:

o   In the Azure Portal, open the PMC VM overview page.

o   From the task list on the left, select Backup + disaster recovery > Restore point.

o   Create a Restore Point, which will include all Disks of PMC VM.

·       Increase the size of the overlay volume filesystem:

o   In the Azure Portal, open the PMC VM overview page.

o   From the VM task list on the left, select Settings > Disks.

o   Under Data disks, locate the one which has the 2 GB size. It should be the LUN 0 one.

This disk is the /dev/sdc device from the above example and is mounted on /run/live/overlay.

o   Open the properties of this disk by clicking its name.

o   From the Disk task list on the left, select Settings > Size + performance.

o   Specify Custom disk size of at least 5 GB and select a Performance tier.

 

o   Click the Save button to update the disk.

o   In the Linux shell of the PMC admin user, execute the following command to extend the filesystem on the disk: ‘sudo resize2fs /dev/sdc’. /dev/sdc is the disk device storing the overlay volume (mounted on /run/live/overlay), as determined in previous steps. If needed, change the /dev/sdc disk device path to a path appropriate for your system. Command output should be like the following:

 

o   Execute the ‘df -h’ command again to confirm that the filesystem from the overlay volume has been really resized:

 

·       Create an Azure Policy to prevent the deployment of problematic Azure Defender for Cloud packages. To create the Policy, follow the instructions from the Microsoft Defender for Cloud considerations section above.

·       The created Azure Policy needs to be assigned to the Resource Group containing the PMC VM. To assign the Policy, follow the instructions from the further part of the Microsoft Defender for Cloud considerations section above.

The Defined Policy and the Assignment should prevent the installation of unwanted Microsoft Defender for Cloud extensions, which overconsume the storage space on the overlay volume and flood the system journal. However, if the extensions have already been installed, they will need to be uninstalled manually.

To uninstall the Microsoft Defender for Cloud extensions:

·       In Azure Portal, open the PMC VM overview page.

·       From the VM task list on the left, select Settings > Extensions + applications.

·       If the MDE.Linux and OmsAgentForLinux extensions are listed, click the name of each one. Click the Uninstall button located at the top of the panel that appears on the right-hand side to uninstall the extension. After uninstallation, the Azure Policy defined and assigned in the previous steps will prevent the reinstallation of these extensions.

After completing the above preparation steps, please follow the procedure below to perform an in-place update of the PMC appliance deployed to Microsoft Azure Cloud:

·       In PMC GUI, under Administration > System Updates, click the [Browse] button to locate the .UPD file containing the PMC 4.0.2 update package. Press the [Apply] button to start the upload process.

Note: Separate PMC update packages are available for on-prem and Azure Cloud deployments. Please make sure you have obtained the update package for deployments in Azure Cloud. An attempt to update PMC with incorrect update package will fail.

·       The upload, extraction, and verification of the update package will take a moment. Once the ‘Ready for update’ message appears, press the [Apply update] button to continue, or [No] to stop the process. PMC functionality will be blocked until the update is completed.

·       PMC appliance will be rebooted at the end of the process to apply the changes.