VCF Migration service engine 9.1.1 Upgrade Issue

Recently, one of our systems was due for a VCF Automation upgrade from version 9.1.x to 9.1.1. What sounded like a routine action dragged on for several hours and ultimately ended with a frustrating error message during the precheck. The experience is worth sharing, as it reveals not only a technical pitfall but also the importance of the correct sequence for VMware Cloud Foundation updates.

The problem typically manifests like this: After starting the upgrade process, the precheck gets stuck. Logs show messages like “PackageDeploymentsCompleteWaitTimeout” – essentially meaning that it waits for ongoing package deployments, but this timeout of over six hours occurs outside the expected timeframe. Additionally, the actual upgrade command for the Migration service engine fails, for example with Reference Code XXXXXXX and the note “VCD_MIGRATOR Precheck Workflow pre-check result status: Failed”. Anyone who subsequently tries to push the VCF Automation upgrade will also fail.

The background is compatibility with a new PostgreSQL version introduced in 9.1.1. The required component sequence for this is: update VCF Automation first, then the Migration service engine. If you click “Upgrade All” in the VCF Operations UI, the system tries to upgrade both components simultaneously – and that is exactly what causes the precheck to crash. A manual upgrade of the Migrator service engine before VCFA will also fail accordingly.

The good news: This scenario is already fixed in the subsequent version 9.1.1.0100. However, if the system is running an older 9.1.1 without this patch release, the following workaround from the Broadcom knowledge base helps.

Troubleshooting

To fix the issue, log into the VCFA appliance via SSH and gain root privileges:

  1. SSH into the VCFA Appliance and switch to the root user:
ssh vmware-system-user@vcfa_fqdn
sudo -i
  1. Now execute the following script, which resets a timeout. The command decodes a Base64 payload and executes it. It should complete with the message “Suspend has held for 70s”. Only when this confirmation appears can the actual upgrade process be safely continued.
 base64 -d <<< '/Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4E//FYxdADkZSf35ZUq9VGwG5P3znvYHDaU+KcJeLtqaY05DcY2Bk1wI6IBIS+NeKyr2jrddVSpU0PXjqJRhBWKLgOHC5QU8qG1ofmFm4s5d6geyIkT+gdSPe2pL+Sm5k0suXI/wAc29yN6/lUAW3fu6EdCnK8/UDal8wf68wtL/TInq8A9BZW+aPquoJAOWOmOOtPjsHf0BxhQJRTW11hEGi4qd/isLPiYRfszrafECnHDW5V0PGO2ttUe2KQjDjw+rE7VAKyT9yzxPI7exKrlBJYYNw7KpG3UyTlkZ5Pg4AccPHh6zkLova0oDl6hj3zvCbq19Yio5hIpWrraxhmNrl8IoxM2N38RztazQ/N40HKIC7JNmIQxKWSDMva0hkrTDc7EZgQqBMgwZ1ln0Nzgy/W1SmBgduobAaJ6+zXAUTYvCv5dSgOXEsl3u1l1G7MHY4EqUyMCwNmLDnCfsN/zG1mcCyaG9Tw3P5Ky6bqWFtKqCbDojZv9q8Nc1ZYVpByHzjcnwNnGzVfgID8Ol1yPQpjEoTHJV3XbKFYJ1J4R664XgY2cOYFctCa3aTkLq7Jvu8h0KoHEb4tl8d98OLANbPskr6X09eAekNw1ZkCgRMUB+C1omJwhvT295Kk93naw84bZyjJhk0nqbSfHAd2zWuf+Y5XoktnNm0VDS8jAStR6EXnWhVTvtiociI9Jw7QBSCzWPlIUg9DSsyspcbGaSIY9wyObXqKxO76nrZHQBflfiDKr9kkWtn25LyjeWzDnaVtAt5ZqvcBf8EGX7KYOw7ICgEPDB1irZ4nZKNPyaws2H3lF5/kmOartQIUHxW6+4coTXj5RYVFSA3I63a34gOLorCOHe6hKQk/gULz1YA/6L0WbtfSlUQJmk7TfFenzja225LlMooToc7hSzxQYX+uvbc/aCXl1RkBZPBSS/VQF+J3Vvhy7uKFOxR8ZPAMvME1Nc7IHw/7+QIVrKK6QSUXY4imS+4zjJtXkye0EEG772/FFixlncEAoG7ZqTmZvfgoTyAkPBDl2xg8QjoSivGKiH0m0mx0b1zBFKFOEMjWrmRWPPgOcUAOxdvXx1R4SiDt8pepH4JZSC/m3O8G6iWWCjiric6mkbdDXA81as36BNtrSHND7yDva7bNHmZSGOFf7LiQl2OWdu1gAjOhcuTOFHo2bUNbG/v1IprwHiEq20LmQm+Wo9yKnzQh9X7IMQcSJDfgrUj1Jv9RMBnhUaP4Bx+nwxMYhfMkSTALey9tO47uljBF3Kx3G+AHJ4DnP5adivNyl//esPr4TJ+TRpI66mSCeufLpvDasPEhGPEt8zshe29SO24N9nq4WMfevSCe0IlsZuUHGtRZWbx/e4pvTeyl9CwTcib00YnxqYFZwYBWvON6FEVRFLqjUu2hdBszc7IR4OfwND8n4+47Z6OJTcKYnL+cCXlkcbT24sBeGSBjNE+icjq65h8VKLpyj1JIU03yP+qBn6fAyd8HpW6XhQ4++Y2pxfDa3K6xYvSSMxlIs+pd+KFFhG8l/wTAx2hsTTNw/PNgIt9saDL/tMV06cBCMqfQ17XjemXD0VnBwwnifpAo27Q+9iTpCW56MP8z2xgZe/tQF+zGyIf2vKF7wtHkf+jPAGkG2IYIChLvIa6/DNl4vO1mmTyAnVV8IMuVabJRBB3SfRkqntVvD6o3tEdfOuQElD10F4ebgms8XAGZf3zDun/cWK17I5XZkEqBjgAAJgyXIeKf6S6wwvc/wQZnOh6Td0KHyRAsr0bOeprVClr8tLfRGFHzvOWS3bMXivjUvJFTlMrSTtIc7qkYVnibMJ8Bo2dcq21cJ2okGfbkEoLxUevV9EyQuyiMWWUavqFUdukXegxF2Ekun3/mcx6fbwOg5imFAPsNls/qCD9PL9f/ZE9Oocay0yR1E+PIGseCuphtYXnhv5ucV+TfKVHykieuIJFhL+qIl0bS/DR9re5Drbx7C5dmqoIec+kEx3ANkmxCBVPRTNpiq+OVzZVtbaJwM6Gj1MHi7upZY9FY2X/t1QgMsXZHlihkM0XN8jmcIs4Nhcc8sPB6A6qCLr8WiBJ6xU6t3RMTjquDbAZ02LWbJPhHUSXzc9OcX++sHFi5IIKvalTBuio/GZ1+vq+aCbaPpmG7x3vT6Oymr5NAqPlhWVf+mlksTrLR5vdspyKchcAo9EeEjIxLg596OFI0ontuPyifNJBEfPC7EreshVNoNFfhiJvquH/Foz6VedItrXTm9/U5JLMGJLZFog7Cdf0NBPHAJEaGyjI23XZ+tROxZ0hJ7y8hDHonq7OGAfNLZpyIx2XlNqGMCIfFQC3NehRcGlbsztwLm+cBqdAo470Sxd5TzLkhkPBR+TklEa8QfJVv/23Fcy2EjtFWdvfbQk8QgotdVsv673YLDneT4MJuI5LoAoMvNC2K2B1o/QP8TwHIOmheu+2OBLJSkNUZuvWeneD8c3ZuCqB9K5IlTR8/8sAGY3kAc94RKZfkZcn0PIMViG8JknOCZVI9s9tJGs0+7W1wi/oX+CqckBe3jPz7A6orHM/XPsCDMIPzQlHdOeMMOF/h/DCrsungIC83i3eLPnIhVrwW7Wtsk3W25HOcCYdbKdfBnoWjgf85vVCVJPnkmmaNqNuztYyyeci+TU07dwoF5PeFUI1xJbR8I6qHFQYXCQZR7RGswwyVYrtF43+yOXml95YtfNo9rmkYkaZQXcakWduirJKSF9vEnoE6LzIZBo4b6tjkxnsZ+/yuozSP0SQnOcqn1F5qWEygYJvF9V12Mz6JeTF0ipwb8Ib9g+ZqDOA4+w+SnOMGdSRi7X77Moz0NeTEdTAXi/ltf08kF3HCGKCDK6sfgSWyHiXRqbzco4N4EI+JLnSIDvgUAdQKobQObbDk5ix+krQSw4+me/yE+2O/cQIXAFDv6Nic1n/FYjR5ktxB2UC8FwGX0A2Hz5D4qytYmUR7oJRtGKZH1ceF/jHjS/qUknq2WKhyy9yYmAp8Q9aAe/bIvhxpNAkc6nZEtJA17ipcNsgNfZ1T7wNOZIA+ZseyB1taLoUfXD2AvMMwsYIlcQqwCdKctFX/nDl440WEXuTjWPNK7cwXDaFzbNbHwdqpCaLpKmobtLCPeRsu/ClCLXcq4AmGob/0clTsBPkoENNLzRBJs9OQ3pT8cevpdKuuHOSSdjQBzz1YCaat/G0nXnSLrfGmVDWM0Ch0Y/DA3CapHJP+q9PV/QNX5SBnZdbA0LLWWKn444K0I8LlN3kS9a24UUY7FdLI4ZfnxW5Eos3UoO44Z+ktSTWAAX0ZpPQa5pt+eh7Yo0p4mgPqpfeG8YfnWtdhLdJwQxoliN0oyXwHNlCQKVI/YBrwwl2BAK0uXHKAHczvSQucp0d/6Ua5ygTLhf/kx3LcdSHusCbhZ0CL1LynMLZqL7vhv6ncn13kNsrhBze/czdlwJVw/X+ny8k9cko2u6GpTPYqAPt8yEEtiSsqgUZJp5/3CdWf4reUC3c05UvmyveP2iv9kHVMZRLH46ZMm4mGfKaNXOCxlbazjUjIQ/QYCQApRVmnF5imeRAMvTcfdFwSUxvEQV9VeVH5MqW0iBScKdQkXzIP5iGgjZm8/FstVUEcftSKhKmlL/tbdOOsfPXYVc4qXg9XqrTHt9HR28DXvb++CNV96T7Voi6mFnTy2X9jgt4w0uKWObXVS/1pwC9mILNPhvtTRGpBUofrL+Y4342/Sb6T8VvPSGmj+JGML0OUkx8k29BPDUMpH9RB5NQ+JVDcCGh0U5B9s7/fdMLzrbxM+6vgIUHn4HLoVu1LwT8yOI3ZWu4pGxOUtmmVBz8mqJQ26p0bbl+R4IxWUP4HEeQ9SZmK8fRx5nks5B1N49S1hggh2VFlgJVlxYAmNVawtr+0RtqP19O9Yj2qpDRxEtmMdxNa3b+V9qBNXlVr7a1xKg5TBiCIY8FKzrQJVw0BbIu6/M4dbfEUB+cBsJENMG4quHOdFzkaNmSvwKFOR+nY6+SsFMSRGnNwHOwaQ1KXprJNIKUMdQwBWx2q6KrENQjQvjoAxcaz5rZhUmRVdmGjj71Fb/BC0YelEO02dfdRSjz9cCXQZ7xCf2dfpEKyfGI3dk9Ft2aZQNfSFyL7OyrSwGvvhPuKYbNL1dXj8V2GinGpBm3dVG5zpHpBGXx2hbsaZA7V0oYo8QFyxWQUQc9nilPM7J9dxvz+6PBdyr90s3o3K3+ZKF9j/16ZGQhyPtZs8tNThCseqsCa2BBDe9jUmqCwBSSSMwlTO/YC+41xEv2I+SWjIDZ/qqNCwQYusCsCYe08qmgrcNsg+MdwYE360uyD5jsDrRxFcDXc9p5nEojnF9JYYB8oPHAUNJtWzBrrKRHtAxTIIOy7Ak65nGjaIBXKp0/ZQRKuMgI7FolfgXbRk7xOeWHcBtN9Ww4tP/JfFmr7qURU9f7iFv9LBZLi+D4Ftd3UaydPACW/CUkNEz11Kgqs6Eh/cVtH2KT9YlMSIgvJrHABPzVNA5u8LJ1Vdb6hL/n7bsV2JhySSxOpQu1fIoG2Ojy3dN0ObA0SdrTwjvTJc1cvr7dbcK5LYzccOlC+jVTvNicl3dP7w6lmrnlBbae092GyFP34nWi+ugkxz+B1lVjaADviNddP3DpB/dmD9VNXyImvi38GV40ESKax9M3eRbSLo/1beByhMdHVUUEzvePx1Ct8QN3WxxMJ/zzQUhi3V5libg3pMEdTAhYJ7ibiSwCujJDo4Um5Imw9JcDWbVBB1jKo0/Ch1FY961LiFdWrsWXgWwhbZDzGegVqIvLLdNF6akvG+lElw8l+kMjih2TWna1eCvxNJ/d4NK3m+fc945qrqv6TX8obRvT2MoivQm1PyiZwk7X8vp81pUyJVyOo7Aj6YsfzmcW6FSceiofwEibEjrafZLrgX9GBVnymxY+WmW2CogT0UvwtVD3CjygJZOv++/QfHqSxuY5Kxt/Iic7L+OQEWo3vnCf7ED9MF5KexSg9rzmpoM8Fukal07frHoRSA62FawVl78jicBRDHMN0gY5o/BNR6BQqPVUJgdsRzwXnwHrfzf1Jb5Jr62o1KVHcaaqkhZWEaBIeJBuFD09muyL2sMApCCp8Mwz0EnAA4ly5QV+51m0wa2N4nQspm6CYxtYkR5ZQtbn3BDWIW93dqQbfhsdWcF33YWUWKdmY2/lvvp4F0LLHkktIBmIvQyj5Gy8ubQuVvU2/gvvay0oHWCxiXA8rGV/HCnv6qCTeeOZgEGdtrHs8FuAHkUsAsfGoT12jRJtlL4DQ0jh6KlLnDZGceQ0Kw6gdxeQGQJ140WBV/XBR+MIWevkYCvZXVEgVOrOXprVpSNAjfqq9wBUJqTvk2NSLcGRnzpNQxk/6GHJOOmbgaF0PSlh4JUFImDApkptJz4ovSOb01/B5FOhHu9FnTlgtKCabnrUYzN05XvRxiHLl5A8KRO+L/JUvXZZ9sGr/oee0oxLncgCAkKl5W8XPcGcAoxusoTkOdTZdLBGvdY4Q62PkYIaYQ5cwU4O620BjtHZX1PhpfrufVR1PoT/aZvnjS5th+ePbDcWOKU5z4CZ1K+KjNCvAQcttvpd+Ftj9+SvjLfUtsQf3zkcHtXyCKJu6otpL8Pgx1zpozPvWS+gcWqcydev0yv59gsOpVanA/GkF9B/AQGJHjjGSIlUWuXh75n9O30uSWvVQFN8AltIzDE6yONFQeQlRJFbskUulfkMqvtj/a8niaVK+FizgZnvyCyFJdD4mMquAgCvdDbKbD2g/Qo5t0ni16GXEfiFLoo4N9Di6C9z+rERRpODqG4X2w965DwLSMMFlMHF50v2TNVVwFz/lDwPPvYZrFOIysSSIKgQmdAmczUlJDxRGrI6LSUKpm6I6AuZw9pz7LnlqXYYsuQ3z5Wb3ob2br1OhMhqF7wN6TsIYdjJb3gO4054wnykw46IaTqs1tIn4lslEGVjMKO0FLqNzCHMnAkS5mBhSDQNjNoEax63Z9lENIUjJ/iMo2/wrciRR9GMGxFrEMZpq0QoMdZdOSLNZmCgv1IuTv8GsU1duBNVlEStYngSbCjDKpUxOYULjL2VryRSib2SOWTcIY+MQxbNYKE3MlbkkX1p8CNJe6+a98+m3k0O/8d2qpYn2xnVdHMPfyHhcVR+ps/PfPWVhB3GTGTCOLD4/VF/nW9R3pQTqslV7CN3rCTvnjLfZfv6rbUh9SiL4zewVToEfWE4KSODuzoGu4NJ8C2x9bWyLOQ+x79Y3AmxssbZR2IxC+PYIbJD9EjQ4FtS677Dw9DHR0EgBJVwiQ1/i/Qd3nS1W3UQoaiqhpEVuaFJFjmeocuBwHe4LPzJB49zADha/uLTVS8wIhN2UtCPg1FozNxQJSbEyZowtsSnxQmLiiOIfT0E2rNyUbPMcKrFnwyPNGnes9ufkkEkuWw4+H/XYYzYAC2gXD0LaA2IlUEAyeqk/BaeUXrqSW9iWCDs/Cdi4Efj3QJBhZMlFDYnlPRHTgQv/nsZ2CDaUVpTRiJ8iT9HdhgiM31ZUdwFP86FGHDypJPbGAnlYk/t8Bp8OuduvOfdUOs1e7clPW5PW28W16BTGGeZtWzDT95tyWXXgmoGeMCUMPeY4SALd3JZC6O0rSa30oPL6PBKwsQghWWq3vcbor8fBNuLE/zQ/gsfOmTIJRo8+PHLNxOjjJzhHEqtfUC4xB9Ux6l6K1MsAoWCt0VgNmA0U9RF61DVVnvsl2vW4RcbtAfv9cM68arodhLE7xlPrSJLmZ2I2AFEny3p0PcEAHlQMGoO6XZb3KRCrCC1cUBH2L2oh1PrTi86qx3DgegfG91eLxyKzxjC0SFS6Q0eTl4yk7jEG3x5lS5tLBRptZbvPifx9jH+CtaXcnL4QFxSpPmDwypE3HZsh3Mx3wczYplYJmpP+2VbQHI8YwNHaWTw5Z4wDrJ3raff76BqbBRVNzdj/jYvgBtLN3IV/DXnTz6d/Fo04bLNcf83gje2xiuui+/yebBSvfOEdd9l5zdumgKz7KWj8/ctxrHMWx2hGQsEP8r8+OeTsu9m9FEnODZiFrf/HJOGV4L+ueRuXjDrtbJuYYrkCsCQjwTd2oixsJ2XMqOCTvLNJALWgHQcUpN8JKAcdeJE3hasIn2bz5tCRXRO6FtaQToBCP6/rPyt4hTJ1EodzCqPRFj+lNYkj1htXSPV7Ryb5IW6o52Psay4JBK4DiBxnFfNovY6nrDbgvkXrlGLJqhjlX3sLXVpfv8YslNJkY+Wuebd6ZjlH5lR57TEMvOU7jqXmmBZuuYuFmmR0YsO9wtWXipHyWFoKf6iFuGZ10mmtlPil4qirFsbciKzr4O7TpEQYCrBH1APDMAAHpapaXGFlmdAAGoK4CgAQBRgzGtscRn+wIAAAAABFla' | tar -xJO | bash

It should end with “Suspend has held for 70s”.

If it says “REMEDIATION INCOMPLETE” or refuses because the failure doesn’t match, stop and investigate before continuing.

  1. Go to the Fleet Lifecycle Management UI and retry/re-run the “VCF Automation” component upgrade. Its backup precheck should now pass.

  2. Once that finishes, go to the Fleet Lifecycle Management UI and retry the “Migration service engine” upgrade.

Reference

https://knowledge.broadcom.com/external/article?articleNumber=455559

You might also like

rchlt

© 2026

LinkedIn GitHub PayPal