Introduction
About Qemu
Qemu (short for Quick Emulator) is a free and open-source hosted hypervisor that performs hardware virtualization.
Qemu is a hosted virtual machine monitor: It emulates CPUs through dynamic binary translation and provides a set of device models, enabling it to run a variety of unmodified guest operating systems. It also can be used together with KVM in order to run virtual machines at near-native speed (requiring hardware virtualization extensions on x86 machines). Qemu can also be used purely for CPU emulation for user-level processes, allowing applications compiled for one architecture to be run on another.
(Taken from Wikipedia)
The Project
Live Migration is a process of moving a running virtual machine from one physical system to another. There are usually two ways, precopy and postcopy. Postcopy migration is used for virtual machines which have a high memory dirtying rate, (higher than the bandwidth of the network that will be used to perform migration). The disadvantage of postcopy migration is, if there is a network failure then we are left with an inconsistent vm at both the source and the destination side.
The aim of this project is to recover from the network failure, and complete the migration process. The project will include completing the process of sending the leftover memory pages.
Project would start by making sure that the threads do not fail at both the source and destination end. This would be followed by re-establishing the network connection between the source and the destination. After this is done, the migration process is restarted. The vCPUs on the destination side were suspended due to page faults, so these pages are transferred first. Followed by the rest of the leftover pages on the source side.
How to Use
Once the network fails, wait for both the sides to error out and go into recovery. This can be understood by the error messages printed at the terminal.
Once both the sides are under recovery, the reconnection can be tried. First in destination side console, enter the migrate_incoming command with the new -r flaga along with the same details as used in the beginning.
Then, same goes for the source side. Use the migrate command with the new -r flag and similar details as used in the beginning.
Work Done
(All commits can be found here)
Detailed explanation
Commit 1
Migration: Reconnect network in case of network failure during pc migration (source)
The migration thread on the source side is kept alive (if the network failure happens during a postcopy migration). The thread waits for the reconnection to take place.
When the user enters the (newly created) reconnection command at the source side, it triggers the code to try to establish a connection to a destination side. After the connection is established, it is tested whether there is an ongoing migration waiting for reconnection or not. If there is, then the waiting thread is awaken (using mutexes). The waiting thread continues with its migration process with the newly created connection.
Commit 2
Migration : General additions for migration recovery
The migration recovery needs mutexes to signal a sleeping thread. Also, after the recovery, the older error needs to be cleared, for the migration to continue normally.
Commit 3
Migration: Reconnect network in case of network failure during pc migration (destination)
The migration thread on the kept side is kept alive (if the network failure happens during a postcopy migration). The thread waits for the reconnection to take place.
When the user enters the (newly created) reconnection command at the destination side, it triggers the code to try to establish a connection to a source side. After the connection is established, it is tested whether there is an ongoing migration waiting for reconnection or not. If there is, then the waiting thread is awaken (using mutexes). The waiting thread continues with its migration process with the newly created connection.
Commit 4
Migration: New bitmap for postcopy migration failure
Now that the connection is established between the source and the destination, they start with their normal task of sending and receiving pages. But, there are some memory pages that has to be recovered. When the network failure would have happened, source would have send some pages that destination would not have received. These pages would not be sent again, since source thinks that it has already sent them. Therefore, to keep track of these lost pages, a new bitmap has to be added.
The new bitmap resides on the destination side and keeps track of all the pages receieved. The bitmap is updated everytime a page is received.
Commit 5
Migration: Recovering pages lost due to n/w failure during pc migration (source)
The pages that were lost due to the network failure has to be sent. This is triggered only if the migration has gone through a reconnection phase. If the migration has gone through a reconnection phase, then the source sends a command to the destination, to request for the lost pages.
Commit 6
Migration: Recovering pages lost due to n/w failure during pc migration (destination)
Once the destination receives a command to request for the lost pages, it creates a thread which goes through the bitmap (which was keeping track of the pages being received) and requests for pages that were not received.
Challenges and Work left to be done
One of the biggest challenge was to resend the lost pages (due to network failure). Dr David Alan Gilbert (my mentor) and I have had many discussion about this during and before the project. There was not a definite answer to this problem. Should we resend a chunk of pages that were immediately sent before the network failure? Should we try to restart the VM back at source? In the end, we ended up deciding, that an extra bitmap on the destination side would be the best way to go about solving this issue.
There were a lot of unseen and unexpected things that popped up during the coding of this project. Like the return value that the migration socket would give in case of a network failure. During testing, the migration thread returned different errors at different scenarios. It became extremely difficult to understand which one to look for. Since there are other failures for which the migration should really fail and not go into recovery. For now, we have settled for an error that seems to be working. But, some amount of testing has to be done to make sure that it works in all the scenarios
Another issue that slowed down the project is, just after reconnection, the VM's uses to suddenly fail and go into recovery again. This issue still persists and needs to be looked into. Most probably there is a subtle bug in the reconnection code that is causing this.
Useful Links
Conclusion
I would like to thank my mentor Dr David Alan Gilbert for the support and the help he provided. He was always there for me, no matter what the problem was. It would not have been possible without him.
I learned a lot from this project. As such, I have all the intentions of continuing my contribution to the qemu community
Contact
In case there are any queries, send an email to haris.phnx@gmail.com; or drop by the chatroom.