Attacking additional virtualisation infrastructure

10 02 2015

I should start by stating that in almost all documents regarding live migration, High Availability (HA) and Fault tolerance configurations, it is stated that these networks should be placed on isolated networks that are not shared or accessible by unauthorised individuals. In an article by (Siebert, 2011) it specifically states to:

“Keep management and storage traffic, for instance, on a physically isolated network that’s away from the regular VM network traffic”

However, this does not diminish the argument that this information is still being moved inside the environment and should these guidelines be overlooked or the networks breached, the consequences of what can be achieved can be unintelligible to staff only familiar with traditional methods.

Live migration traffic

Live migration can be used to move machines between hosts that are located in the same blade chassis, across campuses and even over different continents, with minimal downtime (Travostino, et al., 2006). The action of preforming a live migration from one host to another is a common feature found in many of the larger VMMM such as XEN and VMware. The act of moving a machine across hosts in the VMware environment can be a manual or automated process when combined with other features such as DRS and DPM (Distributed Power Management). While explanations of the DRS feature is discussed in numerous earlier posts, DPM enables vCenter to automate the movement of machines off hosts during quiet periods of usage, allowing them to be powered down thus improving power consumption. It is my personal option that live migration is one of the more intriguing introductions of virtualisation, as the information that is located in RAM contains such important information.

Using the test environment I will demonstrate what is possible when access to the vMotion network is obtained. For this example a vMotion has been initiated specifying that VM1 on ESX1 is to be migrated – and only the host portion is required to move and not the datastore. The VM will be moved to the only other host in the cluster, ‘ESX2’. To understand how this process is vulnerable to attacks I will first breakdown the processes of the steps involved in this operation (Kutz, 2007):

  1. The request is made specifying the VM and which host it will be moved to.
  2. All of the RAM of VM1 is copied over the vMotion network to ESX2, any active changes happening during this time on ESX1 are written to a memory bitmap on ESX1.
  3. VM1 is quiesced on ESX1 and the memory contained in the bitmap is copied over the vMotion network to ESX2.
  4. VM1 is started on ESX2 and access to the VM are sent to the instance on ESX2.
  5. Remaining memory from VM1 is copied from ESX1, while memory is being read and written from VM1 on ESX1.
  6. Once successful, VM1 is unregistered on ESX1 and the task is complete.

In figure 1 we see a basic visual representation of the process in the test environment. For demonstration purposes and ease of result display, I have a separate vMotion network that uses a hub to connect the two machines. This will allow me to sniff traffic without having to perform an additional attack such as MiTM or configure port mirroring, port spanning, SPAN, RSPAN etc.

Basic visual representation of the process of a machine being migrated to another host in the test environment

Basic visual representation of the process of a machine being migrated to another host in the test environment

In the virtual machine that will be moved, I have written information into a text file but not saved it to disk, so that we can be sure it will be located in the machines memory. This information in the example is a representation of a secure document containing a username and password. This text file and content are shown in Figure 2.

An unsaved text file that has been written on the running virtual machine

An unsaved text file that has been written on the running virtual machine

I now initiate a vMotion request and starts a packet capture on the vMotion network using Wireshark. The network card is configured in promiscuous mode, to allow it to capture all of the packets being transmitted on the network. After the vMotion process has finished and the capture stopped, I am able to follow the TCP stream of the communication that took place between the two hosts and view the information in a searchable form. As is shown in Figure 3, the information transmitted in the document is viewable in clear text, although separated by some punctuation

A TPC stream of communication between the two hosts showing the notepad content

A TPC stream of communication between the two hosts showing the notepad content

Although this example used a notepad document to demonstrate the ability to sniff data, there are much greater risks to consider when addressing passive snooping attack on migration traffic. Microsoft stores LM authentication hashes for active sessions in memory (Pilkington, 2012) in all current versions of Windows. As a result, should a passive snooping attack take place while a machine is locked or logged on, the LM hash will also be viewable in the capture. Windows LM hashes are also reversible back into the user’s original password using online services such as (OS – Objectif Securite, 2012) and other hash cracking tools. Finally, and possibly of most concern, is the accessibility of encryption keys. Even when full disk encryption is used to secure data, the keys (after initial user input) are cached in memory by the operating system. On traditional hardware this is considered the safest place, as volatile memory is erased quickly once the power is taken away. This is something that usually requires physical access to the machine to exploit, unless the machine is already infected. It is now possible to sniff the encryption key during a live migration to obtain the encryption hash.

Live migration attacks are not only vulnerable to sniffing attacks. In a paper written in 2007 by (Oberheide, et al., 2007), the researchers describes a number of attacks that are possible on live migration traffic. In the paper the researchers discus three ‘classes’ of threats that can be used against live migration environments:

  • Control plane
  • Data Plane
  • Migration Module

Control plane attacks

Control plane attacks target the mechanisms that are employed to initiate and manage the migrations within the infrastructure manager. While there were no active attacks in the paper, the theory behind the attacks is still relevant and should be considered in a risk analysis exercise. The examples in the paper that (Oberheide, et al., 2007) highlights, demonstrate how successful attacks to the control plane could result in:

  • The migration of machines onto illegitimate hosts that are owned by the attacker.
  • Mass migration of a large number of machines, thus overloading the network and causing disruption to service.
  • Manipulating the resources management for hosts in a DRS style environment, so that hosts are not evenly distributed and overwhelm single resources.

Data plane attacks

Data plane attacks are threats that take place on the networks on which the migrations are situated. The passive snooping attack that was demonstrated previously is one of the attacks that are briefly mentioned in the paper. The second attack on the data plane class is described as ‘active manipulation”. As the name suggests, active manipulation is when data is changed during the migration of the machine. Although in the paper (Oberheide, et al., 2007) introduce their custom tool Xensploit for preforming this attack, it is merely the collection of existing attacks collated into one tool for ease of use. The attack works by using MiTM characteristics to intercept traffic between the two hosts and manipulating sections of the traffic (RAM) during transit. In the example of the Xensploit software in the paper (Oberheide, et al., 2007, p. 4) showed how the attack could be used to establish an SSH (Secure Shell) session to a machine configured to only allow connections from authorized sources. Using Xensploit they were able to manipulate within the object code of the SSHD process to add their key as an authorised source, thus allowing them to SSH once the new instance had completed its migration.

Migration Model attacks

Lastly, although these attacks are included in the paper, I feels that this section falls under one of my other previous topics of the hypervisor rather than on live migration. The paper briefly covers attacks that exploit vulnerabilities in the migration models – that are part of the VMM (virtual machine monitor).

Combining attacks

Gaining access to any of these attack classes should be considered a high risk to security staff, although with access to just one of the vectors, it may still not be possible for an attacker to target a specific machine depending on its physical location. If a combination of the attacks were available, the attacker would be able to leverage them to better achieve control of the environment. An example of this is in the case of data plane class attacks. These attacks are only useful if the target machine is being migrated during the period of capture. In environments where features such as DRS and DPM are not in use, it is possible for machines to stay fixed to a host for a considerable period of time. If the attacker was able to utilize an attack at the control plane they would be able to trigger the migration of the necessary machines for the attack to then take place at the data plane.

Shortly after the paper was released, VMware’s (Wu, 2008) comments on this attack:

“Although impressive, this work by no means represents any new security risk in the datacentre… Rather, it a reminder of how an already-compromised network, if left unchecked, could be used to stage additional severe attacks in any environment, virtual or physical.”

While I can appreciate what (Wu, 2008) is saying, I must disagree that these types of attacks are comparable to physical environments. Data-In-Transit on physical systems do not tend to include such sensitive information. It is also better understood in traditional systems that unsafe protocols such as email and ftp should not be used to transfer confidential information. While the previous example required physical access to the vMotion network, it is also possible to access this data remotely through misconfiguration, access to the management interface or manipulation of virtual infrastructure.

Wu, W., 2008. VMware Security & Compliance Blog. [Online] Available at: http://blogs.vmware.com/security/2008/02/keeping-your-vm.html

Siebert, E., 2011. Five VMware security breaches that should never happen. [Online] Available at: http://searchvmware.techtarget.com/tip/Five-VMware-security-breaches-that-should-never-happen

Travostino, F., Daspit, P. & Gommans, L., 2006. Seamless live migration of virtual machines over the MAN/WAN. Future Generation Computer Systems – IGrid 2005: The global lambda integrated facility, 22(8), pp. 901-907.

Kutz, A., 2007. How to obtain, configure and use VMotion and how VMotion works. [Online]
Available at: http://searchvmware.techtarget.com/tip/How-to-obtain-configure-and-use-VMotion-and-how-VMotion-works
Pilkington, M., 2012. Protecting Privileged Domain Accounts: LM Hashes — The Good, the Bad, and the Ugly. [Online]
Available at: http://computer-forensics.sans.org/blog/2012/02/29/protecting-privileged-domain-accounts-lm-hashes-the-good-the-bad-and-the-ugly

Oberheide, J., Cooke, E. & Jahanian, F., 2007. Empirical Exploitation of Live Virtual Machine Migratio. [Online]
Available at: http://www.eecs.umich.edu/techreports/cse/2007/CSE-TR-539-07.pdf

Advertisements




Mitigation techniques for hypervisors

3 01 2014

There are both vendor specific and general vendor agnostic network security mitigation techniques that can be used to eliminate scanning and direct connection to the hypervisor. These include IDS/IPS (Intrusion Detection Systems / Intrusion Prevention Systems) on the network portion – to detect and block known scanning patterns performed by common scanning tools and specific vendor hardening options, such as VMware’s Lockdown Mode (VMware, 2012), which turns off the ability to connect directly to the host from anything other than the vCenter’s ‘vpxuser’ account.  However, there are many known evasion techniques available to bypass IDS/IPS systems and while permitted communication may be rejected by VMware’s ‘Lockdown mode’, vulnerabilities in the hypervisor code may still leave other communication channels open to abuse and this would be the case in all hypervisors.

In the author’s opinion, hypervisors should be situated on their own network/subnet, with strict ACL’s in place thus making them directly inaccessible to anyone outside of a limited scope of technical staff. There will be few systems in traditional networks that require this kind of isolation from users, as most require at least some level of user interaction for services etc. While this measure should be sufficient to eliminate attacks being performed directly on the hypervisors, utilising additional layers of defence such as IDS/IPS and specific hypervisor hardening mechanisms such as VMware’s ‘lockdown mode’ should also be used in parallel, to further ensure the integrity of the host.

Unfortunately these multiple network layers of protection do not address the issue of VM escape attacks as mentioned in my last post.  To achieve this, technicians should apply the same security mentality to the hypervisor layer that has previously been applied to the network layer. Depending on the size and nature of a virtualisation platform, layers should be applied to this new portion of infrastructure. One of the layers should be able to mitigate a worst case scenario attack, where an attacker targets less secured and more accessible VM’s on the infrastructure to attack higher valued VM’s running on the same hypervisor. To do this (again depending on the size and risk model of the environment), clusters could be configured to divide virtual machines into categories based on their security rating and grouped together accordingly. While this may be less beneficial when calculating the total ROI that virtualisation offers it is beneficial from a security perspective as the larger the resource pools are (in terms of the number of shared hardware elements) the more efficient the use of hardware is and therefore results in lower overheads. This cost should be offset against the likelihood and impact this kind of attack could cause in order to factor in the increased costs of grouping machines by security rating.

One method of eliminating the threat of machines running on the same hypervisor, even in an automated resource allocation environment such as the VMware’s DRS feature (Distributed Resource Scheduler), is to utilise the groups and rules available in the VMMM. A specific example of this is in the latest version of VMware’s vCenter (Version 5), there are options to collate groups/clusters of machines and set preferences on to which hypervisor they are situated on. Although this feature is intended for the purpose of grouping and separating machines from a resource and availability perspective, the authors also considers that this could also be used to ensure the security of machines against many hypervisor attacks, as well as other attacks, which will be discussed a later post.

In the example below, the author demonstrates a method that can be used for ensuring that machines with differing security classes are not located on the same physical host in automated distributed resource environments by using the resource rules in the VMware vCenter suite. This is done by assigning virtual machines into groups based on the security rating they are considered to have by the organisation. This method of using machine groups for security purposes in DRS environments is one that the author has not seen documented or discussed elsewhere.

These rules could be scaled depending on the size or risk index determined by the organization. There is also scope within these rules to balance out the resource/security overhead by specifying that the machine should not run on a certain host rather than must not. It should also be noted that VMware’s vCloud Datacenter offers a (Lodge, 2010) “Dedicated VDC” option, which provides physically separate hardware – ideal for meeting security or regulatory requirements, where physically sharing isn’t an option”.

Using vCenter groups for segmentation

In this example, the author has set up a simple scenario demonstrating how the rules in VMware’s vCenter suite could be used to separate a group of machines, considered insecure, from running on the same hypervisor/host as another group that are considered secure. This method of using machine groups for security purposes in DRS environments is one that the author has not seen documented or discussed anywhere else prior to writing this.

Using DRS groups, two groups are created - ‘Secure-servers’ and ‘Insecure’. Machines are associated with the appropriate group based on their service etc

Using DRS groups, two groups are created – ‘Secure-servers’ and ‘Insecure’. Machines are associated with the appropriate group based on their service etc

A rule is created specifying that all servers in the 'Secure-Server' group must run on ESX1

A rule is created specifying that all servers in the ‘Secure-Server’ group must run on ESX1

Another rule is created specifying that all machines in the 'Insecure' group must not run on ESX1

Another rule is created specifying that all machines in the ‘Insecure’ group must not run on ESX1

These rules could be scaled depending on the size or risk index determined by the organization. There is also scope within these rules to balance out the resource/security overhead by specifying that the machine should not run on a certain host rather than must not. It should also be noted that VMware’s vCloud Datacenter offers a (Lodge, 2010) “Dedicated VDC” option, which provides physically separate hardware – ideal for meeting security or regulatory requirements, where physically sharing isn’t an option

Lodge, M., 2010. Getting rid of noisy neighbors: Enterprise class cloud performance and predictability. [Online]
Available at: http://blogs.vmware.com/rethinkit/2010/09/getting-rid-of-noisy-cloud-neighbors.html





Attacking the hypervisor

27 11 2013

The hypervisor has the disadvantage of being potentially attacked in one of two ways, from either the network layer or from the host running on that hypervisor. The default behaviour of a hypervisor on a network is to respond to connections through standard TCP/IP, much the same as other desktop machines, devices and infrastructure. This results in the hypervisor being locatable on the network and consequently susceptible to traditional network enumeration attacks such as Nmap (nmap.org, 2012) and Nessus (Tenable, 2012). While enumeration tools are primarily used as a discovery mechanism, they are often able to extract further information about a system by analysing characteristics and information returned by the host. An example of this technique using the currently most utilised enumeration software (Nmap) would be by specifying the ‘–O’ switch, which compares the host’s packet response against a large database of software. Once this extra information about the host has been identified, additional approaches can be used to cross-examine the hosts further to identify attributes such as patch levels and service packs. Depending on the software that is found, using these approaches the attacker is then able to determine the appropriate CVE (Common Vulnerabilities and Exposures) that the host may be vulnerable to. After the vulnerabilities have been identified, the attacker is able to exploit the system using the exploit and insert a payload to further control the host and maintain access. Current examples of software that can be used to exploit systems and insert malicious payloads are Metasploit (Metasploit, 2012) and CORE Impact (Core Security, 2012). This method of enumeration and exploitation will already be familiar to security staff responsible for scanning traditional clients as it is identical.

It is the second method used to attack the hypervisor from the guest or virtual machine that is much more dangerous and an unfamiliar concept, especially for companies invested in the cloud computing or hosting servers in large datacentres.

The term virtual machine (VM) escape is the concept of breaking out of an isolated VM in order to execute malicious code on the host. There have been a number of vulnerabilities on both Type 1 and 2 hypervisors that demonstrate this concept of escape (CVE-2009-1244, CVE-2011-1751, CVE-2012-0217 (Xen, 2012), CVE-2012-3288). While the danger of ‘Type 2’ hypervisor escape is still a threat, the implications of breaking out of a guest, running on an enterprise ‘Type 1’ hypervisor such as ESXi or the Xen hypervisor would be much greater due to the environments that they are often employed in. In traditional networks, security can often be achieved through the segmentation of networks into either physical or virtual networks. This segmentation is still applicable within virtual networks; however this only offers security at the network layer, rather than this new layer of ‘guest–host’ exploitation. While this might sound like an unlikely threat, due to HA features found in VMMM such as VMware’s DRS (Distributed Resource Scheduler) the movement of machines across hypervisors is often determined by the management server rather than by a human. That is unless specified rules are created by the administration. This dynamic movement of virtual machines has the potential to result in an unpatched, publically addressable server being hosted on the same hypervisor/hardware as domain controllers and other high value target machines. This threat is certainly a cause for concern when considering mid to large size networks hosting tens to hundreds of machines within the same infrastructure separated by VLAN’s. The implications and likelihood of this attack is greatly increased when considering multi-tenant public cloud infrastructures. The topic of how hackers could potentially start to rent hosted machines on public clouds to attack other machines will be covered at a later date, but in the authors opinion this could become an actual threat that needs to be considered during a company’s risk analysis process.

There are a number of methods of assessing the security of virtual environments; one of the tools that was recently developed to assist in the evaluation of virtual environments is the VASTO project (Virtualization Assessment Toolkit) (Criscione, et al., 2012). The VASTO project is essentially a collection of Metasploit modules written to query and attack virtual environments, although mainly the VMware platform. The modules are added to the Metasploit project to leveraging an already established and robust framework.

As highlighted earlier, hypervisors are often located on the same subnet as the rest of the servers and, in some cases, the clients. This means that if an attacker is able to gain access to a network that is able to communicate with the hypervisor due to placement or incorrectly configured ACL’s (Access Control Lists), the hypervisor could be attacked directly. Shown in the following example are three simple methods that can be used to locate and query a hypervisor in order to retrieve important information such as version, build number and vulnerabilities that it is susceptible to.

For this demonstration, the author is using a laptop wired into the 192.168.20.0 network in a test environment. Shown in Figure 1, the author uses an NMAP command with the ‘-sV’ switch to scan the entire subnet to return a list of live hosts and associated services. The scan correctly identifies both of the ESXi servers located on the network.

Figure 1- Section of results from an NMAP scan “nmap –sV –T 192.168.20.0/24”

Figure 1- Section of results from an NMAP scan “nmap –sV –T 192.168.20.0/24”

As shown in figure 2 NMAP returns results showing that ESXi is installed on two IP addresses on the 192.168.20.0 subnet and has several open ports. While NMAP does identify the product and version correctly on this occasion, it is not always completely reliable in returning the exact version of the host running on the host. To do this there are a number of methods including VASTO, Nessus or OpenVAS. Using the “vmware_version” module found in VASTO (shown in Figure 2) we are able to detect the exact version of the host including build number.

Figure 2 - Section of results from VASTO vmware_version scan

Figure 2 – Section of results from VASTO vmware_version scan

This now gives the attacker the information needed to locate existing exploits against this version or even develop new exploits, depending on the value of a target. Shown in Figure 4 is a screen shot of a Nessus report generated after a scan against the IP address of the ESXi host. Nessus is an automated scanning and vulnerability assessment tool that fingerprints the host against numerous plugins in order to detect exploits that the host is vulnerable to. While the full report highlighted a number of vulnerabilities found on the host, Figure 3 shows that this particular host is vulnerable to one plugin tested – containing 3 CVE’s (CVE-2012-2448, CVE-2012-2449, CVE-2012-2450).

The number of exploits and risks associated with them is not the area being addressed in this demonstration, but rather the ability to identify the hypervisors and attack it directly. The quantity and complexity of the attacks involved in exploiting type 1 hypervisors is currently much greater than those found on type 2 implementations. As with any technology, as popularity grows and new features are added then the greater the likelihood is that easy to acquire automated attacks will exist.

Figure 3 - Section of Nessus report highlighting highly rates vulnerabilities

Figure 3 – Section of Nessus report highlighting highly rates vulnerabilities

VMware greatly increased the security of their hypervisor through the replacement of their ESX product in favour of adopting the new lightweight (smaller code footprint) ESXi, which did not include their service console within the architecture of the code (VMware, 2012). However there are still elements within the hypervisor that continue to threaten its security. One of these is the notion that ESXi is by default configured to be accessible through a browser. Clients with Port 80 and 443 access to the hosts are able to directly access the hypervisor through a browser and even use the host to download the vSphere management client. While this may be convenient, it is the author’s opinion that this ‘out of the box’ configuration lacks the fundamental security posture that should be taken against such a high value target. An attacker with the vSphere client is able to directly manage the hypervisor once a username and password have been provided. It should also be noted that all ESXi servers (by default) are configured using the ‘root’ account, meaning that the only unknown credential required to manage the host is the root password and it would be possible to ‘brute-force’ this. Furthermore there is a customised brute forcing tool in the VASTO suite called “vmware_login”, which allows automatic dictionary or ‘brute-force’ login attempts.  In addition to all of these vectors, there are also the pertinent issues of existing network security issues such as MITM’s (Man-in-the-middle attack), which could expose these credentials.

To demonstrate the prevalence of exposed hypervisors, using the online search tool ‘Shodan’ (SHODAN, 2012) the author is able to search the internet for exposed ‘ESX’ hosts. In figure 4 we are able to see that Shodan has returned 749 results fitting that description.

Figure 4 - Results of a Shodan search for host containing the term "esx"

Figure 4 – Results of a Shodan search for host containing the term “esx”

While not all of the hosts returned by Shodan are active, a large number of them are still current and allow remote connections to be made over the internet. Shown in figure 5 is a valid connection to one of the returned addresses through a web browser showing an ESXi 5 host.

Figure 5 - Connection to the IP address of one of the hosts found by Shodan

Figure 5 – Connection to the IP address of one of the hosts found by Shodan





Introduction to the hypervisor

22 11 2013

The hypervisor is arguably one of the more misunderstood concepts of virtualization for technical professionals who are more familiar with traditional methods of computing, as it can often be viewed as simply another operating system. While the hypervisor may be a form of operating system, the implications surrounding the impact of a successful exploitation against the system cannot be likened to that of a traditional network operating system. The most obvious element that distinguishes a hypervisor from a traditional operating system is the far reaching implications that vulnerabilities in the hypervisor could have upon the entire system. There are numerous implementations of vendor hypervisors which all have differing levels of vulnerabilities associated. In this post a definition of typical hypervisor implementations is given to establish a baseline of understanding before continuing into the attacks.

In its simplest form, a hypervisor is a piece of code that controls the flow of instructions between guest operating systems and the physical hardware. The hypervisor emulates the physical characteristics of the actual machine such as the processor, RAM, network cards, etc. and presents a homogeneous environment to all the guests.  There are two types of hypervisors – ‘Native’ (also known as ‘Bare Metal’ or Type 1) and ‘Hosted’ (also known as Type 2).

Native hypervisors are installed directly onto the hardware, as would be done with any traditional operating system. There are also implementations of hypervisor that come preinstalled on the host ROM. Native hypervisors benefit from having direct access to the underlying physical hardware of the host, resulting in improved performance. As these systems are not full operating systems they also have the benefit of having a smaller attack surface and are therefore considered more secure.

Hosted hypervisors are installed onto the existing operating system, eg Windows or Linux, which is responsible for communication between the hardware and the hypervisor. This type of hypervisor is less efficient in terms of performance and security and is typically used on desktops rather than servers.

While most common ‘Type 1’ hypervisors are a fraction of the size of a typical desktop operating system – such as Windows 7, the code is still an additional layer of software that is added to the total attack surface of the machine. This underlying dependence, which all hosted machines have on the hypervisor, is one of the most contested factors around virtualisation security ie the ability to compromise the hypervisor and use it to ‘escape’  to other machines hosted on that software.

The security of a Hypervisor is comparable to that of a standard operating system, when considering the size and surface attack area. Systems such as OpenBSD and TrustedBSD allow a greater level of customisation and ability to greatly reduce the features available for a particular task. This lower default functionality offered by systems often has a direct correlation to its security. Hypervisors have typically based their security and efficiency on the amount of Source lines of code (SLOC) used. The core functionally of a hypervisor is to translate and schedule the flow of instructions from the guest to the hardware, anything additional to this could be described as a non-essential feature.

One example of a security focused hypervisor is IBM’s ‘sHype’ implementation of the Xen hypervisor. The total code of this project is claimed to be around 2600 lines in length. It is reported that ESXi and KVM hypervisors were around 200,000 SLOC in 2010 (Steinberg & Kauer, 2010, p. 3), with indications that this could rise and therefore further increasing the attack surface.