Attacking management interfaces

29 09 2014

The management interfaces and everything that is incorporated into that software is, in the author’s opinion the most problematic area in virtualisation security today. There have been numerous attempts over the last few years to demonstrate how management interfaces can be breeched. The majority of these attacks are general attacks that use pre-existing attack methods such as brute forcing, MiTM (man-in-the-middle) and the numerous flaws with the PKI infrastructure. There are multiple proven attack methods available for exploiting management interfaces and below are descriptions of some of these attacks that have been discovered by researchers.

In an online blog (Mluft, 2011)  talks about how a brute force attack is achievable on the Amazon Web Services (AWS) portal by leveraging existing hacking software tools. In the attack (Mluft, 2011) demonstrated how it is possible to determine a successful logon using the exemplary payloads in the Burp Suite (Burp Suite, 2012). The use of the burp suite in this example is to simply automate the process of attempting logins to the interface. The payload in the software is also able to identify failed login attempts to the portal by returning a HTTP status code of 200 (Network Working Group , 1999). A correct password attempt is identified by a returned HTTP status code of 302. Using the documentation provided by Amazons services in relation to password policies (Mluft, 2011) created an appropriate wordlist and used the burp suite to attempt all the possible permutations. After 400,000 attempts the attack was paused and the results purged for the 302 status code. The code was found and also shown alongside was the value and the password attempted. This gave the attacker the username and password for the administration of all servers managed by that account. It should be noted that all of these attempts were originated from one IP address without the account being locked-out or subject to any account throttling.

As discussed earlier my earlier blog artical regarding the hypervisor, the Virtualization Assessment Toolkit (VASTO) has been developed to exploit multiple weaknesses, predominantly in the VMware family. As well as the identification module that returns the exact version of the server, it includes numerous attacks on virtual systems including a specific VMware brute forcing module, which mimics the attack on the AWS portal by (Mluft, 2011). One of the main contributors to the VASTO project (Criscione, 2010) demonstrated a number of the different functions found in VASTO at Blackhat USA 2010.  Although (Criscione, 2010) demonstrated how VASTO can be used at multiple layers of the virtual stack (Client, Hypervisor, Support, Management and internal), the majority concentrated on the management portion. (Criscione, 2010) confirms that although the (VMware, 2012) hardening guide recommends segmentation of management networks, these recommendations are often ignored and left situated on the same networks as traditional servers.

These servers that manage the entire fabric of the infrastructure have multiple attack vectors – from the operating systems they are installed on to the web services running the interfaces. Vulnerabilities in any one of these platforms can potentially jeopardise the security of an entire environment and should be taken very seriously.

The other element used in the VASTO modules which can target the management portion of the virtual infrastructure uses target flaws in the VMware components and implementation to expose threats in the infrastructure. One of the exploits that is included in the VASTO suite that best demonstrates how multiple components in these systems can be used for exploitation, originates via a flaw in the Jetty (Eclipse, 2012) web server that is used by vCenter Update manager. In the author’s opinion, this attack signifies how the complexity and code overhead that these management servers introduce, make securing virtual environments in an efficient manner, one that needs to be understood and prioritised. I will briefly give a breakdown of this attack to highlight the multiple elements that were used to complete the attack.

The Update Manager component of the vSphere suite is designed to secure the environment by automating the patching and updating process of hosts that fall under its management scope. However, (Criscione, 2010) recognised that the update manager requires a version of Jetty web server to operate. This is an additional component that is added to the total footprint of the management server. The version of Jetty installed prior to version 4.1 u1 (update 1) of the update manager was a version vulnerable to a directory traversal attack (Wilkins, 2009), which allowed attackers to view any files on a server that the Windows SYSTEM user has privileges. Consequentially vCenter stored a file on the server called “vpxd-profiler-*” which is a file used by administrators for debugging purposes. In this extensive file the, SOAP Session ID’s of all the users that have connected to that server are contained. With this ID the vmware_session_rider module, found in the VASTO toolkit, acts as a proxy server to allow the attacker to then connect through it into the vCenter server using the selected administrator SOAP ID. Once this is completed, the attacker is able to create a new admin credential within vCenter to ensure future access.

Another example of how different elements of the management interface could be used to gain access to vCenter is through VMware’s use of Apache Tomcat technology (The Apache Software Foundation, 2012). When navigating to a vCenter server through a web browser one is presented with the standard vSphere “Getting started” screen as is shown in figure 1

Web browser connection to vCenter server

Web browser connection to vCenter server

Connection to that same servers IP address, but specifying the default tomcat Tomcats index page port of “8443” over an SSL connection shows further information, including a link to login as the “Tomcat manager”. This page is shown in figure 2

The web interface seen when you navigate to vCenter with a port of 8443

The web interface seen when you navigate to vCenter with a port of 8443

In VMware version 4.1 there is a user named “VMwareAdmin” that is automatically added to the Tomcat server, which has full admin rights to the Tomcat service. In the earlier versions of VMware, the password for this admin account was 5 characters long starting with 3 uppercase, 1 number and one lowercase. This leaves an attacker with a number of options for an attacking perspective. The most obvious is to brute force the credentials with a compatible tools or script such as the Apache tomcat brute force tool (Snipt, 2011). A second (and more sophisticated attack) would be to use the folder traversal vulnerability introduced by the Jetty service to gain read access to the server. From here the attacker could navigate to the “tomcat-users.xml” file (C:\Program Files\VMware\Infrastructure\tomcat\conf) as shown in Figure 3, which is an XML file found in VMware 4.1 and which shows the clear text credentials of the account.

(left) The tomcat-users.xml file showing the username and password of a default admin account (Right) tomcat manager login prompt

(left) The tomcat-users.xml file showing the username and password of a default admin account (Right) tomcat manager login prompt

Using this access, an attacker is able to control elements of the web service with admin rights. As shown in Figure 4, one is able to change a number of settings through the tomcat interface, including the ability to upload custom WAR files, which can be created using Metaspolit to upload meterpreter payloads to the server.

Logged in to the tomcat manager using the credentials found on server

Logged in to the tomcat manager using the credentials found on server

Although some of the attacks using the VASTO toolkit are specific and use vulnerabilities that have almost all been patched by VMware (at the time of writing), the management interfaces are still vulnerable to more general network attacks that are not as fundamental to secure as simply applying a patch or updating to the newest version. As is explained briefly in by post on hypervisors, access to these interfaces are vulnerable to MiTM attacks and the implementations dependence on a highly insecure certificate/PKI model. These vulnerabilities are not directly the responsibility of the vendors, but certainly nothing has been done by them to address this issue.

I will not be explaining the process of how MiTM attacks and flaws in the certificate infrastructure can be used to capture login credentials, as this a fundamental part of security and has been covered on numerous occasions by multiple sources (Irongeek, 2012) (Schneier, 2011). I have also written about the overarching problems with the certificate model and how it can be bypassed by in a blog post from 2011.

 

Mluft, 2011. The Key to your Datacenter. [Online] Available at: http://www.insinuator.net/2011/07/the-key-to-your-datacenter/

Criscione, C., 2010. Blackhat 2010 – Virtually Pwned. USA: Youtube.

Wilkins, G., 2009. Vulnerability in ResourceHandler and DefaultServlet with aliases. [Online] Available at: http://jira.codehaus.org/browse/JETTY-1004

Irongeek, 2012. Using Cain to do a “Man in the Middle” attack by ARP poisoning. [Online] Available at: http://www.irongeek.com/i.php?page=videos/using-cain-to-do-a-man-in-the-middle-attack-by-arp-poisoning

Schneier, B., 2011. Schneier on Security. [Online] Available at: http://www.schneier.com/blog/archives/2011/09/man-in-the-midd_4.html

Advertisements




Mitigation techniques for hypervisors

3 01 2014

There are both vendor specific and general vendor agnostic network security mitigation techniques that can be used to eliminate scanning and direct connection to the hypervisor. These include IDS/IPS (Intrusion Detection Systems / Intrusion Prevention Systems) on the network portion – to detect and block known scanning patterns performed by common scanning tools and specific vendor hardening options, such as VMware’s Lockdown Mode (VMware, 2012), which turns off the ability to connect directly to the host from anything other than the vCenter’s ‘vpxuser’ account.  However, there are many known evasion techniques available to bypass IDS/IPS systems and while permitted communication may be rejected by VMware’s ‘Lockdown mode’, vulnerabilities in the hypervisor code may still leave other communication channels open to abuse and this would be the case in all hypervisors.

In the author’s opinion, hypervisors should be situated on their own network/subnet, with strict ACL’s in place thus making them directly inaccessible to anyone outside of a limited scope of technical staff. There will be few systems in traditional networks that require this kind of isolation from users, as most require at least some level of user interaction for services etc. While this measure should be sufficient to eliminate attacks being performed directly on the hypervisors, utilising additional layers of defence such as IDS/IPS and specific hypervisor hardening mechanisms such as VMware’s ‘lockdown mode’ should also be used in parallel, to further ensure the integrity of the host.

Unfortunately these multiple network layers of protection do not address the issue of VM escape attacks as mentioned in my last post.  To achieve this, technicians should apply the same security mentality to the hypervisor layer that has previously been applied to the network layer. Depending on the size and nature of a virtualisation platform, layers should be applied to this new portion of infrastructure. One of the layers should be able to mitigate a worst case scenario attack, where an attacker targets less secured and more accessible VM’s on the infrastructure to attack higher valued VM’s running on the same hypervisor. To do this (again depending on the size and risk model of the environment), clusters could be configured to divide virtual machines into categories based on their security rating and grouped together accordingly. While this may be less beneficial when calculating the total ROI that virtualisation offers it is beneficial from a security perspective as the larger the resource pools are (in terms of the number of shared hardware elements) the more efficient the use of hardware is and therefore results in lower overheads. This cost should be offset against the likelihood and impact this kind of attack could cause in order to factor in the increased costs of grouping machines by security rating.

One method of eliminating the threat of machines running on the same hypervisor, even in an automated resource allocation environment such as the VMware’s DRS feature (Distributed Resource Scheduler), is to utilise the groups and rules available in the VMMM. A specific example of this is in the latest version of VMware’s vCenter (Version 5), there are options to collate groups/clusters of machines and set preferences on to which hypervisor they are situated on. Although this feature is intended for the purpose of grouping and separating machines from a resource and availability perspective, the authors also considers that this could also be used to ensure the security of machines against many hypervisor attacks, as well as other attacks, which will be discussed a later post.

In the example below, the author demonstrates a method that can be used for ensuring that machines with differing security classes are not located on the same physical host in automated distributed resource environments by using the resource rules in the VMware vCenter suite. This is done by assigning virtual machines into groups based on the security rating they are considered to have by the organisation. This method of using machine groups for security purposes in DRS environments is one that the author has not seen documented or discussed elsewhere.

These rules could be scaled depending on the size or risk index determined by the organization. There is also scope within these rules to balance out the resource/security overhead by specifying that the machine should not run on a certain host rather than must not. It should also be noted that VMware’s vCloud Datacenter offers a (Lodge, 2010) “Dedicated VDC” option, which provides physically separate hardware – ideal for meeting security or regulatory requirements, where physically sharing isn’t an option”.

Using vCenter groups for segmentation

In this example, the author has set up a simple scenario demonstrating how the rules in VMware’s vCenter suite could be used to separate a group of machines, considered insecure, from running on the same hypervisor/host as another group that are considered secure. This method of using machine groups for security purposes in DRS environments is one that the author has not seen documented or discussed anywhere else prior to writing this.

Using DRS groups, two groups are created - ‘Secure-servers’ and ‘Insecure’. Machines are associated with the appropriate group based on their service etc

Using DRS groups, two groups are created – ‘Secure-servers’ and ‘Insecure’. Machines are associated with the appropriate group based on their service etc

A rule is created specifying that all servers in the 'Secure-Server' group must run on ESX1

A rule is created specifying that all servers in the ‘Secure-Server’ group must run on ESX1

Another rule is created specifying that all machines in the 'Insecure' group must not run on ESX1

Another rule is created specifying that all machines in the ‘Insecure’ group must not run on ESX1

These rules could be scaled depending on the size or risk index determined by the organization. There is also scope within these rules to balance out the resource/security overhead by specifying that the machine should not run on a certain host rather than must not. It should also be noted that VMware’s vCloud Datacenter offers a (Lodge, 2010) “Dedicated VDC” option, which provides physically separate hardware – ideal for meeting security or regulatory requirements, where physically sharing isn’t an option

Lodge, M., 2010. Getting rid of noisy neighbors: Enterprise class cloud performance and predictability. [Online]
Available at: http://blogs.vmware.com/rethinkit/2010/09/getting-rid-of-noisy-cloud-neighbors.html