Securing XL Deploy Overthere SSH Hosts with SUDO

At XebiaLabs we talk a lot about the value of our agent-less technology.  Some reasons why agents are not a good idea are as follows:

• tedious process of installing agents on all of your servers
• overhead associated with maintaining them
• security risks that you’re relying on your vendor to identify and fix in a timely manner

  We use the same trusted technology that you would use to access your systems.  On Unix that would be ssh, sudo and su.  These are the industry standards for allowing secure access to your remote Unix systems.  With all of the resources in the development communities for these tools any potential security vulnerabilities are quickly identified and patched.  The patches are quickly made available by the OS providers and your local admins apply these patches to your systems. For security reasons, some companies and system administrators like to control access to the service accounts that applications run as.  The trade off is that some users need access to run commands as those users to administer the systems.  The administrator of a system can configure ssh to prevent people from remotely logging on to the system as the service account.  Then we need to figure out how legitimate users can gain access to the service account to do their work.  This is where tools like sudo and su come in.  XL Deploy can also use these techniques to gain access to the service accounts to execute deployments on these systems. I am going to demonstrate a few of the possible configurations that can be used to tightly control access to these Unix systems and allow XL Deploy to execute deployments.  One of the advantages of using a tool like XL Deploy is that you can control who has access to execute deploys on the systems in your organization as well as maintaining a log of all of the actions that were done.  In addition to the audit logging in XL Deploy, using sudo provides additional logging of all commands.  

XL Deploy SUDO Interactive Host Configuration:

For the examples here we are going to user testusr as our XL Deploy service account to log into the remote systems.  We will also use xebia as the account that has the permissions to do the deployments on the Unix system.  First, let's look at an interactive sudo configuration.  Let's start by configuring a host in XL Deploy, then we will look at some of the possible configurations with ssh, sudo and su.  For our first set of configurations, we will use sudo in an interactive configuration.  For many users, when they try to execute a command on a Unix system, they are required to retype their password, just to make sure someone has not come up to an open terminal.  This way we know that the user is actually who they say they are before we allow them to sudo execute commands.  XL Deploy can emulate this type of behavior.  We can configure the Connection Type for the host to be INTERACTIVE_SUDO.  This means that XL Deploy will ssh to the host and execute all of the commands via sudo supplying the password as requested.  In the repository under infrastructure create an Overthere SshHost as follows:

XL Deploy SUDO Interactive:

For our first use case we will configure XL Deploy and our Unix server to allow XL Deploy to ssh into the box and use sudo to execute the commands as our application service account.  XL Deploy has a property file to configure all of the default properties of the system.  For the interactive sudo the only property we need to change is the overthere.SshHost.allocateDefaultPty.  This parameter makes sure there is a PTY defined for the session. Change the <xl_deploy>/conf/deployit-defaults.properties:

overthere.SshHost.allocateDefaultPty=true

 Next we need to change sudoers file to add the following:

testusr ALL=(xebia)  ALL

In XL Deploy we can run a connection test to verify our configuration.  By using the journalctl or some other facility based you your servers configuration you can look in the system log or sudoers log and see what commands were run using sudo.  An example log is as follows:

sudo[3359]: testusr : TTY=pts/4 ; PWD=/home/testusr ; USER=xebia ; COMMAND=/bin/ls /tmpsudo[3359]: pam_unix(sudo:session): session opened for user xebia by testusr(uid=0)sudo[3359]: pam_unix(sudo:session): session closed for user xebiasshd[3358]: Received disconnect from XXX.XXX.XXX.XXX: 11:sshd[3352]: pam_unix(sshd:session): session closed for user testusrsystemd-logind[485]: Removed session 307. 

SSH + SUDO + SU Interactive:

The next configuration we can look at is interactively using sudo to su to our service account.  The advantage of using su in the sudo command is that su will load the software service accounts profile setting useful environment variables.  To make this configuration work we need to change some default properties. Change the <xl_deploy>/conf/deployit-defaults.properties:

overthere.SshHost.sudoCommandPrefix=sudo su - {0} –c overthere.SshHost.sudoQuoteCommand=true overthere.SshHost.allocateDefaultPty=trueNext we need to change sudoers file to add the following:

Cmnd_Alias XLD = !/bin/su, /bin/su - xebia -c *, !/bin/bash testusr ALL=(ALL) NOPASSWD: XLDIn XL Deploy we can run a connection test to verify our configuration.  By using the journalctl or some other facility based you your servers configuration you can look in the system log or sudoers log and see what commands were run using sudo.  An example log is as follows:

sudo[656]: testusr : TTY=pts/4 ; PWD=/home/testusr ; USER=root ; COMMAND=/bin/su - xebia -c ls /tmpsudo[656]: pam_unix(sudo:session): session opened for user root by testusr(uid=0)su[679]: (to xebia) testusr on pts/4su[679]: pam_unix(su-l:session): session opened for user xebia by testusr(uid=0)su[679]: pam_unix(su-l:session): session closed for user xebiasudo[656]: pam_unix(sudo:session): session closed for user rootsshd[655]: Received disconnect from XXX.XXX.XXX.XXX: 11:sshd[648]: pam_unix(sshd:session): session closed for user testusrsystemd-logind[485]: Removed session 287.su[29563]: pam_unix(su-l:session): session closed for user testusr 

XL Deploy SUDO Host Configuration:

For our next set of use cases we are going to look at a configurations that are better suited for automation.  Since a person will not be logging into the Unix box we don't really need to worry about a person walking up to an open terminal.  Since we don't need to worry about someone walking up to a an open terminal we can configure sudo to not require a password.  For this use case we will configure the connection type to be SUDO.  Create a Overthere SshHost as follows:SSH + SUDO :NOPASSWD:Just like our first use case above we will need to set the PTY properties in XL Deploy's defaults file.  We will make a slight change in the sudoers files as well.  With these changes we will see the same behavior except XL Deploy does not need to provide the password again for the sudo command. Change the <xl_deploy>/conf/deployit-defaults.properties:

overthere.SshHost.allocateDefaultPty=true

Change the sudoers file to add the following:

testusr ALL=(xebia) NOPASSWD: ALL

In XL Deploy we can run a connection test to verify our configuration.  By using the journalctl or some other facility based you your servers configuration you can look in the system log or sudoers log and see what commands were run using sudo.  An example log is as follows:

sshd[2950]: pam_unix(sshd:session): session opened for user testusr by (uid=0)sudo[2956]: testusr : TTY=pts/4 ; PWD=/home/testusr ; USER=xebia ; COMMAND=/bin/ls /tmpsudo[2956]: pam_unix(sudo:session): session opened for user xebia by testusr(uid=0)sudo[2956]: pam_unix(sudo:session): session closed for user xebiasshd[2955]: Received disconnect from XXX.XXX.XXX.XXX: 11:sshd[2950]: pam_unix(sshd:session): session closed for user testusr

SSH + SUDO + SU :NOPASSWD:

The last configuration we can look at is using sudo to su to our service account.  To make this configuration work we need to change some default properties. Change the <xl_deploy>/conf/deployit-defaults.properties:

overthere.SshHost.sudoCommandPrefix=sudo su - {0} –c overthere.SshHost.sudoQuoteCommand=true overthere.SshHost.allocateDefaultPty=true  Change the sudoers file to add the following:

Cmnd_Alias XLD = !/bin/su, /bin/su - xebia -c *, !/bin/bash testusr ALL=(ALL) NOPASSWD: XLDIn XL Deploy we can run a connection test to verify our configuration.  By using the journalctl or some other facility based you your servers configuration you can look in the system log or sudoers log and see what commands were run using sudo.  An example log is as follows:

sudo[656]: testusr : TTY=pts/4 ; PWD=/home/testusr ; USER=root ; COMMAND=/bin/su - xebia -c ls /tmpsudo[656]: pam_unix(sudo:session): session opened for user root by testusr(uid=0)su[679]: (to xebia) testusr on pts/4su[679]: pam_unix(su-l:session): session opened for user xebia by testusr(uid=0)su[679]: pam_unix(su-l:session): session closed for user xebiasudo[656]: pam_unix(sudo:session): session closed for user rootsshd[655]: Received disconnect from XXX.XXX.XXX.XXX: 11:stella sshd[648]: pam_unix(sshd:session): session closed for user testusrsystemd-logind[485]: Removed session 287.su[29563]: pam_unix(su-l:session): session closed for user testusr There are many more configurations that are possible with XL Deploy.  Because XL Deploy can use existing tools like ssh, sudo and su we can leverage existing security protocols that are current standard practice in the industry.  The existing security practices around ssh, sudo and su have proven to be highly effective, combined with XL Deploy's auditing capabilities you can automate your deployments in a highly secure and auditable way.  Resources:• https://support.xebialabs.com/hc/communities/public/questions/202353819-Configuring-your-sudoers-file-for-automated-remote-automation• https://docs.xebialabs.com/xl-deploy/how-to/connect-xl-deploy-to-your-infrastructure.html• https://docs.xebialabs.com/xl-deploy/how-to/troubleshoot-an-ssh-connection.html• https://support.xebialabs.com/hc/communities/public/questions/201999115--Permission-denied-error-when-starting-wsadmin-sh-over-ssh-with-sudo• https://docs.xebialabs.com/xl-deploy/how-to/choose-an-overthere-host-type-and-connection-type.html• http://www.sudo.ws/sudo.html• http://serverfault.com/questions/197340/sudoers-to-execute-commands-as-another-user