MySQL SSH Tunneling with AWS Session Manager

For the past several months, the DevOps team in our organization has worked on finding ways to increase the security of our AWS cloud infrastructure projects. This resulted in creating an extensive list of requirements that should be implemented for all existing and future projects.

As of right now, almost all of the projects make use of an EC2 instance which acts as a bastion host (jump box) and provides us a way of accessing resources in our private subnets. Even though we make sure to harden the bastion host so it won’t represent a security issue, the issue with this approach is that the bastion host resides in a public subnet and ingress rules do allow connections from the outside world.

AWS Session Manager To The Rescue

Session Manager is a capability of AWS Systems Manager which allows us to manage the EC2 instances through an interactive one-click-browser-based shell or through the AWS CLI. AWS Session Manager provides us with secure instance management without the need to open inbound ports or maintain bastion hosts.

However, we won’t go into the details of setting up Session Manager for your EC2 instances since the official documentation is detailed enough and you can also check it out here.

Furthermore, the Session Manager capability seems to be an improvement to our cloud security, but now we are facing a new challenge. We still need a way to access our RDS instances residing in a private subnet.

How we did this in the past by creating a ssh tunnel via our public bastion host and accessing the private MySQL RDS instances. So naturally, the first thing we searched on google was ‘AWS Session Manager tunneling’. Numerous tutorials popped out, but none of them thoroughly explained the complete process of creating the ssh tunnel.

Creating the SSH tunnel

Even though we said that Session Manager eliminates the need for maintaining bastion hosts, in order to access resources in our private subnet, we still need to create an EC2 instance that will serve as a bastion host. The benefit of using Session Manager is that the bastion host will now reside in a private subnet and its security groups won’t allow any inbound traffic. 

The next thing is to modify our local ssh config file which is typically located in ~/.ssh/config (Linux and MacOS) or C:\Users\username\.ssh\config (Windows) with the following content:

# SSH over Session Manager
host i-* mi-*
ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

This allows running a proxy command that starts a Session Manager session and transfers all data through the connection.

We are now ready to start a Session Manager connection with SSH. You can try logging into your Bastion host with the following command:

ssh -i KEY-FILE.pem username@INSTANCE-ID

where KEY-FILE is the name of the key you created or chose for your bastion host and username is ec2-user if you’re using Amazon Linux AMI, ubuntu if you’re using Ubuntu AMI, etc… and INSTANCE-ID looks something like i-0ac772dd11dc4b4e8.

If you successfully log in to the bastion host, you’re now ready for the next step which is creating a tunnel and accessing the RDS instance. This is done by running the following command:

ssh -i KEY-FILE.pem ec2-user@INSTANCE_ID -L 9090:RDS-ENDPOINT:3306

The command creates access to the database on the RDS instance — the local port 9090 tunnels to port 3306 on the RDS instance.

Please note that INSTANCE-DNS should look something like ip-10–0–1–204.region.compute.internal and RDS-ENDPOINT should look something like database1.chefcaow2ket.region.rds.amazonaws.com.

With the session still active, you’re now ready to open a new terminal window and try and connect to the database with the following command:

mysql -u USERNAME -h 127.0.0.1 -P 9090 -p password

or you can simply use MySQL Workbench to connect to the database.

Important to know:

The RDS instance security group must allow access from your bastion host.

When trying to create an SSH connection, if the instance cannot be found in the default region, the command fails with the error ‘kex_exchange_identification: Connection closed by remote host’. To prevent this, remember to update the default region in the ~/.aws/config file.