How to interactively debug GitHub Actions with netcat

by on under blog
5 minute read

Update: This was a fun experiment and I recommend you check out the post for a fun read on setting up reverse shells. But I’ve since discovered this awesome tmate action which lets you interactively debug in the browser or via SSH.

- name: Debug with tmate on failure
  if: ${{ failure() }}
  uses: mxschmitt/[email protected]

With this step if any previous step in your workflow fails a tmate session will be started and the connection info will be repeatedly printed in the workflow output.

Created new session successfully
ssh [email protected]

https://tmate.io/t/xMMK8vwSQyCXdZfTCS9hN7fgx

Much easier!


Original post

When a GitHub Actions workflow fails it would be really nice to be able to interactively debug things with a shell. GitHub doesn’t provide anything like a web console or SSH access to workflow runners so in this post we walk talk through throwing shells with netcat and catching them with netcat and ngrok.

Throwing a reverse shell

The most common way to get a shell on a remote system is to log in via SSH. This provides encryption and authentication and makes the whole process simple. However it requires you to run an SSH server on that system and have network and firewall rules configured to allow incoming traffic, and authentication credentials for that system.

Alternatively you can use a reverse shell, which is where a system will connect out to some other machine on the internet and then forward a shell over that connection. This technique is commonly used in the security community to open backdoors in compromised systems, but is also extremely useful for debugging on a restricted environment such as a CI worker.

Catching a shell

In order to “throw” a shell to a remote system you first have to set up a machine to “catch” the connection.

For this example we are going to use netcat to catch our shell, nc is a standard linux utility that is available on most systems.

$ nc -nlvp 4444
Listening on 0.0.0.0 4444

Now we are listening for incoming connections on port 4444. Beware that this is an unauthenticated and unencrypted connection and we are going to expose it to the internet. For a bit of interactive debugging on open source projects on GitHub this is fine, but this shouldn’t be used for sensitive information or long term solutions.

Forwarding ports with ngrok

I’m also assuming here that the machine you are running this on (my developer laptop in my case) cannot receive traffic on port 4444 via the internet. So we can use ngrok to forward our ports.

Ngrok is a service which allows you to expose ports on your local machine to the internet, for the purposes of developing and testing software.

Once you’ve downloaded and authenticated ngrok you can set up the tunnel.

$ ngrok tcp 4444
ngrok by @inconshreveable                                                                                                                                                                                                     (Ctrl+C to quit)

Session Status                online
Account                       Jacob Tomlinson (Plan: Free)
Version                       2.3.35
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    tcp://2.tcp.ngrok.io:13604 -> localhost:4444

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

In this example we can see that port 4444 on localhost is now also available on port 13604 at 2.tcp.ngrok.io. This will be different every time you create a connection.

Configuring GitHub Actions

Now that we are listening for a shell connection we need to add a step to our GitHub Actions workflow to make the outbound connection.

We probably do not want to leak our connection information into our config. It’s ephemeral so it’s not a huge problem, but storing the connection info in a secret is still good practice.

In your repository head to Settings > Secrets and create DEBUG_HOST and DEBUG_PORT secrets with the hostname and port that ngrok gave us.

Secrets

Then add a last step to your GitHub workflow.

name: Interactive debugging example
on:
  push:

jobs:
  interactive:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/[email protected]

      # Rest of my workflow steps

      - name: Thow interactive shell
        shell: bash -i {0}
        run: |
          rm /tmp/f>/dev/null 2>&1;mkfifo /tmp/f;cat /tmp/f|/bin/sh -i 2>&1|nc ${{ secrets.DEBUG_HOST }} ${{ secrets.DEBUG_PORT }} >/tmp/f

In this last step we use a combination of mkfifo, cat, sh and nc to forward a shell to our remote host.

When your workflow gets to this step it will appear to run indefinitely with no output.

Workflow running

But if we look at the nc session we have running on our local machine we should now see a shell prompt.

Connection received on 127.0.0.1 57724
/bin/sh: 0: can't access tty; job control turned off
$

We can then run bash here to get a more useful shell.

$ bash -i
bash: cannot set terminal process group (2507): Inappropriate ioctl for device
bash: no job control in this shell
[email protected]:~/work/github-actions-shell/github-actions-shell$

What can I do here?

Now that you have a shell on your remote system you can do whatever you like. Just be aware that nc will not forward control commands like ctrl+c and will instead close the nc connection. If this happens you will need to restart nc and restart your workflow.

This is also a simple shell so something like SSH which require a pseudo-tty may not work as expected.

But we can still do things like poke around the runner’s system.

[email protected]:~/work/github-actions-shell/github-actions-shell$ whoami
runner

[email protected]:~/work/github-actions-shell/github-actions-shell$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[email protected]:~/work/github-actions-shell/github-actions-shell$ hostname -f
fv-az12-647.gip0skj2w3au3jd4qdtkx3lorh.cx.internal.cloudapp.net

And most importantly we can now start debugging our CI steps to see what went wrong.

[email protected]:~/work/github-actions-shell/github-actions-shell$ pytest myapp
GitHub Actions, Debugging
Spotted a mistake in this article? Why not suggest an edit!
comments powered by Disqus