Automatic Django Shells Using Expect

One of the standout features of tools like Docker and Kubernetes is the ability to quickly exec into a production shell. With environment variables configured, working directory set, and everything ready to go, you’re primed for action. But not every project uses Docker in production, which makes getting into a production shell a small hassle. When working in these environments, starting a session often involves several repetitive steps:

  • SSH into server
  • Change user
  • Activate the virtualenv
  • Change to the correct directory
  • Set an environment variable
  • Finally, open the shell

Maybe you have less steps, maybe you have more. The ultimate goal is the same: to get an interactive shell after running a sequence of commands. This is where Expect, an underappreciated automation tool for Unix, shines.

Introducing Expect

I first came across expect while reading Don Libes’ Expect: A Surprisingly Underappreciated Unix Automation Tool. Initially, I dismissed it, thinking it wouldn’t fit into my day-to-day work. I’ve never had to automate command-line programs before. However, a recent challenge changed my perspective. I wanted to automate the process of jumping into a remote Django shell, and Expect turned out to be the perfect solution.

While my specific use case involved Django, this solution is language-agnostic and works for various scenarios requiring interactive command automation. Here’s a quick example of what Expect can do:

spawn ssh mysshhost

expect "$ "
send "sudo su - myuser\r"

expect "$ "
send "source /my/virtualenv/bin/activate\r"

expect "$ "
send "cd /my/data/directory\r"

expect "$ "
send "python manage.py shell_plus\r"

expect ": "
send "users = User.objects.filter(is_active=True, is_staff=True)\r"

interact

How It Works

Let’s break this script down step by step:

  1. Initiating the Session: It all starts with spawn ssh mysshhost, which initiates the SSH session.

  2. Automating Commands: Expect waits for specific prompts (like $ ) and then sends the corresponding commands, such as switching users (sudo su - myuser), activating a virtual environment (source /my/virtualenv/bin/activate), and navigating to a directory. You can also use this to set environment variables, such as DJANGO_SETTINGS_MODULE

  3. Starting a Python Shell: In this case, the script starts a Django shell using python manage.py shell_plus. My projects rely on django-extensions for its shell_plus feature, which makes working in Django much more pleasant.

  4. Executing Python Commands: You can even send initial Python commands, such as querying active staff users: users = User.objects.filter(is_active=True, is_staff=True). (Pro tip: If your Python code includes square brackets, you’ll need to escape them properly.)

  5. Interactive Mode: Once the setup is complete, the interact command hands control back to you, leaving you in an interactive shell.

When to Use Expect

Expect is an excellent tool for automating repetitive steps in interactive sessions. It shines in scenarios where you need to run a series of commands to reach a particular state or you don’t have to act on any of the command’s output. However, if you are trying get read data back to operate on, it’s time to switch back to bash scripts. For example, if you wanted to do something like get all the users emails and loop through them, it would be better to use bash:

output=$(
    ssh myserver \
        'sudo -u myuser bash -l -c "\
            source /path/to/venv/bin/activate && \
            cd /my/python/dir && \
            python manage.py shell_plus -c \"users = User.objects.filter(is_active=True, is_staff=True); print(\\\"||||\\\"); print(users) \"\
        "'
)

users=$(echo "$output" | sed -n '/||||/,$p' | tail -n +2)

If you find yourself jumping into servers on the regular, Expect can be a great asset to have in your tool belt. It’s a simple yet powerful tool, designed for a specific use case. One that any platform engineer is sure to find a use for.