Bash in Jupyter for Infrastructure Automation

Infrastructure engineers increasingly want notebook-style execution for their automation. Here’s how to run bash in Jupyter Notebook for infrastructure work—and where it breaks down.

Why Notebooks for Infrastructure?

Traditional infrastructure automation uses scripts:

#!/bin/bash
terraform plan
terraform apply -auto-approve
kubectl apply -f manifests/

Scripts are all-or-nothing. When step 2 fails, you start over or hack around it.

Notebooks offer:

• Step-by-step execution: Run one command, inspect, continue
• Inline documentation: Explain what each step does
• Visible state: See outputs without scrolling through logs
• Modification on the fly: Change commands based on earlier output

This is valuable for infrastructure work where you need to observe and react.

Setting Up Jupyter for Infrastructure

Install Required Tools

# First cell: verify tools are available
!which kubectl terraform aws docker

# Add to PATH if needed
import os
os.environ["PATH"] += ":/usr/local/bin"

Configure Authentication

# Kubernetes
%env KUBECONFIG=~/.kube/config
!kubectl config current-context

# AWS
%env AWS_PROFILE=production
!aws sts get-caller-identity

# GCP
!gcloud config get-value project

Create Helper Functions

import subprocess
import json

def run(cmd, check=True):
    """Run command and return output"""
    result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    if check and result.returncode != 0:
        print(f"Error: {result.stderr}")
        raise Exception(f"Command failed: {cmd}")
    return result.stdout

def run_json(cmd):
    """Run command and parse JSON output"""
    output = run(cmd)
    return json.loads(output)

Kubernetes Operations

Deployment Notebook

# Cell 1: Check current state
!kubectl get deployments -n production

# Cell 2: View current image
!kubectl get deployment api -n production -o jsonpath='{.spec.template.spec.containers[0].image}'

# Cell 3: Update image
new_image = "myregistry/api:v2.1.0"
!kubectl set image deployment/api api={new_image} -n production

# Cell 4: Watch rollout
!kubectl rollout status deployment/api -n production --timeout=300s

# Cell 5: Verify
!kubectl get pods -n production -l app=api

Debugging Notebook

# Find problematic pods
pods = run_json("kubectl get pods -n production -o json")

for pod in pods["items"]:
    name = pod["metadata"]["name"]
    phase = pod["status"]["phase"]
    if phase != "Running":
        print(f"⚠️ {name}: {phase}")

# Get logs from failing pod
pod_name = "api-7d4f8b6c9-x2k4j"  # Set from above
!kubectl logs {pod_name} -n production --tail=50

# Describe for events
!kubectl describe pod {pod_name} -n production | tail -20

Terraform Operations

Plan and Apply Notebook

# Cell 1: Initialize
%%bash
cd /path/to/terraform
terraform init

# Cell 2: Plan
%%bash
cd /path/to/terraform
terraform plan -out=tfplan

# Cell 3: Review plan output above before continuing
# Cell 4: Apply (only if plan looks good)
%%bash
cd /path/to/terraform
terraform apply tfplan

State Inspection

# List resources
!terraform state list

# Show specific resource
resource = "aws_instance.web"
!terraform state show {resource}

AWS Operations

EC2 Management Notebook

# List instances
instances = run_json("aws ec2 describe-instances --query 'Reservations[].Instances[]' --output json")

import pandas as pd
df = pd.DataFrame([{
    "id": i["InstanceId"],
    "type": i["InstanceType"],
    "state": i["State"]["Name"],
    "name": next((t["Value"] for t in i.get("Tags", []) if t["Key"] == "Name"), ""),
} for i in instances])
df

# Stop an instance
instance_id = "i-0123456789abcdef0"
!aws ec2 stop-instances --instance-ids {instance_id}

# Wait for stopped state
!aws ec2 wait instance-stopped --instance-ids {instance_id}
print(f"Instance {instance_id} stopped")

S3 Operations

# List buckets
!aws s3 ls

# Sync files
%%bash
aws s3 sync ./dist s3://my-bucket/app/ --delete --dryrun

# Remove --dryrun and run for real
%%bash
aws s3 sync ./dist s3://my-bucket/app/ --delete

Where Jupyter Breaks Down

After building several infrastructure notebooks, you’ll hit walls:

No SSH

Infrastructure often requires running commands on remote servers:

# This works for simple commands
!ssh bastion "uptime"

# This doesn't work
!ssh -t bastion "sudo systemctl restart nginx"  # Interactive sudo fails

Jupyter can’t handle interactive SSH sessions.

Kernel State Loss

If your Jupyter kernel dies mid-operation:

• Environment variables gone
• Working directory reset
• No recovery of partial state

During critical infrastructure changes, this is dangerous.

Slow Startup

When infrastructure is broken, you need to act fast:

1. Open browser
2. Navigate to Jupyter
3. Start server
4. Wait for kernel
5. Open notebook
6. Run cells

That’s 2-3 minutes before your first command runs.

Team Sharing Friction

Sharing infrastructure notebooks means:

• Everyone needs Jupyter installed
• Notebook format doesn’t render well in PRs
• Merge conflicts on outputs are painful

No Audit Trail

Jupyter doesn’t track:

• Who ran which cells
• When commands executed
• What the outputs were

For infrastructure compliance, this matters.

When to Use Jupyter for Infrastructure

Jupyter works for:

• Exploratory automation: Figuring out what commands to run
• Local development: Testing IaC locally
• Learning: Understanding how tools work
• One-off investigations: Debugging with flexibility

When to Use Something Else

For production infrastructure work, consider tools with:

Native SSH Support

Run commands on remote servers without workarounds:

​```bash @production-bastion
kubectl get pods
​```

Plain Text Format

Markdown files that:

• Edit in any tool
• Diff cleanly in git
• Render on GitHub

Terminal-First

No server startup, no browser requirement:

stew run deploy-api.md

Audit Logging

Track every execution for compliance.

Stew: Infrastructure Notebooks That Work

Stew is built for infrastructure automation:

• Markdown format — Plain text, git-friendly
• SSH support — Run commands on remote servers
• Terminal or browser — Works both ways
• Step-by-step — Like Jupyter, but without the overhead
• Audit trail — Who ran what, when

If you’re learning how to run bash in Jupyter Notebook for infrastructure, Stew offers a purpose-built alternative.

Join the waitlist and automate infrastructure the right way.