Pod

Often in time, as an AI application developer, it is of great importance to have a handy development environment to test your AI applications. With Pod, you'll be able to have a AI development environment with:

  • Fast startup time: get your dev environment ready in seconds
  • Easy to use: no need to install anything, just use your browser or terminal to access the dev environment
  • Storage integration: mount your storage to the dev environment and access it

Craete a Pod

To create a Pod, you can use the Lepton CLI to do so or visit the pods page in the Dashboard to create a Pod.

During the pod creation, you can specify the following parameters:

  • Resource Shape: the resource type of the Pod. A10, A100, and H100 are supported. The GPU Count will be 1 by default. If you need more GPUs, you can specify the GPU Count.
  • Node Groups: (Optional) the node group that the Pod will be created in. If not specified, the Pod will be created using the on-demand resource.
  • Nodes: (Optional) Only available once the Node Groups is specified. Specify the list of nodes that the Pod will be created in. If not specified, the Pod will be created in the first available node within the Node Group specified.
  • Pod Name: the name of the Pod, only alphanumeric characters (a-z, 0-9) and '-' allowed.
  • Container Image: (Optional) the container image that will be used to create the Pod. Choose from default image lists or pass in docker image urls like ollama/ollama:latest. If not specified, the default image will be used.
  • File System Mount: (Optional) the file system that will be mounted to the Pod. You can access the file system in the Pod if specified. If not specified, the storage will be ephemeral and will be deleted once the Pod is deleted or restarted.
  • SSH Public Key: (Optional) the public key that will be added to the authorized_keys file in the Pod. You can use this to specify the public key that you want to use to access the Pod.

Advanced Settings

  • Private Image Registry Auth: (Optional)the private image registry authentication that will be used to pull the container image. You can use this to specify the private image registry authentication if the container image is hosted in a private registry.
  • Environment Variables: (Optional)the environment variables that will be injected into the Pod. You can use this to specify the environment variables that your AI application needs.
  • Visibility: (Optional)the visibility of the Pod. You can use this to specify the visibility of the Pod. If the visibility is set to private, only the creator of the Pod can access the Pod. If the visibility is set to public, all the users in the workspace can access the Pod.

Once the pod creation request is submitted, the Pod will be created in seconds. You will be able to see the access information of the Pod in the Pod detail page.

Access the pod via terminal SSH

You can use the SSH command to access the Pod. The SSH command is shown in both the Pod detail page and the Pods list.

Access the pod via browser

You can also access the Pod via Web terminal in the Pod detail page as shown below. Do notice that the Web terminal is more suitable for temporary access. If you want to have a long-term session to the Pod, please use the SSH command.

Access services hosted in a Pod

If you have a service running inside the Pod, you can access the service by using the internal port 18888 and access this service via the public ip and a randomly assigned port. The randomly assigned port is shown in the Pod detail page. Here is an example code of a Gradio application running in a Pod.

import gradio as gr

def greet(name, intensity):
    return "Hello " * intensity + name + "!"

demo = gr.Interface(
    fn=greet,
    inputs=["text", "slider"],
    outputs=["text"],
)

demo.launch(server_name="0.0.0.0", server_port=18888) # speciy the server name and port

You can save the code to a file named main.py and start this application with command python main.py. Then you can access the application via the public ip and the randomly assigned port as indicated below.

Preserve long running sessions

To preserve the long running sessions, you can use the nohup command to run the application in the background. For more information on how to use nohup, you can refer to the nohup documentation. Here is an example code of running the Gradio application in the background.

nohup python main.py --port 18888 --listen 0.0.0.0 > main.log 2>&1 &

After executing this command, the terminal will display the process ID of the Gradio application. You can now disconnect from the Pod, and the application will continue to run in the background. For more information on how to use nohup, you can refer to the nohup documentation

Connect to Pod via SSH

If you are using default image provided by Lepton AI, the SSH service is already configured. If you are using your own image, you can configure the SSH service by change the code below and run it in the Web terminal.

Edit line 4 to 8 below to be your SSH password, SSH public key and Jupyter Lab token. Leave it empty to skip installation.

#!/bin/bash
############ User Space ##############
# Please fill in your Jupyter Lab token here, leave empty to use env value
USER_JUPKEY=""
# Please fill in your SSH public key here (something like ssh-rsa AAAxxxxx lepton@sampleDomain.com), leave empty to use env value
USER_SSHPUB=""
# Please fill in your SSH root password here, leave empty to use env value
USER_SSHKEY=""
######################################

JUPKEY=$LEPTON_POD_JUPKEY
SSHPUB=$LEPTON_POD_SSHPUB
SSHKEY=$LEPTON_POD_SSHKEY
if [[ -n "$USER_JUPKEY" ]]; then
  JUPKEY=$USER_JUPKEY
fi
if [[ -n "$USER_SSHPUB" ]]; then
  SSHPUB=$USER_SSHPUB
fi
if [[ -n "$USER_SSHKEY" ]]; then
  SSHKEY=$USER_SSHKEY
fi


export DEBIAN_FRONTEND=noninteractive
export TZ=Etc/UTC

function InstallSSH {
  if service sshd status > /dev/null 2>&1; then
    echo "OpenSSH server is already started."
    return
  fi
  # Check if OpenSSH server is already installed
  if ! command -v sshd &> /dev/null; then
      echo "OpenSSH server is not installed. Installing..."

      apt update
      apt install -y openssh-server

      echo "OpenSSH server installation complete."
  else
      echo "OpenSSH server is already installed."
  fi

  # Set root password if SSHKEY is provided
  if [[ -n "$SSHKEY" ]]; then
      # Enable password authentication in SSH configuration
      sed -i '/^#PasswordAuthentication/c\PasswordAuthentication yes' /etc/ssh/sshd_config
      sed -i '/^PasswordAuthentication/c\PasswordAuthentication yes' /etc/ssh/sshd_config
      # Enable root login in SSH configuration
      sed -i '/^#PermitRootLogin/c\PermitRootLogin yes' /etc/ssh/sshd_config
      sed -i '/^PermitRootLogin/c\PermitRootLogin yes' /etc/ssh/sshd_config
      echo "Root login is enabled."

      # Display a message indicating that user/password SSH access is enabled
      echo "User/password SSH access is enabled."
      echo "root:${SSHKEY}" | chpasswd
      echo "Root password has been set."
  fi

  # Check if LEPTON_PUBLIC_KEY variable is set and not empty
  if [[ -n "$SSHPUB" ]]; then
      # Create the .ssh directory and authorized_keys file if they don't exist
      if [ ! -d "$HOME/.ssh" ]; then
          mkdir -p "$HOME/.ssh"
          chmod 0700 "$HOME/.ssh"
          echo "Directory $HOME/.ssh created."
      fi
      if [ ! -f "$HOME/.ssh/authorized_keys" ]; then
          touch "$HOME/.ssh/authorized_keys"
          chmod 0600 "$HOME/.ssh/authorized_keys"
          echo "File $HOME/.ssh/authorized_keys created."
      fi
      # Check if the public key is not already present in authorized_keys
      if ! grep -q "${SSHPUB}" "$HOME/.ssh/authorized_keys"; then
          # Append the public key to authorized_keys
          echo "$SSHPUB" >> "$HOME/.ssh/authorized_keys"
          echo "Public key from env variable added."
      fi
  fi

  # turn off PAM to fix sshd login issue
  sed -i 's/UsePAM yes/UsePAM no/' /etc/ssh/sshd_config

  # set default port to 2222
  sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config

  echo "Exposing ENV variables"
  env | sed 's/=/="/' | sed 's/$/"/' > /etc/environment
  echo "set -a; source /etc/environment; set +a;" >> /root/.bashrc

  mkdir /run/sshd
  chmod 0755 /run/sshd

  service ssh start
  echo "sshd service started"
}

function InstallJupyter {
    if pgrep jupyter-lab > /dev/null 2>&1; then
      echo "jupyter already started"
      return
    fi

    # Check if jupyter lab is already installed
    if ! command -v jupyter-lab &> /dev/null; then
        echo "jupyter lab is not installed. Installing..."

        apt update
        apt install python3 python3-pip -y
        pip install -U virtualenv --break-system-packages
        pip3 install jupyterlab --break-system-packages

        echo "jupyter lab installation complete."
    else
        echo "jupyter lab is already installed."
    fi

    jupyter lab --generate-config

    {
     echo "c.ServerApp.ip = '0.0.0.0'"
     echo "c.ServerApp.open_browser = False"
     echo "c.ServerApp.port = 18889"
    } >> ~/.jupyter/jupyter_lab_config.py

    # Set root password if LEPTON_POD_ROOT_PASSWORD is provided
    if [[ -n "$JUPKEY" ]]; then
        echo "c.ServerApp.token = '${JUPKEY}'" >> ~/.jupyter/jupyter_lab_config.py
        echo "Root password has been set."
    fi

    jupyter lab --allow-root > /var/log/jupyter.log 2>&1 &
}

if [[ -n $SSHKEY ||  -n $SSHPUB  ]]; then
  InstallSSH
fi

if [[ -n $JUPKEY ]]; then
  InstallJupyter
fi

Once you have edited the code accordingly, you can copy the code and paste it into the Web terminal and press enter. The SSH service will be configured and you can access the Pod via SSH.