Deploy a LLM Chat-GPT like service on Jetstream

jetstream
llm
Published

October 31, 2024

In this tutorial we will deploy a LLM Chat-GPT like service on a GPU node on Jetstream. For experimentation purposes we are using the smaller and cheapest GPU node available on Jetstream, the g3.small which has a virtual GPU with 8GB of memory, and deploy the meta-llama/Llama-3.2-1B-Instruct model which is a 1.3B parameter model.

However the same instructions can be used to deploy any other model available on the Hugging Face model hub.

This tutorial is based on work by Tijmen de Haan, the author of Cosmosage.

Choose a model

Jetstream has GPU nodes with 4 NVIDIA A100 GPUs, a user can create a Virtual Machine with 1 entire GPU or a fraction of it.

The most important requirement is the GPU memory available to load the model parameters, Jetstream provides:

Instance Type GPU Memory (GB)
g3.small 8
g3.medium 10
g3.large 20
g3.xl 40

So g3.xl is the largest available and gets an entire A100 GPU with 40GB of memory.

Therefore we need to make sure that the model we want to deploy fits in the available memory.

Llama 3.2 1B model uses about 6.5GB of memory, so it fits in the g3.small instance.

We also need to make sure the model has been fine-tuned for responding to text prompts, generally those models are marked as Instruct.

Create a Jetstream instance

Login to Exosphere, request a Ubuntu 24 g3.small instance, name it chat and ssh into it using either the SSH key or the passphrase generated by Exosphere.

Install miniforge

We will use Miniforge to create 2 separate Python environments, one for the Hugging Face model serving and one for the web interface.

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

Configure vllm to serve the model

Create the environment:

conda create -y -n vllm python=3.11
conda activate vllm
pip install vllm

As we are using a Llama model, we need specific authorization, you can login to Hugging Face and request access to the model on the model page.

Next we can serve the model:

huggingface-cli login
vllm serve "meta-llama/Llama-3.2-1B-Instruct" --max-model-len=8192

If this starts with no error, we can kill it with Ctrl-C and create a service for it.

Create a file /etc/systemd/system/vllm.service with the following content:

[Unit]
Description=VLLM model serving
After=network.target

[Service]
User=exouser
Group=exouser
WorkingDirectory=/home/exouser

# Activating the conda environment and starting the service
ExecStart=/bin/bash -c "source /home/exouser/miniforge3/etc/profile.d/conda.sh && conda activate vllm && vllm serve 'meta-llama/Llama-3.2-1B-Instruct' --max-model-len=8192 --enforce-eager"
Restart=always
Environment=PATH=/home/exouser/miniforge3/envs/llm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

[Install]
WantedBy=multi-user.target

Then enable and start the service:

sudo systemctl enable vllm
sudo systemctl start vllm

In case of errors:

  • Check the logs with sudo journalctl -u vllm
  • Check the status with sudo systemctl status vllm

You can also check the GPU usage with nvidia-smi:

Thu Oct 31 16:51:09 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.06             Driver Version: 535.183.06   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  GRID A100X-8C                  On  | 00000000:04:00.0 Off |                    0 |
| N/A   N/A    P0              N/A /  N/A |   6400MiB /  8192MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     53332      C   ...miniforge3/envs/vllm/bin/python3.11     6399MiB |
+---------------------------------------------------------------------------------------+

Configure the chat interface

The chat interface is provided by Open Web UI.

Create the environment:

conda create -y -n open-webui python=3.11
conda activate open-webui
pip install open-webui
open-webui serve

If this starts with no error, we can kill it with Ctrl-C and create a service for it.

Create a file /etc/systemd/system/webui.service with the following content:

[Unit]
Description=Open Web UI serving
After=network.target

[Service]
User=exouser
Group=exouser
WorkingDirectory=/home/exouser

# Activating the conda environment and starting the service
ExecStart=/bin/bash -c "source /home/exouser/miniforge3/etc/profile.d/conda.sh && conda activate open-webui && open-webui serve"
Restart=always
Environment=PATH=/home/exouser/miniforge3/envs/open-webui/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

[Install]
WantedBy=multi-user.target

Then enable and start the service:

sudo systemctl enable webui
sudo systemctl start webui

Configure web server for HTTPS

Finally we can use Caddy to serve the web interface with HTTPS.

Follow the instructions to Install Caddy

Modify the Caddyfile to serve the web interface:

sudo vim /etc/caddy/Caddyfile

to:

chat.xxx000000.projects.jetstream-cloud.org {

        reverse_proxy localhost:8080
}

Where xxx000000 is the allocation code of your Jetstream instance.

Then reload Caddy:

sudo systemctl reload caddy

Connect the model and test the chat interface

Point your browser to https://chat.xxx000000.projects.jetstream-cloud.org and you should see the chat interface.

Create an account, click on the profile icon on the top right and enter the “Admin panel” section, open “Settings” then “Connections”.

Under “OpenAI API” enter the URL http://localhost:8000/v1 and leave the API key empty.

Click on the “Verify connection” button, then to “Save” on the bottom.

Finally you can start chatting with the model!

Use a larger model using quantization

The weights of LLMs can be quantized to a lower precision to reduce the GPU memory required to run them, often larger models with quantization outperform smaller models with no quantization. Hugging Face has several quantized models, the most popular are GGUF models, but vllm has just experimental support for that format, so better search explicitely for a model “quantized for vllm”, for example hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4.

sudo systemctl stop vllm
vllm serve "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4" --max_model_len 4096 --gpu_memory_utilization 1 --enforce-eager

Modify the systemd service replacing the relevant line with:

ExecStart=/bin/bash -c "source /home/exouser/miniforge3/etc/profile.d/conda.sh && conda activate vllm && vllm serve 'hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4' --max-model-len=4096 --gpu_memory_utilization 1 --enforce-eager"

Then restart the service:

sudo systemctl daemon-reload
sudo systemctl start vllm

Check nvidia-smi, memory consumption should be about 7.5 GB.

Unfortunately we needed to also decrease max-model-len to fit in such a small GPU, so the model will only support 4096 tokens, so it would be best to deploy this model on a slightly larger Virtual Machine and increase the number of tokens.