Docs / Security and deployment

Run a model server for your team

For whoever runs the GPU server that people’s Verbiqo apps call. If you only use Verbiqo, you need Choose and check a model instead. The examples use a service account, loopback binding and an API key. Opening the service to laptops comes last, through a TLS gateway that exposes only the required API routes.

Previously checked models

The project records these model checks from 1 October 2026. They describe the model routes checked then; they do not validate the service units or TLS proxy below. The original reports do not record complete server versions, model checksums or driver details.

Deployment status: the Linux service examples below were checked against upstream source and documentation, but have not been runtime-validated on a GPU host. Record the installed versions and run the checks below before using them for your team.

A 40 GB GPU is a starting point to evaluate, not a measured capacity guarantee for these units. Weights, cache precision, context and simultaneous requests all affect memory use. The recorded llama.cpp check used two GPUs; the same configuration on a single smaller card has not been checked.

llama.cpp or vLLM

Both speak the same OpenAI-compatible API, so Verbiqo works the same with either. Start with llama.cpp. Move to vLLM when several people ask at the same time and answers start to wait.

Before you start: versions and installed files

These templates target Linux with systemd 255 or later, vLLM 0.30.0, and llama.cpp commit 8172e6577ac2b35de1ec1e5d1c0aaad6c4a2129f (the revision reported by an earlier model route). Hardening was checked against systemd 255 documentation, and all three service units were parsed by a local systemd 257 validator. These are the source-check references, not verified versions of the historical deployments.

Install a CUDA-compatible server build using its official instructions first: llama.cpp build or vLLM GPU installation. The units assume the resulting executable is at /opt/llama.cpp/bin/llama-server or the Python environment is at /opt/vllm/venv. Install the complete executable and required libraries there, outside home directories, owned by root and readable/executable by llm. Install and verify the selected weights under the exact model path shown below. Do not change the service to root when a path fails.

systemctl --version
/opt/llama.cpp/bin/llama-server --version
/opt/llama.cpp/bin/llama-server --help
/opt/vllm/venv/bin/vllm --version
/opt/vllm/venv/bin/vllm serve --help
nvidia-smi

Run only the commands for your chosen server. Save the versions, GPU/driver details, model revision and weight checksum with the model-check report. The model-specific vLLM parsers below must match your installed model; the historical checks do not prove compatibility of newly downloaded weights.

1. A service account, a model folder and a key

The model server runs as its own account. Its model and credential files are read-only; persistent caches belong under /var/lib/llm. Private temporary storage remains writable. Run these setup commands once on a new host; reuse an existing account or key rather than rotating credentials accidentally.

sudo useradd --system --home-dir /var/lib/llm --create-home --shell /usr/sbin/nologin llm
sudo install -d -m 750 -o llm -g llm /var/lib/llm
sudo install -d -m 750 -o root -g llm /opt/models /etc/llm
sudo sh -c 'umask 027; test ! -e /etc/llm/api-key && openssl rand -hex 32 > /etc/llm/api-key'
sudo chown root:llm /etc/llm/api-key
sudo chmod 640 /etc/llm/api-key

• Put the model files under /opt/models, owned by root, readable by the group and not group-writable: sudo chown -R root:llm /opt/models && sudo chmod -R u=rwX,g=rX,o= /opt/models. Every parent directory must allow traversal. Copy actual files here; symlinks into /home remain blocked by ProtectHome.
• Check ls -l /dev/nvidia* /dev/dri and sudo -u llm /usr/bin/nvidia-smi. If required, add llm to the actual device-owning group with sudo usermod -aG <device-group> llm, then restart the service. Do not make GPU devices world-writable. The units keep device access available; they do not use PrivateDevices=yes.
• The key is what each Verbiqo app sends. Give it to people through your usual secret channel. Verbiqo keeps it in the Keychain or Credential Manager.

2a. llama.cpp as a service

Save this template as /etc/systemd/system/llm.service, owned by root:root with mode 644. It uses the model filename recorded in the earlier check; the new service configuration is not runtime-validated.

[Unit]
Description=Qwen3-30B-A3B-Instruct-2507 (llama.cpp) for Verbiqo
After=network-online.target
Wants=network-online.target

[Service]
User=llm
Group=llm
WorkingDirectory=/var/lib/llm
UMask=0027
Environment=XDG_CACHE_HOME=/var/lib/llm/cache
ExecStartPre=/usr/bin/test -s /etc/llm/api-key
ExecStart=/opt/llama.cpp/bin/llama-server \
  -m /opt/models/qwen3-30b/Qwen_Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf \
  -ngl 99 -fa on \
  -c 32768 -np 1 \
  --alias qwen3-30b --jinja --no-webui \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --temp 0.2 \
  --host 127.0.0.1 --port 8080 \
  --api-key-file /etc/llm/api-key
Restart=on-failure
RestartSec=5
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/llm

[Install]
WantedBy=multi-user.target

• --host 127.0.0.1: only the server itself can reach it. Step 3 opens it to laptops.
• -ngl 99 requests GPU offload for up to 99 layers; inspect startup logs for what was actually offloaded. On two suitable GPUs, -ts 1,1 requests an equal split. Different GPU memory capacities need a different split.
• -c 32768 -np 1 starts with one slot. Quantized q8_0 K/V cache reduces cache memory; -fa on enables Flash Attention for the quantized V cache. Before adding concurrency, check the effective per-slot context and GPU memory in startup logs and repeat the model check under load.
• --temp 0.2 keeps answers steady. Verbiqo sends no temperature unless one is set in its Settings, so this is the one used.
• Tool calling: --jinja enables the model’s Jinja template in the referenced build. Confirm the selected template in startup logs and the actual tool round trip in the model check. A historical successful route does not establish the flags or template of a new service.
• --alias qwen3-30b gives people a stable model name. --api-key-file is supported in this llama.cpp revision and reads one key per line. Keep the file nonempty. It is not a vLLM flag.

2b. vLLM as a service

Create a root-readable systemd environment file from the generated hex key. The system service manager reads it before switching to llm:

sudo sh -c 'umask 077; test -s /etc/llm/api-key && printf "VLLM_API_KEY=%s\n" "$(cat /etc/llm/api-key)" > /etc/llm/vllm.env'
sudo chown root:root /etc/llm/vllm.env
sudo chmod 600 /etc/llm/vllm.env

Use this template instead of the llama.cpp unit. The model path and parser names reflect the previously checked route, but this new unit has not run on a GPU. Verify your installed model supports these parsers before starting it.

[Unit]
Description=Qwen3.6-35B-A3B-AWQ (vLLM) for Verbiqo
After=network-online.target
Wants=network-online.target

[Service]
User=llm
Group=llm
WorkingDirectory=/var/lib/llm
EnvironmentFile=/etc/llm/vllm.env
Environment=HF_HOME=/var/lib/llm/huggingface
Environment=PYTHONUNBUFFERED=1
Environment=PATH=/opt/vllm/venv/bin:/usr/local/cuda/bin:/usr/bin:/bin
Environment=XDG_CACHE_HOME=/var/lib/llm/cache
Environment=VLLM_CACHE_ROOT=/var/lib/llm/cache/vllm
Environment=VLLM_CONFIG_ROOT=/var/lib/llm/config/vllm
Environment=VLLM_NO_USAGE_STATS=1
UMask=0027
ExecStartPre=/usr/bin/test -s /etc/llm/api-key
ExecStart=/opt/vllm/venv/bin/vllm serve /opt/models/Qwen3.6-35B-A3B-AWQ \
  --served-model-name qwen3-30b-instruct \
  --host 127.0.0.1 --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --language-model-only \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --override-generation-config '{"temperature": 0.2, "top_p": 0.8, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.5}' \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder
Restart=on-failure
RestartSec=5
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/llm

[Install]
WantedBy=multi-user.target

• The key: vLLM 0.30.0 reads VLLM_API_KEY; --api-key-file belongs to llama.cpp. The key is absent from the unit and process arguments, but remains in the service environment, accessible to root and potentially other processes with the same account. Use this account only for the model services. API-key authentication covers selected API prefixes, so the remote gateway must restrict routes as shown below.
• Default reasoning setting: --default-chat-template-kwargs '{"enable_thinking": false}'. This is a template default; a request may override it. --reasoning-parser qwen3 separates reasoning content when supported by the installed weights and template. Confirm both behavior and tool calling with the model check.
• Steady answers: --override-generation-config sets server generation defaults. Requests may override them; these settings are an example, not a guarantee of deterministic answers. presence_penalty 0.5 discourages repetition.
• Tool calling: --enable-auto-tool-choice --tool-call-parser qwen3_coder is the choice recorded for the earlier route; check it against your exact model and version.
• Prefix caching: --enable-prefix-caching can reuse shared prompt prefixes. Speculative decoding is omitted: its method and model support need separate version and runtime validation.
• Capacity: --max-num-seqs 8 sets an upper bound of eight active sequences; throughput still depends on the hardware. --language-model-only disables multimodal inputs. It is not a guarantee of a particular memory saving.
• The name: --served-model-name is what people enter as Model in Verbiqo. Keep it stable, so the model can be swapped without changing every computer.
• Compilation and caches: install the compiler tools required by your vLLM/CUDA build, including ninja when required, in the listed PATH. The service has HF_HOME, vLLM cache/config paths and the XDG cache under its writable directory. VLLM_NO_USAGE_STATS=1 disables vLLM usage statistics in this version.
• Hardening: NoNewPrivileges blocks gaining privileges, ProtectSystem=strict makes the filesystem read-only except permitted writable paths, ProtectHome hides home directories, and PrivateTmp isolates temporary files. ReadWritePaths does not grant Unix ownership permissions. For a cache write failure, relocate that cache under /var/lib/llm; retain the protections. GPU device failures need device/group or driver checks.

Start it and test it on the server

sudo -u llm /usr/bin/test -r /etc/llm/api-key
sudo -u llm /usr/bin/test -r /opt/models/Qwen3.6-35B-A3B-AWQ/config.json
sudo -u llm /usr/bin/nvidia-smi
sudo systemd-analyze verify /etc/systemd/system/llm.service
sudo systemctl daemon-reload
sudo systemctl enable --now llm
sudo systemctl status llm --no-pager
sudo journalctl -u llm -n 80 --no-pager
ss -ltn
# Expect 401 with no key and with a wrong key:
curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8000/v1/models
curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8000/v1/models -H "Authorization: Bearer wrong-key"
# Read the real header from stdin, rather than putting the key in process arguments:
sudo sh -c 'printf "header = \"Authorization: Bearer %s\"\n" "$(cat /etc/llm/api-key)"' | \
  curl --config - --fail-with-body -sS http://127.0.0.1:8000/v1/models

For llama.cpp, change port 8000 to 8080 and the model-readability path to the GGUF file. Expect a successful authenticated model list and only a loopback listener; stop before remote access if authentication fails. systemd-analyze verify checks the unit, not GPU execution. A missing model or library needs its installation/path corrected; Permission denied needs parent-directory and ownership checks; a CUDA error needs driver/device checks; out-of-memory needs lower concurrency/context or suitable hardware, followed by another model check. Do not remove authentication or run as root to clear an error.

3. Let laptops reach it over TLS

Each question carries what Verbiqo read from your systems, and the key. So laptops reach the model over HTTPS only, never the server’s plain port. Choose one:

• A gateway you already run, such as LiteLLM, in front of the server. It authenticates clients over TLS, keeps the backend key separately and can serve models under stable names. If it runs on a different machine, use an authenticated TLS backend connection; the loopback examples require a gateway on the same host. An example LiteLLM entry.
• A TLS proxy on the same server. For example Caddy 2.11.4 with a certificate from your organization’s CA. Install Caddy using its official package instructions first. Its service account must read the certificate and private key. With the packaged caddy account, place them under /etc/caddy/tls, directory mode 750 and file mode 640, owned by root:caddy. Keep the private key out of model folders. Save this block in /etc/caddy/Caddyfile:

  llm.example.internal {
      tls /etc/caddy/tls/server.crt /etc/caddy/tls/server.key
      @model_api path /v1/models /v1/chat/completions
      handle @model_api {
          reverse_proxy 127.0.0.1:8000
      }
      handle {
          respond "Not found" 404
      }
  }

  The model server checks keys on these API routes. Other routes, including vLLM’s unauthenticated /invocations, are excluded. For llama.cpp use port 8080. Validate and reload with sudo -u caddy caddy validate --config /etc/caddy/Caddyfile and sudo systemctl reload caddy. The allowlist, authentication forwarding and TLS checks passed locally with Caddy 2.11.4 and a synthetic backend/certificate; the complete GPU deployment remains unvalidated.

Then allow only port 443, and only from the networks your people use (an office range or the VPN). In Verbiqo: provider OpenAI-compatible, endpoint https://llm.example.internal/v1, the key, and the CA under Settings › Trusted CA if it is your organization’s own. To set these for everyone, see Deploy and manage.

Before handing out the endpoint, test from an allowed laptop using your CA: unauthenticated and wrong-key GET /v1/models must return 401, an authenticated request must return the configured model, and /invocations, /metrics and /docs must return 404. Use curl --cacert /path/to/organization-ca.crt with the HTTPS URL; do not use -k. An untrusted CA or certificate for a different hostname must fail. Check that backend ports 8000/8080 cannot be reached from the laptop.

4. Run the model check against it

From a computer with Verbiqo and its saved API key (or VERBIQO_API_KEY supplied by your secret tool), against the address people will use:

verbiqo model-check --provider openai \
  --endpoint https://llm.example.internal/v1 \
  --model "qwen3-30b-instruct" --timeout-ms 300000 | tee model-check.md

Use the name returned by your model list (qwen3-30b for the llama.cpp example). For an internal CA, use the configured Trusted CA or your organization’s trusted CA bundle; do not disable certificate verification. Supply the key through your secret tool or a saved credential, rather than putting a real key into shell history. Keep the report with the exact unit, versions and model provenance. It says Fit for Verbiqo or what to change. What is checked.

vLLM settings that matter

• Context: --max-model-len 32768 or more. Check that the effective context is reported by the endpoint or set it explicitly in Verbiqo; the model-check report records what the client observed.
• Tool calling: --enable-auto-tool-choice --tool-call-parser <parser> with the parser for the model family (the model family alone is not enough: verify its template and version-specific parser). Without them Verbiqo still works, by plain text, more slowly.
• Repetition: if the check reports repeats, --override-generation-config '{"repetition_penalty": 1.05}'. Keep it small: higher values start to alter numbers and names. Check the flag with vllm serve --help, as its form has changed between versions.

The earlier qwen3.8-27b model check used an int4 route behind LiteLLM. Its complete server version and deployment provenance were not recorded, so it is not a reproducible tested service recipe. Validate any replacement weights and parser choices with a fresh model check.