Setup Docker image

Start from the CUDA 8 image from NVIDIA:

FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04

Setup the environment:

RUN echo "/usr/local/cuda-8.0/lib64/" >/etc/ld.so.conf.d/cuda.conf

# For CUDA profiling, TensorFlow requires CUPTI.
RUN echo "/usr/local/cuda/extras/CUPTI/lib64/" >>/etc/ld.so.conf.d/cuda.conf

# make sure we have a way to bind host provided libraries
# see https://github.com/singularityware/singularity/issues/611
RUN mkdir -p /host-libs && \
	echo "/host-libs/" >/etc/ld.so.conf.d/000-host-libs.conf

Then build the container

Launch a test pod

apiVersion: v1
kind: Pod
metadata:
  name: testcuda
  namespace: jup
spec:
  restartPolicy: Never
  containers:
    - image: your_image
      name: ml
      resources:
        limits:
          alpha.kubernetes.io/nvidia-gpu: 1 # requesting 1 GPU
      args: [ "nvidia-smi" ]

      volumeMounts:
        - name: nvidia-driver
          mountPath: /host-libs
          readOnly: true
  volumes:
      - name: nvidia-driver
        hostPath:
          path: /var/lib/nvidia-docker/volumes/nvidia_driver/384.90/lib64

Configure Jupyterhub