\input{../../src/parts/template.tex}

\begin{titlepage}
  \newcommand{\HRule}{\rule{\linewidth}{0.5mm}} % Defines a new command for the horizontal lines, change thickness here

  \center % Center everything on the page

  \textsc{\LARGE QubesOS articles}\\[1.5cm] % Name of your university/college
  %\textsc{\Large Nvidia driver debugging}\\[0.5cm] % Major heading such as course name
  %\large Yukikoo\\[0.5cm] % Minor heading such as course title

  \HRule \\[0.4cm]
  { \huge \bfseries GPU}\\[0.4cm] % Title of your document
  \HRule \\[1.5cm]
 % If you don't want a supervisor, uncomment the two lines below and remove the section above
  \Large \emph{Author:}\\
  Neowutran \\[3cm] % Your name

    \begin{multicols}{2}
  \includegraphics[width=80mm]{shinra.png}\\[1cm] % Include a department/university logo - this will require the graphicx package
      \columnbreak
  \includegraphics[width=80mm]{neowutranlogo.png}\\[1cm] % Include a department/university logo - this will require the graphicx package
    \end{multicols}
  \vfill % Fill the rest of the page with whitespace

\end{titlepage}

 \tableofcontents                                                                
    \pagebreak 


\section{Hardware}\label{hardware}

To have an `HVM' for gaming, you must have

\begin{itemize}
\item
  A dedicated GPU. By dedicated, it means: it is a secondary GPU, not
  the GPU used to display dom0.
	In 2023, `Nvidia' and `Amd' GPU work.
  Not tested with Intel GPUs.
		(\href{https://forum.qubes-os.org/t/create-a-gaming-hvm/19000/8}{External GPU using thunderbolt work})
\item
  A lot of patience. GPU passthrough is not trivial, and you will need
  to spend time debugging.
\item
  A screen available for the gaming `HVM'. (It can be a physical monitor
  or just to have multiple cables connected to the screen and switching
  between input source) \textbf{[Optional]}
\item
  Dedicated gaming mouse and keyboard \textbf{[Optional]} .
\end{itemize}

\section{IOMMU Group}\label{iommu-group}

\subsection{Goal}\label{goal}

\paragraph{What}\label{what}

The goal of this step if to retrieve the default IOMMU Group
(\href{https://www.kernel.org/doc/html/latest/driver-api/vfio.html\#groups-devices-and-iommus}{VFIO - "Virtual Function I/O" - The Linux Kernel documentation})
of your hardware.

\paragraph{Why}\label{Why}
It can help understanding potential issue
with your setup (what devices live in the same IOMMU group as your GPU)
/ finding potential workaround. If you feel lucky, skip this step.\\

\paragraph{How}
You can't see your IOMMU Group when you are using Xen (the information is hidden from dom0).

\begin{itemize}
\item
  Boot a live linux distribution
\item
  In the grub, enable iommu: Add the parameters
		\mintinline{text}{iommu=1 iommu_amd=on} to the linux commandline
\item
  Once you logged in your live linux distribution, you need to retrieve
		the folder structure of \mintinline{text}{/sys/kernel/iommu_group}. You can use
  the following script to do that:
\end{itemize}


\begin{minted}{bash}
#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
 echo "IOMMU Group ${g##*/}:"
 for d in $g/devices/*; do
  echo -e "\t$(lspci -nns ${d##*/})"
done
done
\end{minted}

\section{GRUB modification}\label{grub-modification}

You must hide your secondary GPU from dom0. To do that, you have to
modify the GRUB. In a dom0 Terminal, type:

\begin{minted}{bash}
qvm-pci
\end{minted}

Then find the devices id for your secondary GPU. In my case, it is
\mintinline{text}{dom0:0a_00.0}. Edit /etc/default/grub, and add the PCI hiding.

\begin{minted}{bash}
GRUB_CMDLINE_LINUX="... rd.qubes.hide_pci=0a:00.0 "
\end{minted}

then regenerate the grub

\begin{minted}{bash}
grub2-mkconfig -o /boot/grub2/grub.cfg
\end{minted}

If you are using UEFI and Qubes OS 4.2 or earlier, the file to override
with \mintinline{bash}{grub2-mkconfig} is \mintinline{text}{/boot/efi/EFI/qubes/grub.cfg}.

Note: if after this step when you reboot the computer you get stuck in
the QubesOS startup that means you are trying to use the GPU you just
hid. Check your BIOS options. Also check the cables, BIOS have some GPU
priority based on the type of cable. For example, DisplayPort can be
favoured over HDMI.

Once you have rebooted, in dom0, type \mintinline{bash}{sudo lspci -vvn}, you
should see ``Kernel driver in use: pciback'' for the GPU you just hid.

\section{Preparing the guest}
\label{preparing-the-guest}

As of 2025, I recommend using a Linux guest instead of a window guest.

\subsection{Windows}\label{windows}

Install a window VM, you can use this
\href{https://github.com/elliotkillick/qvm-create-windows-qube}{qvm-create-windows-qube}

\subsection{Linux}\label{linux}

Create a new standalone Qube based on the template of your choice.

You must run the kernel provided by the guest distribution, because we
will use some non-default kernel module for the GPU driver. Just follow
the doc:
\href{https://www.qubes-os.org/doc/managing-vm-kernel/\#distribution-kernel}{managing-vm-kernel}.

Install the GPU drivers you need.

\section{Pass the GPU}
\label{pass-the-gpu}

In qubes settings for the HVM, go to the `devices' tab, pass the ID
corresponding to your GPU.

\begin{minted}{bash}
qvm-pci attach gpu_gaming_archlinux dom0:0a_00.0 --persistent
\end{minted}

You may or may not need to add the option ``permissive'' or
``no-strict-reset''. You may or may not need to passthrough additional
devices depending on the result of the IOMMU script.

\href{https://www.qubes-os.org/doc/device-handling-security/\#pci-security}{Some
word about the security implication of thoses parameters.}

\begin{minted}{bash}
qvm-pci attach gpu_gaming_archlinux dom0:0a_00.0 -o permissive=True -o no-strict-reset=True --persistent
\end{minted}

\section{Starting the guest}
\label{starting-the-guest}

\textbf{This is where you will have a lot of issues to debug. Prepare
for intense suffering}

For Linux guests, run `sudo dmesg' to have all the kernel log indicating
you if there is a issue with your GPU driver. For some hardware, the MSI
calls won't work. You can work around that using for example
\texttt{pci=nomsi} or \texttt{NVreg\_EnableMSI=0} or something else.
Check your drivers options. Check if alternative drivers exist (amdgpu,
nvidia, nouveau, nvidia-open, using drivers from the official website,
\ldots). Check multiple kernel version.

For nvidia GPU, I recommand using ``nvidia-open'' drivers instead of
``nvidia''

Some links that could help you to debug the issues you will have

\begin{itemize}
\item
  \url{https://forum.qubes-os.org/t/ryzen-7000-serie/}
\item
  \url{https://dri.freedesktop.org/docs/drm/gpu/amdgpu.html}
\end{itemize}

For windows guests you will probably have the same issues but it will be
harder to debug. I recommend using the drivers from Windows Update
instead of the official drivers from the website of the constructor.

Some things that may be useful for debugging:

\begin{itemize}
\item
  Virsh (start, define, \ldots)
\item
  /etc/libvirt/libxl/
\item
  xl
\item
  /etc/qubes/templates/libvirt/xen/by-name/
\item
  /usr/lib/xen/boot/
\item
  virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/\ldots{}
\end{itemize}

Issues with the drivers could be related to
`qubes-vmm-xen-stubdom-linux', `qubes-vmm-xen', and the Linux kernel you
will be using.

\section{[Case: No dedicated screen, mouse or keyboard] VirtualGL and vulkan}
\label{case-no-dedicated-screen-mouse-or-keyboard-virtualgl-and-vulkan}

In some cases, it will just work out of the box (application will
automatically detect the GPU and will be able to use it without
modification.

In some cases, an additionnal layer is required: using VirtualGL,
mentionned here:
\href{https://forum.qubes-os.org/t/seamless-gpu-passthrough-on-qubes-os-with-virtualgl/20265}{Seamless GPU passthrough on Qubes OS with VirtualGL}

In that case you may need to specify a new variable

\mintinline{bash}{LD_LIBRARY_PATH=/usr/lib}

For nvidia GPU owner, with a hardware that support "Nvidia Prime Offload", Prime offload can be activated with environment variable: 

\begin{minted}{bash}
__NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only __GLX_VENDOR_LIBRARY_NAME=nvidia
\end{minted}

\href{https://forum.qubes-os.org/t/gpu-passthrough-guarantees-with-new-hardware/34336/6}{QubesOS forum - Atrate configuration}

You may have issue with how the mouse is handled by the game. AFAIK, no
one tried to solve this issue.

\section{[Case: Dedicated Screen, mouse and keyboard] Linux guest --- Integration with QubesOS}
\label{case-dedicated-screen-mouse-and-keyboard-linux-guest-integration-with-qubesos}

\subsection{Xorg}\label{xorg}

Now Xorg. From XKCD:

\href{https://xkcd.com/963/}{\includegraphics{x11.png}}

Things you need to install:

\begin{itemize}
\item The Xorg input driver to support your mouse and keyboard
\item Your favorite Windows Manager
\end{itemize}

In my case, it is:

\textbf{archlinux version:}

\begin{minted}{bash}
pacman -S xorg i3
\end{minted}

\textbf{debian version:}

\begin{minted}{bash}
apt install xserver-xorg-input-kbd xserver-xorg-input-libinput xserver-xorg-input-mouse i3
\end{minted}

Then create a XORG configuration file for your GPU and screen. My file
named `xorg.conf':

\begin{minted}{text}
Section "ServerLayout"
Identifier "Passthrough Layout"
Screen 0 "Passthrough Screen" Absolute 0 0
EndSection

Section "Device"
Identifier  "Passthrough GPU"
# name of the driver to use. Can be "amdgpu", "nvidia", or something else
Driver      "driver"
Option "Coolbits" "4"
# The BusID value will change after each qube reboot. 
BusID       "PCI:0::0"
EndSection

Section "Monitor"
Identifier "Passthrough monitor"
EndSection

Section "Screen"
Identifier "Passthrough Screen"
Device     "Passthrough GPU"
Monitor    "Passthrough Monitor"
EndSection
\end{minted}

We can't know what is the correct BusID before the qube is started. And
it change after each reboot. So let's write a script --- named
``xorgX1.sh'' --- that update this configuration file with the correct
value, then start a binary on the Xorg X screen n°1.

\begin{minted}{bash}
#!/bin/bash
binary=${1:?binary required}

# Find the correct BusID of the AMD GPU, then set it in the Xorg configuration file
lspci | grep "VGA" | grep -E "NVIDIA" && sed -i 's/^Driver .*/Driver "nvidia"/g' /opt/xorg.conf
lspci | grep "VGA" | grep -E "AMD/ATI" && sed -i 's/^Driver .*/Driver "amdgpu"/g' /opt/xorg.conf
pci=$(lspci | grep "VGA" | grep -E "NVIDIA|AMD/ATI" | cut -d " " -f 1 | cut -d ":" -f 2 | cut -d "." -f 1 | cut -d "0" -f 2)
sed -i 's/"PCI:[^"]*"/"PCI:0:'$pci':0"/g' /opt/xorg.conf

# Start the Xorg server for the X screen number 1.
# The X screen n__0 is already used for QubesOS integration
sudo startx "$binary" -- :1 -config /opt/xorg.conf
\end{minted}

\subsection{Audio}\label{audio}

\begin{itemize}
\item
  Create a script to launch your favorite Windows Manager and force it
  to use a specific pulse server. Example ``i3.sh'':
\end{itemize}

\begin{minted}{bash}
#!/bin/bash
sleep 5 && sudo setxkbmap -display :1 fr & 
/bin/sudo -u user PULSE_SERVER=unix:/run/user/1000/pulse/native bash -c 'sudo xhost + local:;/usr/bin/i3'
\end{minted}

And launch it:


\begin{minted}{bash}
sudo ./xorgX1.sh ./i3.sh
\end{minted}

\section{Automating most of that}
\label{automating-most-of-that}

I tried to write a script to automate all the previous steps. It is
available here: \href{https://git.sr.ht/~yukikoo/gpu_template}{gpu\_template}

Please be carefull and read the code, not much tests have been done

\subsection{Issues and fixes}
\label{issues-and-fixes}

In one case in a setup with Intel IGPU + Nvidia DGPU, dom0 xorg crashed.
Solved the case by adding a Xorg configuration to explicitly use Intel:

\href{https://forum.qubes-os.org/t/gpu-passthrough-lightdm-service-wont-start/22220/20}{GPU Passthrough - lightdm.service won't start}

\subsection{References}\label{references}

\begin{itemize}
\item
  \href{https://wiki.archlinux.org/index.php/PulseAudio}{Archlinux:
  PulseAudio}
\item
  \href{https://wiki.archlinux.org/index.php/PulseAudio/Troubleshooting}{Archlinux:
  PulseAudio/Troubleshooting}
\end{itemize}

\begin{center}\rule{0.5\linewidth}{0.5pt}\end{center}

\subsection{Contributors}
\begin{itemize}
	\item @deeplow 
	\item @xvrhthxn 
	\item neowutran
\end{itemize}

\batchmode
\end{document}

