Device specific pass through in host systems - discuss user interface

[Sai Praneeth Prakhya]

> It's interesting to see this from a fresh angle which isn't clouded by 
> other SoC GPU details - thanks for the proposal! A couple more thoughts 
> jump out immediately...
> 

Thanks a lot! for jumping in and providing your valuable insights :)
Sorry! for taking time to reply on this. Did some quick testing on
unbinding/binding stuff.

> > Since, this feature could be generic across architectures, we thought it
> > would be better if the user interface is discussed in the community first.
> > We are envisioning this to be used both during boot time and runtime and
> > hence having a kernel command line argument along with a sysfs entry are
> > needed. So, please pour in your suggestions on how the user interface
> > should look like to make it architecture agnostic.
> > 
> > 
> > 1.      Have a kernel command line argument that takes a list of BDF's as
> > an input and puts them in pass through mode
> > 
> > a.      Accepting BDF as an input has a downside - BDF is dynamic and
> > could change if BIOS/OS enumerates a new device in next reboot
> > 
> > b.      Accepting <vendor_id:device_id> pair as an input has a downside -
> > What to do when there are multiple such devices and user would like to put
> > only some of them in PT mode
> > 
> 
> c. Not all devices are PCI in the first place.

Agreed, maybe we could limit this feature only to PCIe devices :(

> 
> > 2.      Have a sysfs file which takes 1 or 0 as an input to enable/disable
> > pass through mode. Some places that seem to be reasonable are
> > 
> > a.      /sys/class/iommu/dmar0/devices/
> > 
> > b.      /sys/kernel/iommu_groups/<id>/devices
> 
> Note that this this works out a bit tricky to actually use, since 
> changing the meaning of DMA addresses under the device's feet would be 
> disastrous.

Yes, that makes sense.

> It can only safely take effect by unbinding and rebinding 
> the driver and/or resetting the device (as a side note, this starts to 
> overlap with the IOMMU-drivers-as-modules concept, and whatever solution 
> to this we end up with may be helpful in making that work too). In most 
> cases that's probably viable, but not every driver supports unbinding, 
> and either way what if the device in question is the one hosting the 
> root filesystem... :/

We would like to propose something like this

1. User will first unbind the device driver by
echo "BDF" > /sys/bus/pci/drivers/<device_driver>/unbind

2. Set the Pass Through bit (this is to say the intent of the user that upon
the next rebind, please make this device as pass through)
echo 1 > /sys/class/iommu/dmar0/devices/<BDF>/pt

3. Re-bind the driver again
echo "BDF" > /sys/bus/pci/drivers/<device_driver>/bind

So, if the driver doesn't support unbind then the user cannot put the device
in pass through mode, in which case he has to use kernel command line
argument.

I did unbind on my device that has root file system and after that I was
unable to run any command (like cat or echo), which made sense. So, it's upto
the user to decide which devices can go through unbind and hence can be put in
pass through mode. Since this feature requires sudo privileges, I think, we
can assume admin does the right thing.

Please let me know what you think about this and also please do suggest if you
have any other better ways to deal with this.

Regards,
Sai