[Ksummit-discuss] [TECH TOPIC] Sensors and similar - subsystem interactions, divisions, bindings etc.

Hans Verkuil hverkuil at xs4all.nl
Thu Jul 21 07:39:04 UTC 2016



On 07/20/2016 11:18 PM, Jonathan Cameron wrote:
> Hi All,
> 
> This topic would be around the way the various subsystems interact, in the
> rough area of 'sensors' (I haven't yet had much of an issue with subsystem
> crossing with output devices but maybe that's just over the next hill!)

This happens regularly with drm/kms vs v4l, especially for embedded devices
where video inputs and outputs are often connected internally (i.e. paths
straight from video capture to video output without needing to go to memory
first).

> 
> Scope may well be wider but includes:
> * input (some of it)
> * hwmon
> * iio
> * comedi(?)
> * thermal
> * power/battery
> * gpio - the blur between gpios and beaglescope / PLC type I/O.
> * v4l - when does a device jump from being a multi pixel thermopile
> to a thermal camera? Smart fingerprint scanners etc.  Gesture recognition
> sensors (effectively 9ish pixel cameras)

See for an example of that this patch series that adds support for v4l-touch
devices:

http://www.spinics.net/lists/linux-input/msg45918.html

> * Lots of random things we haven't seen yet.
> 
> All these make use of devices that are at their hearts ADCs. Whilst it
> would nice if the underlying devices always fell into one clear
> category that is not always the case. Hardware descriptions (e.g.
> device tree) are an important area of contention.
> 
> The examples that follow are mostly based around IIO interactions with
> hwmon / input as that's my area of expertise. However, I know that
> similar issues occur at other boundaries.
> 
> Guenter Roeck has been working on hwmon / thermal interfacing recently.
> We also regularly push drivers one way or the other around that boundary.
> 
> Dmitry and I often have to decide on the IIO / input scope boundary
> (and we've moved it over time) + a number of cross over drivers have
> turned up recently.
> 
> The other subsystems I encounter have been 'quieter' in their
> interactions with me but many of the issues below apply to them as well.
> 
> A rough list of related topics that might be worth face to face
> discussion includes:
> 
> 1) Are we happy with the somewhat adhoc divisions of subsystems as they
> stand?
> 
> I am personally happy enough with this, but it's worth checking
> everyone else is. This probably causes more pain for new submitters
> than it does for old hands. Perhaps we can come up with a checklist style
> doc to save people having to ask.
>  
> We have had quite a few cases where a whole driver is submitted to the
> 'wrong' subsystem. Not the best first interaction with the kernel
> world for new contributors. (This sort of area is attracts newbies
> as it's relatively simple and the hardware is fairly cheap and more
> than once their first experience with review as exactly this!)
> 
> If we aren't happy, what do we do about it?
> 
> I'm hoping we are happy or, at least, resigned to the current state
> of affairs, but think the question should be asked from time to time
> as the answer may well change + the discussion will highlight pinch
> points we can work on.
> 
> 2) Bridging drivers - there will always be cross over cases.
> E.g. Generic ADCs used for battery voltage monitoring.  We have a
> number of bridges in mainline already and others out of tree.  How
> important are these and what features would make them more useful?
> 
> Is it ever sensible to have two drivers for the same part because
> the use cases are so different? (I hope there is always a better way
> but maybe not)

One example of this is video output: both drm/kms and v4l can do video
output. Video output using v4l is specifically geared towards frame-based
video output whereas drm/kms is for desktop environments/GPUs. Luckily there
are very few v4l video output drivers and the driver duplication occurs
only for the i2c video transmitter devices. I know of two that are duplicated:
adv7511 and (I think) saa7127. The rarity is the reason nobody ever created
a shim or something similar to allow for one i2c driver to serve both
subsystems. Technically this would be perfectly possible, it's just work.

> This covers both generic bridges (e.g. iio-hwmon) and device
> specific bridges (a good example of a touch screen driver turned
> up in my inbox yesterday).
> 
> 3) Bindings (device tree and similar).  When it is appropriate to use a 
> device tree to describe the overlap (bridging of channels)?
> Sometimes there is obvious real hardware involved (a thermistor on a
> generic ADC input for example), but there are many grey areas.
> 
> Finally bashing out an answer to the issue device tree maintainers
> have with the iio-hwmon bindings would be great. 
> On a less biased note, that example is pretty fiddly to define but would
> act as a good basis to work from more generally.
> 
> We know we got it wrong (back in the dark ages), but it isn't obvious how
> to get it right!
> 
> IIO-input has been partly held outside mainline for years by this issue.
> 
> Also some practical design decisions to make around how to implement
> these mappings to work best with deferred probing etc.
> 
> 4) Should we drop all the bindings for bridging between such subsystems
> and do it all from userspace?
> 
> I think we may end up with a hybrid of the two, but need to be able to
> make it work in 'standard' cases without userspace being involved. That
> hybrid solution may well be devicetree overlay based... Unclear so far.
> + plenty of crazy things that 'might' make sense where there isn't even
> the pretence of representing real hardware.
> 
> 5) Complex device interaction usecases.  At the moment the ones I've come
> across are mostly contained in IIO.  The moment we start sticking in
> MUXes, AFEs (Analog Front Ends) and straightforward analog sensors in the
> mix it can get fiddly.  Swapping war stories may well be worthwhile on
> this. This stuff also turns up in ASoC for example so probably lessons
> to be learned from there.

V4L has a long history of this: if you have multiple video scalers in your
pipeline, how do you control them? E.g. there can be scalers in the video
sensor and in the SoC. We came up with the media controller device to
expose the topology of the hardware and to be able to control each bit
directly from userspace.

While the media controller is quite powerful and can be used to expose
connections across subsystems, it also requires more work from userspace,
and that's an area where not enough attention has been given.

> The analog devices software defined radios are another possible case
> study.

Note that the V4L2 subsystem has support for several SDR devices. Probably
another area where drivers can end up in different subsystems.

> 
> There of course may well be lessons to be learned from similar
> interactions elsewhere in the kernel.
> 
> There is a lot of history in how we ended up where we are (it all made
> absolute sense at the time). Sitting back and taking the time
> to discuss the future would be great.  Whilst this might be solvable
> by email we've made no definitive progress for years
> (and what has been made has been on a case by case basis deep in
> driver reviews.)
> 
> I threw comedi in the list above but, at the moment, I think the more
> likely direction there is a single userspace library abstracting
> the underlying subsystem (Analog Devices are working in that
> direction - perhaps Lars can offer more on that?).

When you get complex systems you probably have to move in that direction.
I'd be interesting in such work going on in other subsystems. As mentioned
above, the media controller needs something like that and it would be
good to learn from others.

> GPIO is another interesting case - a lot of hardware is capable of
> parallel sampling, some at high speeds. It's another area that
> is probably too specialist for this discussion, but if people want
> to dive into the details it might be interesting.
> 
> I think we have only a small amount of fuzz around the v4l boundary,
> but wanted to leave the door open if anyone wants to discuss that
> one further as it's come up a few times over recent years.

There is more of that than we wish, but quite often it is just dropped
in the end as being too much work to solve and not quite important enough
to spend the time and money on.

> The SoC world is a major case of one device, many uses.  Some SoCs
> are turning up with multiple ADC units, sometimes with different
> designs, sometimes simply so that the same hardware can be used for different things.
> 
> A few suggestions for people:
> 
> Guenter Roeck (hmwon)
> Dmitry Torokhov (input)
> Me (Jonathan Cameron - IIO)
> 
> Someone thermal related (not sure who would be most interested)
> Someone power supply related? (Guenter, any suggestions?)
> 
> Someone device tree related (Mark Rutland? Rob Herring?)
> 
> Linus Waleij (gpio)
> 
> Mark Brown - as some of concepts of IIO bridging and how far it
> can be taken (everything via a bridge driver) came from
> discussions with him on SoC ADC handling.
> 
> Lar-Peter Clausen for generally doing insane things with ADCs + having
> one foot in the userspace side of things.

Laurent Pinchart or myself for v4l/mc

Regards,

	Hans


More information about the Ksummit-discuss mailing list