[Ksummit-2013-discuss] NUMA locality for storage

[Ben Hutchings]

On Tue, 2013-07-30 at 14:08 +0000, Christoph Lameter wrote:
> On Tue, 30 Jul 2013, Mel Gorman wrote:
> 
> > I had not given any consideration to what a static API would look like and
> > how it would interact with mempolicies or if it would be an extension of
> > mempolicies but would be interested in discussing it.
> 
> You may also want to consider caching affinities that affect the
> performance. F.e. It may be possible to reduce the locking overhead by
> performing more actions from a single cpu.
> 
> If we cannot do that then at least allow us to set affinities on the
> threads that deal with storage so that we can restrict the threads to one
> or the other caching domain (l3 most promimently).
> 
> Note also that recent storage controllers warm up the L3 cache of the
> socket that connects to their PCI bus when transferring data into memory.
> Access from local cores will be much faster than remote.

It sounds like you're talking about DDIO, but that is a feature of
recent Intel CPUs that doesn't require anything special from the device.

> Performance will degrade significantly in a multi socket system if the
> processing occurs on another socket.
> 
> This effect is so significant that I would recommend that the
> default behavior should be a detection of the NUMA locality of the PCI
> device and the associated cores. I/O processing for that device then needs
> to be restricted to those cores.

PCIe devices may use TLP Processing Hints (TPH) to indicate that DMA
writes will be consumed by a particular CPU; this should cause them to
be delivered to the L3 cache on the appropriate node.  I'm not sure how
much you can expect this to narrow the I/O performance gap, in general.

Ben.

-- 
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
                                                              - Albert Camus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 828 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxfoundation.org/pipermail/ksummit-2013-discuss/attachments/20130730/531f7371/attachment.sig>