[Accessibility-ia2] Handling mathematical expressions in documents

[George Kraft IV]

-------- Forwarded Message --------
From: Bill Haneman
To: George Kraft
Subject: Handling mathematical expressions in documents
Date: Wed, 21 Feb 2007 17:33:18 +0000

Hi folks;

Since I'm not on the IA2 alias I've asked George to forward the 
following comments on our Math discussion from last week.

One thing which I feel strongly about, looking at the proposed API for 
the MathML plugin, is that the modality-specific API doesn't belong in 
IA2 or in the user-agent side of things - first of all, we have to date 
avoided _any_ modality-specific API in our application-side 
accessibility APIs - our model calls for presentation in alternate 
modalities (braille, speech, alternate onscreen-keyboard "menus", etc.) 
to be controlled by the AT client rather than the content provider.  I 
won't revisit the long discussions behind those choices here, but will 
just point out that we've come to that fork in the road before.

It does mean that we need an abstraction API rich enough to cope with 
'real' content, and for math that can be difficult.  That said, I agree 
that we need to be consistent with the APIs and models we are using for 
other kinds of content - i.e. mathematical blocks should still support 
caret navigation (of some kind!), the Text interface when and where (and 
only when!) they draw actual text glyphs to the screen, and the 
'name'/'description' properties.  I am not sure we need a separate Math 
interface, and in fact I think it makes more sense to handle parsing and 
rendering into various modalities as a service which the AT client 
interacts with "outside of" the user-agent interfaces.  The way ATK and 
AT-SPI were designed, at least (and this is the area where I can speak 
most knowledgeably about prior art and intent), this maps well onto the 
existing StreamableContent interface, which would allow complex 
mathematical information to be streamed to the AT client as part of the 
content negotiation model.  The client would not be expected to directly 
cope with presenting this intelligibly to the end-user, but it would 
instead use a purpose-built, possibly mime-type-specific rendering 
service or set of services which understood both the complex data type 
(MathML) and one or more output modalities.

This service model is more in keeping with the way other types of 
content are rendered by clients - for instance, AT clients are not 
usually directly responsible for TTS, but instead use a TTS engine 
(often via some private API); similarly, braille device drivers are 
generally a module separate from the core AT client code (in our 
Linux/Unix world the TTS drivers are a third party component/service 
which is entirely separate from AT such as screen readers - similarly we 
implement magnification via a service model).  In this case, rendering 
of MathML to text or braille becomes another service which the AT client 
accesses via agreed-upon interfaces (public/standard or 
private/contractual).

To summarize:

1) use ROLE_EQUATION or ROLE_MATHEMATICS to hint to client that an 
object contains a mathematical, rather than a textual, expression;
2) support AtkText/Text and AtkComponent/Component interfaces, possibly 
in conjunction with a multi-node hierarchical decomposition of the 
object, to expose UTF-8 textual equivalent content where possible.  This 
is primarily to facilitate simple rendering of non-complex mathematical 
expressions, facilitate simple caret tracking and magnification, etc., 
and for consistency with other objects which are required to expose 
AtkText/Text if they render characters/glyphs to the screen.
2) use AtkStreamableContent (or similar in IAccessible2) to expose 
complex mathematical content with better semantics/structural 
information, with content negotiation which may optionally include 
"text/plain", but should include MathML as one of the streamed data types.
3) the intention of streaming MathML is not in order to force the 
problem onto the client, but rather to allow the client to interact more 
effectively with MathML rendering services. 


best regards

Bill