[lsb-discuss] Multi-lib standards

[tytso at mit.edu]

On Mon, Nov 23, 2009 at 01:29:04PM -0600, Bruce Dubbs wrote:
> I wonder how much memory users actually save by running in 32-bit mode 
> on 64-bit systems.  Memeory is pretty cheap, but there are also many who 
> persist on running on 512M when memory is under $40/G.

It depends on the application.  If you have lots of data structures
with pointers or longs, those data structures in the worst case double
in size.  A common problem I've seen is with applications (and in the
kernel as well, BTW) is use of 'long' variables where 32-bit 'int'
would do just fine, but since on x86 longs and ints were the same
sizes, it never got noticed until the app was ported to a 64-bit
platform.

And if you have a large number of VM's the "memory is cheap" argument
may no longer be true.  Sure, 2 gig DIMM's are cheap, but 4 gig and 8
gig DIMM's or SODIMM's --- not so much.  And if you run into the
motherboard-imposed limit to the number of DIMM or SODUMM's on your
system, you can get forced to use the higher density memory sticks,
and then suddenly memory isn't so cheap any more.

(I've had to learn this lesson the hard way, since my laptop only has
2 SODIMM slots, and ever since I upgraded to 64-bits, I've had to use
a VM to run a 32-bit-only Java application which I am forced by my
employer to run --- and boy does life suck.  The 32-bit-only Java
application, running inside a 32-bit VM, is such a pig that 4 gigs of
memory is no longer comfortable for reading work e-mail, now that my
main OS has been upgraded to 64-bits.  Sigh... I may be forced to buy
much more expensive 4-gig SODIMM's for my laptop.)

> > In practice, a 32-bit LSB compliant application will very likely run
> > on a LSB-certified 64-bit version of RHEL and SLES, and less likely to
> > run on Debian/Ubuntu, because of their historical approach to
> > biarch/multiarch.  At the moment we're waving our hands on this issue,
> > and I don't think a realistic solution is to assume that 64-bit "pure"
> > systems are the wave of the future, and that 32-bit applications will
> > disappear.  That potential future would be convenient for
> > Debian/Ubuntu systems, certainly, but I don't think it's likely one.
> 
> Just because something can be done and has been done doesn't mean that 
> it should be supported forever.  After all, IIRC, the reason Linus chose 
> the 386 was that he didn't want to support the 16-bit segmented memory 
> model.
> 
> Mixed models are really a clever hack, but not something the standards 
> should encourage.  Still, I do think that user's should have a choice.

The old story about King Canute and the tide comes to mind.  If you
think the LSB is going to influence the Linux industry about dropping
support of mixed mode, I've got a bridge to sell you.

The reality is that if some customers continue to want to use 32-bit
systems (or are using CPU's that are 32-bit only) then some ISV's will
see the need to support 32-bit apps.  And they may decide, since they
know their application won't see any benefit to 64-bits, to *only*
support a 32-bit application.  At that point, if some users want to
use 64-bit distribution, perhaps because they have more than 4 gigs of
memory, mixed 32/64 support will be mandatory for them.

To have the LSB spec "discourage" such mixed models, when Customers,
ISV's, and Distributions have various use cases which require them, is
like telling the tide not to come in.

> To me, the standard for multi-arch should be an add-on capability that 
> can be removed as easily as a package.  It is up to a distro to ship 
> that capability as default or not, but the user should be able to easily 
> remove it if it is not appropriate for their system.

Whether it is an add-on capability or not is a distribution question,
and not something the LSB should dictate.  The point is that the LSB
needs to be able to *allow* for a mixed model, and it's made difficult
because the some distro's are putting the 64-bit libraries in /lib64
and the 32-bit libraries in /lib, while other distributions have the
64-bit libraries in /lib and the 32-bit libraries in /lib32 (if x86_64
is "native"), and the 32-bit libraries in /lib if (x86_32 is
"native").

The advantage of the method used by Red Hat and Novell is that for a
64-bit platform, the libraries are always in /lib64, and the 32-bit
platform, the libraries are always in /lib.  The disadvantage is that
if we do ever move back to a "pure" model, it seems ugly to have
/lib64 around for ever.

The advantage of the Debian model is that if and when we do get to a
"pure" 64-bit model, /lib will be /lib, forever and always.  The down
side of the Debian model is depending on whether you are booting a
64-bit or 32-bit kernel (i.e., which arch is "native"), the 32-bit
libraries might be in /lib or they might be in /lib32.

Ultimately, the question is whether you think 32-bit and 64-bit
applications will need to co-exist for a short-time, or a long-time.
I personally think the answer is a long-time --- there are many
customers which are still using RHEL 2 and RHEL 3 --- and there are
even some production websites (some of them with highly recongizeable
web URL's) that up until *very* recently were running Red Hat 9
(believe it or not).  You can say that these customers are losers, but
there are real operational reasons why they take their time upgrading
--- and as long as they do, they will real economic pressures on
Distributions and ISV's that are just as irresitable as the tide.

In any case, the problem is that the LSB needs to be able to
accomodate five different cases:

* /lib64 only (RHEL/SLES x86_64)
* /lib only (containing 32-bit libs) (Debian x86_32)
* /lib only (containing 64-bit libs) (Debian x86_64)
* /lib and /lib64 (RHEL/SLES x86_64 mixed)
* /lib and /lib32 (Debian x86_64 mixed)

These already exist today, so we can't just say to everybody, convert
to the One True Way of doing things.  

"I command the tide not to come in!"  :-)

   	   					- Ted