[packaging] Meeting next week to discuss trusted third-party repositories

[Denis Washington]

Jeff Johnson wrote:
>
> On Dec 20, 2008, at 12:23 PM, Denis Washington wrote:
>
>> Jeff Johnson wrote:
>>>
>>> On Dec 20, 2008, at 10:02 AM, Denis Washington wrote:
>>>
>>>> Tortanick wrote:
>>>>>> The very last thing ISVs want is yet another package format.
>>>>>> - Dan
>>>>>>
>>>>>
>>>>> I was under the impression that they didn't mind witch package format
>>>>> providing there was a single one that they could support and be 
>>>>> done with it.
>>>>>
>>>>
>>>>
>>>
>>> < ... yet another package format design snipped ...>
>>>
>>>> If I overlooked some fundamental issues, please tell me.
>>>>
>>>
>>> The mechanics of wrapping up {content,scripts,metadata}
>>> in a container are not very hard to achieve. Your approach
>>> is as good (or bad) is every other container around.
>>>
>>> The hardest packaging issue is semantically defining package 
>>> metadata and how the
>>> metadata is to be used by installers and packagers and users and ...
>>
>> Well, the metadata would be:
>>
>
> Good. Below are hysterical comments from well traveled pathways:

OK, I'll be more concrete this time. I'm going to refer a lot to the 
existing LSB here, which already deals with some of these issues.

>> * The package name and provider, for identification and base 
>> directory identification (/opt/provider/name).
>
> Do names/providers have to come from a registry or not?
>
> Are you going to enforce "/opt/provider/name" prefixing for paths?
> If names/provider/paths are not enforced, then how does one handle the
> very predictable name/provider/path collisions that will arise?
>

There is an LSB provider name registry, the The Linux Assigned Names And 
Numbers Authority (LANANA) [1], which would be used for this purpose. If 
a provider doesn't have a LANANA-registered name, a fully qualified 
internet domain name owned by the provider may be used instead, like 
stated in the current LSB specification in section 22.5, "Package 
Naming" (second bullet point) [2].

The prefixing of "/opt/provider/name" is enforced for almost all data. 
For certain types of files (e.g. links to binaries to appear in the 
$PATH or .desktop files for menu entries), common directories for all 
3rd-party packages (e.g. /opt/bin) must exist. In this case, the 
standard would strongly recommend to include the provider and package 
name in the file name to avoid conflicts, e.g. provider-name.desktop, 
but at least in the case of $PATH binaries this may not be enforceable 
(certain binaries may be expected to have a certain name, like gcc).

>>
>> * The package version, to be able to determine if a newer version is 
>> available (for updaters).
>
> You need to make "newer" decideable. for that matter, "available" as 
> well.

The version string would be free-form strings. Only the order of the 
versions in the update feed file would determine if a version is newer 
or older. Consider, for instance, that the feed file defines the 
following versions, in the following order:

 alpha
 beta
 gamma
 delta

Now, if the currently installed version was "beta" and an updater 
checked for updates, it would conclude that "gamma" and "delta" are 
newer versions, and that "alpha" is older. If "beta" wasn't in the 
update feed's version list, it would be assumed that the currently 
installed version is older than the first version listed, that is, that 
all versions are newer than the installed one.

"Available" means the version appears in the feed file.

>>
>> * The package architecture, to check binary compatibility with the 
>> system to install on.
>
> architecture and abi are very different issues. nothing wrong with 
> specifying them at all.
>

The LSB defines an ABI for each of the supported architectures, which 
are "IA32", "IA64", "PPC32", "PPC64", "S390", "S390X", and "AMD64" [3]. 
These lend themselves well to unambigously define the architecture _and_ 
the ABI with a single architecture metadata field (whose value would be 
one of the above-mentioned architectures, or "generic" if the package is 
architecture-independent).

>>
>> * The required LSB version (and modules), for the same reason.
>
> There is no "required" until there is a specification of "required" and a
> mechanism for determining what is "required" (or not).
> ditto for "version" and "modules" for exactly the same reasons.

"Required" means "relies on interfaces defined since the specified 
version of the LSB", or, more broadly, "relies on compliance 
requirements made by the specified version of the LSB specification". 
This means that a package requires the lowest LSB version in which all 
the packaged software's assumptions about existing interfaces and the 
base system in general are met due to mandatory requirements made by the 
LSB.

For checking LSB compliance to a specific version, there are some tools 
directly from the LSB, like the LSB Application Testkit [4]. I don't 
know if those are sufficient, but I think extending these tools to 
(largely) be so wouldn't be impossible.

>>
>> * The URL of an update feed file, to check for new versions (in 
>> combination with the version data).
>
> All of "URL", "update", "feed", "new", "versions", "combination" and 
> "data" are not specified.
>

An "URL" means a standard RFC-3986 URI [5], limited to a well-defined 
list of protocols (HTTP would most likely be in this list).

The "update feed file" is a simple file that lists all versions of a 
package, ordered from the oldest to the newest version. Each "version" 
is is itself just a package with the same provider and package name (but 
a different version, naturally). I have already covered the definition 
of a "new" version and the "combination" of version data and update URL 
(the URL tells the updater where and how to locate and retrieve the feed 
file).

With "data" i simple meant the free-form version string.

>>
>> * The paths of the installed files and directories, plus their 
>> permission modes, for associating the files with the package, and to 
>> inform the user in case files are installed outside of the "trusted" 
>> directories.
>>
>
> Ditto (presumably I don't have to list all the details, but a LSB 
> packaging standard will have to).
>

Well, the file and directory paths are absolute Unix file paths (ok, you 
haven't expected anything else ;)). The permissions are rwxrwxrwx-style; 
a user and group name might be specifiable too if needed (both would 
fall back to "root" if the user/group doesn't exist on the system).

The Filesystem Hierarchy Standard [6], which a LSB-compliant 
distribution must comply to, defines some directories which are a 
natural fit for "trusted" directories:

* Application data: /opt/provider/package
* Configuration: /etc/opt/subdir (/etc/opt/provider/package would be 
enforced)
* Variable data: /var/opt/subdir (same as above)

Furthermore, /opt/bin is also mentioned in the FHS and would too be 
considered trusted. Other directories traditionally found in /usr or 
/usr/local may also need /opt equivalents, like /opt/applications for 
.desktop files.

All other directories would be considered untrusted.

>> As far as I can see, this is all metadata that is required for a 
>> third-party application. I for my part wouldn't need anything else at 
>> least. The only possible useful addition to this list may be the 
>> package files' md5 or sha1 sums for data integrity checks, but this 
>> is not a requirement. All other metadata found in other formats as 
>> RPM or DEB are more of interest for base system distribution 
>> maintainance as far as I can tell (for instance all package 
>> relationship metadata like Requires, Replaces, Epoch etc), and don't 
>> seem to be needed for self-contained third-party packages.
>>
>> So, for third-party packaging needs, we have a very small and 
>> well-defined set of metadata IMO. I hope I haven't misunderstood your 
>> definition of "defining the metadata"; correct me if I'm wrong.
>>
>
> Truly, this is a good start. No place in your suggestion have you 
> mentioned "format" of packages,
> and there are no API/ABI methods being proposed, nor are there any 
> defacto definitions, e.g.
>     just like 0install/deb/installshield/klik/rpm/klik/...
>

Thanks.

> I'd suggest adopting some conventions for specifying metadata concretely.
>
> Note that specification != representation, one has to avoid __ALL__
> existing packaging implementations in order to hammer out packaging
> metadata semantics and committing to __ANY_ representation of package
> metadata that may be distributed.
>
> But sure there's *LOTS* of defacto examples for guidance around, 
> examine any existing
> package manager implementation for assistance in specifying LSB 
> package metadata.
>

This is probably something I haven't done enough. I'll have too look 
into more existing package management solutions.

> But it __MUST__ be a "LSB Package" standard first and foremost and ab 
> initio to
> succeed imho.
>
> 73 de Jeff
>
Thank you for the very constructive criticism. I hope we get something 
worthwhile out of this discussion.

Cheers,
Denis

[1] http://www.lanana.org/
[2] 
http://refspecs.linux-foundation.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/pkgnameconv.html
[3] http://www.linuxfoundation.org/en/Specifications
[4] http://www.linuxfoundation.org/en/Downloads#LSB_Application_Testkit
[5] http://labs.apache.org/webarch/uri/rfc/rfc3986.html
[6] http://www.pathname.com/fhs/pub/fhs-2.3.html