[Openais] logging: final call on configuration, output and implementation

[Fabio M. Di Nitto]

Hi all,

those logging threads have been going on for way too long. It's time to
close them and make a final decision. This is a long email, so please
take time to read it all.

This is a recap of what I believe a user would like to see:

- a consistent, easy and quick way to configure logging.
- a reasonable default if nothing is specified.
- a consistent, easy to read, output.

In order to avoid misinterpretation, "debug" priority means DEBUGLO as
defined here by David:
http://www.redhat.com/archives/cluster-devel/2008-November/msg00002.html

== Configuration ==

XML version:

<logging debug="on|off" (read note below about debugging)
         to_syslog="yes|no"
         syslog_facility="local4"
         syslog_priority="debug|info|notice|etc.."
         to_file="yes|no"
         logfile="/var/log/cluster/foo.log"
         log_priority="debug|info|notice|etc..">

  <logger_subsys subsys="NAME" debug=.. />
</logging>

corosync/openais equivalent:

logging {
        debug="on|off"; (read note below about debugging)

        to_syslog="yes|no";
        syslog_facility="local4";
        syslog_priority="debug|info|notice|etc..";

        to_file="yes|no";
        logfile="/var/log/cluster/foo.log";
        log_priority="debug|info|notice|etc..";

        logger_subsys {
                subsys="NAME";
                debug=..
        }
}

(the two configuration are equivalent)

The global configuration (<logging>/logging {) contains all the keywords
required to configure logging as discussed so far and would affects
every daemon or subsystems. The common user would generally stop here.

The logger_subsys can be used to do more detailed control over logging.
This is where more advance users or developers work.

Equivalent entries in logger_subsys override configurations in the
global section.

Use of environment variables and settings from the command line can
override any configuration. This is used by some people to enable
debugging via init scripts (for example /etc/sysconfig/cman or
equivalent for other distros).


== Defaults ==

- debug = off. Switch to "on" to enable execution of debugging code
(read note below) and set log_priority to debug.

- to_syslog = yes.
- syslog_facility = local4 (default to local4 to respect old behaviour).
- syslog_priority = info (default to respect old behaviour and it seems
to be a recent compromise at this point of development, doesn't flood
logs and has enough info).

- to_file = yes
- logfile = LOGDIR/[daemon|subsystem].log. By default each
daemon/subsystem should log in its own file (mostly valid for daemons as
corosync/openais and plugins all share the same file and logging
system).
- logpriority= info (same reason as above)

to_stderr will disappear from the config options. Set to off by default
as most daemons will fork in background, we can set it "ON"
automatically when not running in background (it's the only case where
it actually makes sense).


== Output ==

to_file:

echo $(date "+%b %d %T") subsystem_or_daemon: entry_to_log
Nov 10 19:46:40 subsystem_or_daemon: entry_to_log

It is important to note 2 things. First we need to have a time stamp to
be able to compare logs if daemons are logging to different files, then
we need to know what subsystem is logging if we are logging to the same
file for everything.

Since we don't want to create N combinations that will add/remove
date/daemon_name etc, this is one format can fit them all and it is the
same that would appear in syslog (go for consistency!).

to_syslog:

openlog("subsystem_or_daemon", options, facility);
syslog(priority, "%s", entry_to_log);

(consistent with to_file).

to_stderr:

fprintf(stderr, "%s\n", entry); <- this is really free format.


== Implementations requirements ==

- (strict requirement) must support the configuration options agreed by
everybody.

- (high priority) logging should never fail. For example, if we can't
open a log file we will not fail, we will try to warn by other meanings,
but nothing should block our operations if logging is not working.

- (wish list) logging should be non-blocking. Best if we can delegate
the logging work to something else than ourselves.


== Note about debugging ==

Debugging is an art. Every developer on this planet has a different view
of what debugging is. What we want with the debug="on|off" option is an
absolute quick way to set all the different log_priority="debug" on and
have a flag that can be used in the code to follow different paths.

I often find myself setting <logging debug="on"/> and be done with
everything. No need to remember fancy keywords or understand the whole
implementation details on what overrides what and how.

At some point in future we should probably talk about debugging in
general and what means for all of us but it's outside the scope of this
email.

Fabio