[Ksummit-discuss] [TECH TOPIC] Overlays and file(system) unioning issues

[David Howells]

I would like to propose a technical session on filesystem unioning.  There are
a number of issues:

 (1) Whiteouts.

     Linus's idea that a union layer or overlay mounted not as part of a union
     but separately, should expose whiteouts as 0,0 chardevs.  Whilst this
     might indeed make the backup tools easier as things like tar can then use
     the stat() and mknod() interfaces rather than having to use special
     ioctls or syscalls, Miklós's idea to implement them as actual 0,0
     chardevs in the underlying filesystem incurs some problems:

     (a) It's slow and resource intensive.

     	 Every whiteout requires an inode to represent it.  This means that if
     	 you, say, have a directory in the lower layer that has a few thousand
     	 inodes in it and you delete them all, you then eat up inode table
     	 space in the upper layer.

	 Further, every chardev inode has to be stat'd to see if it is really
	 a whiteout.

     (b) It has provided lock ordering issues in overlayfs directory reading
     	 because overlayfs has to stat each chardev from within the directory
     	 iterator.

     I have patches to make Ext2 and JFFS2 use special directory entries
     labelled with DT_WHITEOUT and no inode.  This is more space efficient and
     faster and can be extended to Ext3 and Ext4.  XFS has constants defined
     for doing similar.

     I would propose that we change overlayfs to do this.

     Unfortunately, we would still have to support the then obsolete 0,0
     chardevs on disk.

     The stat() and mknod() syscalls would then have to present these objects
     to the user as 0,0 chardevs rather than ENOENT errors.  To do this it
     might be necessary to have a special mount flag to turn off the
     translation to DENTRY_WHITEOUT_TYPE dentries and record them as
     DENTRY_SPECIAL_TYPE instead with an in-memory inode struct showing it to
     be 0,0 chardevs.

     David Woodhouse did make an additional suggestion that would make 0,0
     chardevs less space inefficient - and that's to hard link a reserved
     inode.

 (2) Opaque inodes.

     Should we use an xattr to mark inodes as opaque or should we use an inode
     flag?  I have patches to add such an inode flag for Ext2 and JFFS2.
     Marking the inode would be more space and time efficient.

 (3) Fall-through markers.

     Unionmount - and possibly other filesystem unioning systems - perform
     directory integration on disk.  (Note that overlayfs maintains this in
     memory for the lifetime of a directory inode).

     With unionmount, an integrated directory is marked as being opaque with
     special directory entries of type DT_FALLTHRU indicating where there is
     stuff in lower layers that can be accessed.

     Should we, perhaps, declare that the user sees such markers as 0,1
     chardevs when the layer is not mounted as part of a union?

 (4) Unionmount and other filesystem unioning systems.

     Do we want to add other filesystem unioning systems into the kernel?
     I've brought in a lot of the stuff for unionmount to help support
     overlayfs.  Unfortunately, overlayfs interferes with some of the stuff
     that unionmount wants to do (e.g. doing whiteouts differently and in an
     awkward manner).

 (5) Lack of POSIX characteristics.

     There have been complaints that overlayfs isn't sufficiently POSIX like.
     Now, this is by design on the part of overlayfs and I agree with the
     Miklós that this is the right way to do it.  However, some mitigation
     might be required.

     One of the most annoying features is the fact that if you do:

	fd1 = open("foo", O_RDONLY);
	fd2 = open("foo", O_RDWR);

     then fd1 and fd2 don't necessarily point to the same file.

     I have been given patches by Ratna Bolla that speculatively copy the file
     into the overlayfs file inode as the pages are accessed and direct file
     accesses to the overlay inode rather than one of the two layers.  I saw a
     number of problems with the approach, but it's possible his latest patch
     fixes them.

 (6) File-by-file waiver of unioning.

     Jan Olszak has requested that it be possible to mark files in one of the
     layers to suppress copy up on that file and to direct writes to the lower
     layer.  This causes problems with rename however.

 (7) File locking and notifications.

     These are similar issues.  IIRC, we decided at the Filesystem Summit that
     you get to take locks on the union inode only and that the notifications
     only follow changes to the upper layer.  This means that you don't get
     union/union interactions through a common lower layer.

     However, we've since had complaints that tail doesn't follow changes made
     to the lower layer (from James Harvey).

 (8) LSMs and unions/overlays.

     Path-based LSMs should just work now that file->f_path points to the
     union layer inode, though they may require namespace awareness.

     Label-based LSMs are another matter.  file->f_path.dentry->d_inode points
     to the top layer label and file->f_inode points to the lower layer label.
     Currently the user of the overlay can 'see through' the overlay and
     access lower files in terms of the labels from the lower layer when doing
     file operations, but uses the label from the upper layer when doing inode
     operations.  I think this should be consistent and should only use the
     upper layer label.  I'm working on patches to get this to work, but there
     is dissension over which label should be seen.

     Further, mandating that the upper label should be seen does cause
     unionmount a problem as there's no upper inode to hang the label off.
     This means that the label must be forged anew each time it is required
     until at such time a copy-up is effected.

David