[PATCH] net: limit a number of namespaces which can be cleaned up concurrently

Eric W. Biederman ebiederm at xmission.com
Fri Oct 14 03:06:28 UTC 2016


Andrei Vagin <avagin at virtuozzo.com> writes:

> On Thu, Oct 13, 2016 at 10:49:38AM -0500, Eric W. Biederman wrote:
>> Andrei Vagin <avagin at openvz.org> writes:
>> 
>> > From: Andrey Vagin <avagin at openvz.org>
>> >
>> > The operation of destroying netns is heavy and it is executed under
>> > net_mutex. If many namespaces are destroyed concurrently, net_mutex can
>> > be locked for a long time. It is impossible to create a new netns during
>> > this period of time.
>> 
>> This may be the right approach or at least the right approach to bound
>> net_mutex hold times but I have to take exception to calling network
>> namespace cleanup heavy.
>> 
>> The only particularly time consuming operation I have ever found are calls to
>> synchronize_rcu/sycrhonize_sched/synchronize_net.
>
> I booted the kernel with maxcpus=1, in this case these functions work
> very fast and the problem is there any way.
>
> Accoding to perf, we spend a lot of time in kobject_uevent:
>
> -   99.96%     0.00%  kworker/u4:1     [kernel.kallsyms]  [k] unregister_netdevice_many                                                                      ▒
>    - unregister_netdevice_many                                                                                                                               ◆
>       - 99.95% rollback_registered_many                                                                                                                      ▒
>          - 99.64% netdev_unregister_kobject                                                                                                                  ▒
>             - 33.43% netdev_queue_update_kobjects                                                                                                            ▒
>                - 33.40% kobject_put                                                                                                                          ▒
>                   - kobject_release                                                                                                                          ▒
>                      + 33.37% kobject_uevent                                                                                                                 ▒
>                      + 0.03% kobject_del                                                                                                                     ▒
>                + 0.03% sysfs_remove_group                                                                                                                    ▒
>             - 33.13% net_rx_queue_update_kobjects                                                                                                            ▒
>                - kobject_put                                                                                                                                 ▒
>                - kobject_release                                                                                                                             ▒
>                   + 33.11% kobject_uevent                                                                                                                    ▒
>                   + 0.01% kobject_del                                                                                                                        ▒
>                     0.00% rx_queue_release                                                                                                                   ▒
>             - 33.08% device_del                                                                                                                              ▒
>                + 32.75% kobject_uevent                                                                                                                       ▒
>                + 0.17% device_remove_attrs                                                                                                                   ▒
>                + 0.07% dpm_sysfs_remove                                                                                                                      ▒
>                + 0.04% device_remove_class_symlinks                                                                                                          ▒
>                + 0.01% kobject_del                                                                                                                           ▒
>                + 0.01% device_pm_remove                                                                                                                      ▒
>                + 0.01% sysfs_remove_file_ns                                                                                                                  ▒
>                + 0.00% klist_del                                                                                                                             ▒
>                + 0.00% driver_deferred_probe_del                                                                                                             ▒
>                  0.00% cleanup_glue_dir.isra.14.part.15                                                                                                      ▒
>                  0.00% to_acpi_device_node                                                                                                                   ▒
>                  0.00% sysfs_remove_group                                                                                                                    ▒
>               0.00% klist_del                                                                                                                                ▒
>               0.00% device_remove_attrs                                                                                                                      ▒
>          + 0.26% call_netdevice_notifiers_info                                                                                                               ▒
>          + 0.04% rtmsg_ifinfo_build_skb                                                                                                                      ▒
>          + 0.01% rtmsg_ifinfo_send                                                                                                                           ▒
>         0.00% dev_uc_flush                                                                                                                                   ▒
>         0.00% netif_reset_xps_queues_gt
>
> Someone can listen these uevents, so we can't stop sending them without
> breaking backward compatibility. We can try to optimize
> kobject_uevent...

Oh that is a surprise.  We can definitely skip genenerating uevents for
network namespaces that are exiting because by definition no one can see
those network namespaces.  If a socket existed that could see those
uevents it would hold a reference to the network namespace and as such
the network namespace could not exit.

That sounds like it is worth investigating a little more deeply.

I am surprised that allocation and freeing is so heavy we are spending
lots of time doing that.  On the other hand kobj_bcast_filter is very
dumb and very late so I expect something can be moved earlier and make
that code cheaper with the tiniest bit of work.

Eric


More information about the Containers mailing list