Opened 12 years ago

Closed 12 years ago

#3 closed bug (fixed)

Vserver instances share a common interface and mess themselves up

Reported by: Vincent Caron Owned by: Lucas Bonnet
Priority: major Component: RoR Farm
Keywords: Cc:

Description

Recently we discovered that the network device (and hence all networking support) was "disappearing" quite often in a vserer instance. An ifconfig -a would reveal no device at all.

This FAQ entry looks like a hint:

I suddenly wondered what was the exact meaning of the /etc/vservers/<name>/interfaces/0/dev parameter. I thought it was just a name for the virtual device in the vserver context. The documentation of this parameter is a one-liner.

Actually it looks like it's the name of a real interface in the host, which has the vserver's IP attached on the vserver instance startup (with ip addr add). Vserver is actually partitioning the network device on a per-IP basis. But the interface shown in the vserver is the host's one, not a "virtual one" such as in Xen.

Now in the host:

ziva:~# ip addr ls eth0
2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:16:3e:40:12:18 brd ff:ff:ff:ff:ff:ff
    inet 78.40.120.18/27 brd 78.40.120.31 scope global eth0
    inet 10.0.0.15/16 brd 10.0.255.255 scope global eth0
    inet 10.0.0.16/16 brd 10.0.255.255 scope global secondary eth0
    inet 10.0.0.14/16 brd 10.0.255.255 scope global secondary eth0
    inet 10.0.0.10/16 brd 10.0.255.255 scope global secondary eth0
    inet 10.0.0.12/16 brd 10.0.255.255 scope global secondary eth0

What a mess. Some vserver instance got its IP attached as a primary one (10.0.0.15). When this specific vserver is shutdown, all secondary addresses are dropped. First fix: we need to either:

  • make sure vserver IPs are always attached as secondary ones
  • or create a dummy interface per vserver.
  • or use the net.ipv4.conf.all.promote_secondaries sysctl trick from the FAQ

And in our specific Bearstech case, "eth0" (root server public interface) is definitively the wrong interface to alias. We should use eth2. Or more portable and prettier, rename eth2 as "eth-vrouter" and use that.

Change History (6)

comment:1 Changed 12 years ago by Vincent Caron

You need to be aware of primary/secondary notion of IP addresses interfaces under Linux. It's a bit tricky. For every subnet on a given network interface, you can have one and only one primary address, and any number of secondary address.

The rationale: Linux needs to pick a source IP address when a locally generated packet must be transmitted thru this interface on a given subnet. It uses the primary as a default.

In our case, I think that using only eth2 will solve the problem: it will aways have its primary address set to 10.0.0.1 from the root server setup, and all vservers will use secondaries since they use the same subnet.

Currently, eth0 is a very bad choice: there is a race condition with the first started VM acquiring the primary address for the 10.0/16 subnet. Thus VM are not all created equal. Such a thing will not happen on eth2.

My proposal is to also rename eth2 as eth.vrouter in order we can hardcode this value in source:/ror-farm/ror-farm-add while keeping it generic enough (currently the hardcoded eth0 is not elegant since it depends on the host hardware configuraiton - but I didn't know that until now).

comment:2 Changed 12 years ago by Lucas Bonnet

  • /etc/sysctl.conf edited
  • variable set via sysctl :
ziva:~# sysctl net.ipv4.conf.all.promote_secondaries=1
net.ipv4.conf.all.promote_secondaries = 1
  • eth2 set as interface for new vservers in ror-farm-add (not commited yet, see next question)

How can I rename the interface? ifrename?

comment:3 Changed 12 years ago by Vincent Caron

Nope, I think the sysctl trick is not needed. You should revert it. The or in the 3 proposed solutions were exclusive :).

How can I rename the interface? ifrename?

With Debian you can modify /etc/udev/rules.d/z25_persistent-net.rules if the name must be tied to a specific MAC address (which is suitable in our case I think). Then you have to update /etc/network/interfaces accordingly (s/eth2/eth-vrouter/g) and reboot the domU.

It is also possible to update the interface name dynamically with ip link set dev eth2 name eth-vrouter, although I've not tested this and I don't know how it fits in the ifup/ifdown Debian way of things.

comment:4 Changed 12 years ago by zerodeux

(In [70]) (vcaron) vserver network should now use the proper host interface (see #3)

comment:5 Changed 12 years ago by Vincent Caron

Sysctl conf reverted (to none), eth2 renamed to eth-vrouter in udev rules, root server rebooted. Looks much better now:

ziva:~# ip addr show eth-vrouter|sort -n
    inet 10.0.0.1/16 brd 10.0.255.255 scope global eth-vrouter
    inet 10.0.0.10/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.11/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.12/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.13/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.14/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.15/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.16/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.17/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.18/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.6/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.7/16 brd 10.0.255.255 scope global secondary eth-vrouter
    inet 10.0.0.8/16 brd 10.0.255.255 scope global secondary eth-vrouter
    link/ether 00:16:3e:01:12:18 brd ff:ff:ff:ff:ff:ff

The [[[10.0.0.1}}} host address is primary and permanent, now vserver's seconday may come and go as they wish.

comment:6 Changed 12 years ago by Vincent Caron

Resolution: fixed
Status: newclosed

Works, fixed.

Note: See TracTickets for help on using tickets.