After a lot of pain and lookup for configuration, I have finally succeed to an very complex network setup that lead to an easy administration result.
I have one server on ESXI, and I host it at “Online.net”. For one server, I have 1 router available to allow my virtual machine to go out.
I have multiple public IP addresses, and generally one public ip is dedicated to one virtual machine.
But the configuration I want, is :
- One virtual machine, the “router”
- One virtual network card on the “router” per public ip address
- One virtual network card on the “router” for the internal network where all my servers will be connected
- OpenVPN with a tun network to access to my internal network
- A wide subnetwork (10.90.0.0/16), dispatch into multiple class C (10.90.10/24, 10.90.20/24) network, each one go thought different public IP
- A dhcp / dns server (dnsmasq) to attribute automatically the right IP to each server
The goal is for any server that need to be server by a specific public address, they receive a fixed internet address on a specific subnetwork. So they will go out and receive connection from a specific public address.
For any other kind of server (private one), they can obtain automatically a internal address and go out with the main public address (the one with the VPN access).
I cannot setup easily the route process, because I only have one IP to goes out for the router, and I will need to share that IP with multiple public address.
And you cannot receive a request from an IP address, and answer by another IP (martians package). If the route by default is you main access, then it will try automatically to go thought this connection when he want to go on the internet. And if the incoming request was from another IP address, it fail.
What we need to setup :
- One main routage for the main access (the one with the VPN)
- One table per public IP address
- One rule for the each public ip address
- One rule for the subnetwork dedicated to this public ip address
- One rule for the VPN address
Here the schema of my network :
Now let see the configuration :
We need a Virtual Machine with 1 network card with automatic hardware address (eth0), one network card with the main public ip address (eth1 with a fixed hardware address), one network card with a secondary public ip address (eth2 with fixed hardware address). Of course you can add all your public ip addresses the same way.
Here my file /etc/network/interfaces :
# The loopback network interface
iface lo inet loopback
# The primary network interface
#local sub network
iface eth0 inet static
post-up ip rule add to 10.90.8.0/24 lookup main prio 1000
#ip static of access
iface eth1 inet static
post-up route add 126.96.36.199 dev eth1
post-up route add default gw 188.8.131.52
dns-nameservers 184.108.40.206 220.127.116.11
#ip static of devel
iface eth2 inet static
post-up route add 18.104.22.168 dev eth2
post-up ip route add default via 22.214.171.124 dev eth2 table devel
post-up ip rule add from 10.90.20.0/24 lookup devel prio 1001
post-up ip rule add from 126.96.36.199 lookup devel prio 1002
The main card add in the “main” route the default gateway. I also add a “ip rule” to use the “main” route table when a request go to the VPN. It avoid trying to reach the vpn by one of the public ip address.
The secondary public IP has several “route” rules :
- First we say if we use the eth2 ethernet, we will reach the “Online” router thought the eth2 card. It will receive the correct hardware address and accept the connection. We set this in a “devel” table, not the “main” one.
- Then we configure any network packet that come from the DEVEL subnetwork or come from the public ip address to use the “devel” table in higher priority. It will result of internet transfert thought the eth2 ethernet card, and not the default one “eth1″.
We also need a “devel” table. It’s better to use a name instead a number. It simpler to understand and it’s also easier to have the list of available table and the meaning like this.
Here my /etc/iproute2/rt_tables files :
# reserved values
You can now reboot your server. The network card will be configured properly. We need next, to configure the firewall.
I use “shorewall” for that purpose. Inside /etc/shorewall directory :
The “interfaces” file :
#ZONE INTERFACE OPTIONS
loc eth0 dhcp
net eth1 routeback
devel eth2 routeback
vpn tun0 dhcp
My local network and my vpn obtain ip by dhcp. The main network “net” and the devel network “devel”, has a “routeback” rule, to send back anydata that come from this card to the same way.
My “masq” file :
#INTERFACE SOURCE ADDRESS PROTO PORT(S) IPSEC MARK
The “vpn” has a masquerade with the local network. The subnetwork “10.90.10.0/24″ use the “main” network as a masquerade, and the subnetwork “10.90.20.0/24″ use the “devel” network as a masquerade.
My “policy” file :
#SOURCE DEST POLICY LOG LEVEL LIMIT:BURST
loc all ACCEPT
vpn loc ACCEPT
vpn $FW ACCEPT
$FW all ACCEPT
net all DROP info
# THE FOLLOWING POLICY MUST BE LAST
all all REJECT info
It allow all the local network to reach any network. You can also forbad the communication between each subnetwork. The vpn has only access to the firewall and the local network. I don’t share the internet connection to the VPN. The firewall can reach any network.
Any other communication is drop by default.
My “zones” file :
#ZONE TYPE OPTIONS IN OUT
# OPTIONS OPTIONS
I simply declare all the network zone in the “zones” file.
You will have to setup the “rules” file based on your need.
The important rule for the vpn :
OpenVPN(ACCEPT) net $FW #VPN
After that setting, you can connect to the vpn only by “OpenVPN”. Add a rule for SSH to simplify the starting configuration.
To set this configuration at “boot”, in the “shorewall.conf” change this :
And in /etc/default/shorewall :
Then reboot, and your firewall will be configured properly.
Let’s add “dnsmasq” has a dns and dhcp server.
In my “/etc/hosts” file, I setup some static ip on the secondary network :
#static devel route
10.90.20.10 postfix postfix.celogeek.fr postfix.celogeek.com
10.90.20.11 tasks tasks.celogeek.fr tasks.celogeek.com
10.90.20.12 gitorious gitorious.celogeek.fr gitorious.celogeek.com
In my “/etc/dnsmasq.conf” I add the dhcp setting :
Any short name will automatically add the “in.celogeek.fr” subdomain and resolv thanks to the local host file and the dhcp information.
A machine will send his name, and it will be save in the dhcp server and it will be possible to resolv his name afterward.
For the statics “ip”, we need to get their hardware address, and give them a name in dnsmasq :
my “/etc/dnsmasq.d/devel-hosts” :
You can restard again. Then when you start your server, it will obtain a dynamic ip that is assign to the main ethernet network, and a static ip in another subnetwork and go thought the “devel” network card.
Now let’s configure the “OpenVPN” service. I strongly encourage the use of the “easy-rsa” to generate the key of the server and the key for the client.
Here the special part for the network in your “OpenVPN” config file :
key easy-rsa/keys/server.key # This file should be kept secret
server 10.90.8.0 255.255.255.0
push "route 10.90.0.0 255.255.0.0"
push "dhcp-option DNS 10.90.8.1"
push "dhcp-option DOMAIN in.celogeek.fr"
When you connect to the VPN, the subdomain “in.celogeek.fr” will be resolve thought the VPN. So if I do a “ping access” or “ping postfix”, it will change to “ping postfix.in.celogeek.fr” and then you will have the private ip address.
If that configuration work, you can remove the “SSH” service from shorewall, and only allow “OpenVPN” connection.
The big advantage, is I have one router, with all the dns configuration. The server obtain automatically their IP address, and any private server can just start and match the private network. They will have internet, so configuring a new server is very easy. And the IP can be fixed afterward.
Well, I hope it could help. Tell me if you have suggestion, I am very interested to improve that configuration.