this post was submitted on 22 Dec 2023
22 points (92.3% liked)

Selfhosted

40394 readers
375 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
ruud@lemmy.world
Loki@lemmy.world
CannaVet@lemmy.world
devve@lemmy.world
HybridSarcasm@lemmy.world
HybridSarcasm@lemmy.hybridsarcasm.xyz
22
Retain source IP when proxying through VPS (lemmy.procrastinati.org)
submitted 11 months ago* (last edited 11 months ago) by SeeJayEmm@lemmy.procrastinati.org to c/selfhosted@lemmy.world
29 comments fedilink hide all child comments
 

So, I'm experimenting with running a Mailu instance on my home server but proxying all of the relevant traffic through a WireGuard tunnel to my VPS. I'm currently using NGINX Proxy Manager streams to redirect the traffic and it all seems to be working.

The only problem is that, all connections appear to come from the VPS. It's really screwing with the spam filter. I'm trying to figure out if there's a way to retain the source IP while still tunneling the traffic.

The only idea I have, and I don't know if it's a bad one, is to us iptables to NAT the ports inbound on the VPS and on my home router (opnsense) route all outbound traffic from that IP back through the VPS instead of the default gateway. This way I shouldn't need to rewrite the destination port on the VPS side.

It sound a bit hacky tho, and I'm open to better suggestions.

Thanks

Edit: I think I need to clarify my post as there's some confusion in the comments. I would like the VPS to masquerade/nat for my mailu system accessible over a WG tunnel so that inbound traffic to the SMTP reports it's actual public IP instead of the IP of the VPS host that's currently proxying.

After giving that some thought I think the only way this could work would be if I treated the VPS as the upstream gateway for all traffic. My current setup is below:

[VPS] <-- wg --> [opnsense] <--eth-->[mailu]

I can source route all traffic from mailu to the VPS, via wg, but I don't know how to properly configure iptables to do the masquerading as I'd only want to masquerade that one IP. I'm not concerned about mailu not having internet access when wg is down, and frankly, I think I'd prefer it didn't.

Edit 2: I got the basic masquerading working. Can ping public IPs and traceroute verifies it's taking the correct path.

iptables -A FORWARD -i wg0 -s <mailu-ip> -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -s <mailu-ip> -j MASQUERADE

I think I got the port forwarding working.

iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 25 -j DNAT --to-destination <mailu-ip>
iptables -A FORWARD -p tcp -d <mailu-ip> --dport 25 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
  • tcpdump on the VPS eth0 shows traffic in.
  • tcpdump on the VPS wg0 shows the natted traffic.
  • tcpdump on mailu shows both inbound and outbound traffic.
  • tcpdump on opnsense shows 2 way traffic on the vlan interface mailu is on.
  • tcpdump on opnsense only shows inbound, but not outbound traffic on the wg interface.

I think the problem is now in opnsense but I'm trying to suss out why. If I initiate traffic on mailu (i.e. a ping or a web request) I see it traversing the opnsense wg interface, but I do not see any of the return SMTP traffic.

Edit 3:

I found the missing packets. They're going out the WAN interface on the router, I do not know why. Traffic I initiate from the mailu box gets routed through the WG tunnel as expected but replies to traffic sourced from the internet and routed over the WG tunnel, are going out the WAN.

The opnsense rule is pretty basic. Source: , Dest: any, gateway: wg.

Edit 4:

I ran out of patience trying to figure out what was going on in opnsense and configured a direct tunnel between the mailu vm and the VPS. That immediately solved my problems although it's not the solution I was striving for.

It was pointed out to me in the comments that my source routing rule likely wasn't configured properly. I'll need to revisit that later. If I was misconfiguring it I'd like to know that.

top 29 comments
sorted by: hot top controversial new old
[–] mholiv@lemmy.world 10 points 11 months ago (2 children)

You want to set the appropriate X-Forwarded-For or Forwarded headers in Nginx. The final application server being proxied (if well written) should be able to handle that.

Documentation can be found here. https://www.nginx.com/resources/wiki/start/topics/examples/forwarded/

Contrary to that other comment reverse proxies with actual IPs forwarded through them via the appropriate headers are normal and used commonly. Almost 100% so at scale.

Don’t let the wannabe elitists get you down. I personally would not host my production email server at home but self hosting is a learning journey. If you learn how email serves work along with reverse proxies you got it! That’s a win. Hack away.

[–] SeeJayEmm@lemmy.procrastinati.org 3 points 11 months ago

You nailed it on the head. This is a project for the experience and because I enjoy experimenting. If I can make this work to my satisfaction I may consider putting my primary domain behind it some day.

Thanks for the info and the support.

[–] Lichtblitz@discuss.tchncs.de 1 points 11 months ago

Exactly this. This procedure is so common that you need to take care in situations where you don't want the headers, as some tools set them per default.

[–] themoonisacheese@sh.itjust.works 6 points 11 months ago (2 children)

Not really. Your VPS's public IP is not yours to change, for obvious reasons, and it's unlikely that your hosting provider will let you send packets from your VPS using a source address that is incorrect. if they let you, then any replies to those packets will evidently get routed to the actual IP, ie your home IP. If you really want to forward SMTP to your VPS (which has less chance of being on a Blocklist by virtue of not being a residential IP), I suggest declaring your VPS as your SMTP sender in SPF, instead of declaring your home IP and trying to make that work with the VPS IP. The VPS can then be configured as an SMTP relay (this is a key feature of SMTP) to your home instance, or you could forward all traffic on the appropriate ports at the TCP level, but I don't advise doing this.

I hope you understand that if what you're asking was possible, I could rent a VPS, spoof your IP and receive traffic meant for your IP without any issues. For the same reasons, I think the other commenter mentioning x-forwarded-for headers is wrong if you're not using DKIM (and even then it's iffy). Otherwise I could just write a payload with mailto: whatever, from:you@yourdomain and x-forwarded-for: your home IP and pass SPF checks without having control over your IP.

if you're still confused about SMTP feel free to ask more questions

[–] ninjan@lemmy.mildgrim.com 2 points 11 months ago (1 children)

We have more stuff than SPF checks these days because they're wholely inadequate alone. DNSSEC, DKIM and DMARC are all important in their own right if you want a secure mail server.

That said I also don't agree with your example, because you assume a proxy for outgoing which I see no real need for. Generally speaking you proxy incoming traffic due to CGNAT making port exposure on a residential IP unfeasible. Further SPF checks will always use the actual IP source not what's in a X header.

[–] themoonisacheese@sh.itjust.works 1 points 11 months ago (1 children)

You need a proxy for outgoing to avoid your source server being on a residential adress, which all but guarantees all mailservers using spamhaus etc will block you by default. DKIM and DMARC are needed in their own right but an SPF fail will already make your mail fall into spam.

[–] ninjan@lemmy.mildgrim.com 1 points 11 months ago (1 children)

Sure, but if you proxy both in and outgoing then your SPF record should of course point to your VPS and thus again not be a problem.

[–] themoonisacheese@sh.itjust.works 1 points 11 months ago

Indeed, but in that case an off-the-shelf SMTP relay works fine.

[–] SeeJayEmm@lemmy.procrastinati.org 1 points 11 months ago (1 children)

I don't think I was clear in my post and I'm a little confused by your response. Rather than take the inbound traffic on the vps and proxy it over to the mailu server, I'd like to NAT (masquerade) that traffic so that source IP reports the actual source.

[–] themoonisacheese@sh.itjust.works 1 points 11 months ago (1 children)

So to be clear, you want traffic coming out of your VPS to have a source address that is your home IP?

let's go back to fundamentals and assume for a second that your VPS provider allows these packets out and your VPS initiates a TCP connection like that. It sends a TCP SYN with source: home address and dest: remote.

The packet gets routed to the remote. The remote accepts and responds SYN/ACK with source: remote and dest: home address.

Where do you think this packet will get routed? When it gets there, do you think the receiving server (and NAT gateways in between) will accept this random SYN/ACK that doesn't appear to have a corresponding outgoing packets sent first? If so, how?

[–] adam@doomscroll.n8e.dev 1 points 11 months ago

So to be clear, you want traffic coming out of your VPS to have a source address that is your home IP?

No that's not how I read it at all. He wants his VPS to act as a NAT router for email that routes traffic through a wireguard tunnel to the mail server on his home network. His mail server would act as if it was port forwarded using his home router, only it won't be his home IP, it'll be the VPS's

[–] MSgtRedFox@infosec.pub 1 points 11 months ago (1 children)

I don't know what open sense calls this, but you want a route map in Cisco world.

A route map overrides the routing table based on ACL.

Your local router thinks unless there is a moreb specific route, then use default out WAN. Route map says ACL if source is mailu, then next hop will be VPS over tunnel. I did this with Cisco DMVPN between Germany and states.

Sorry I don't know terms of open sense, but the concept is the same.

[–] SeeJayEmm@lemmy.procrastinati.org 0 points 11 months ago (1 children)

I believe the policy based routing is the same thing. I'm starting to think I'm encountering an opnsense bug.

[–] MSgtRedFox@infosec.pub 1 points 11 months ago (1 children)

When you did your dump of the tunneled traffic, was the VPS preserving the public IPs?

I was only assuming since you mentioned seeing the traffic at each segment.

[–] SeeJayEmm@lemmy.procrastinati.org 2 points 11 months ago (1 children)

Once I got masquerading configured it was preserving the public IP. I tcpdumped every interface in the path and watched the traffic. When it hit opnsense instead of respecting the policy based routing it was routing the traffic out the WAN.

What baffles me is if I initiated traffic from the mailu server (ping, wget, etc...) I could see that opnsense was routing all traffic in that conversation out the WG interface, none of it hitting the way.

I need to update the post because after fighting with it all day, I realized I was being stubborn (I have a need to solve the problem). I configured a direct WG tunnel between the VPS and the mailu VM and routed the traffic that way. It's all working exactly as I need it to now.

I'd still like to know if opn has a bug or if I was missing some setting as I'd rather not be littering my network with tunnels when I shouldn't need to and I can leverage some smarts in opn (i.e. if the tunnel is down, the gateway would get marked down in opn and it would ignore the policy route).

[–] MSgtRedFox@infosec.pub 2 points 11 months ago

Yeah, the tunnel would annoy me unless it was open and I could scan the traffic as it went through my proxy. My OCD kicks in when stuff doesn't do what it's supposed to

You could always swap out that OPNS for Cisco, NBD. /Sarcasm

[–] 2xsaiko@discuss.tchncs.de 1 points 11 months ago (1 children)

Since you mention nginx, I assume you're talking about proxying HTTP and not SMTP/IMAP... For that, you have the X-Forwarded-For header which is exactly for that, retaining the real source IP through a reverse proxy.

You should be able to add proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; to your location block.

Alternatively, looks like there's a Forwarded header (RFC from 2014) which I've never seen before but it seems cool: https://www.nginx.com/resources/wiki/start/topics/examples/forwarded/

I guess it comes down to what mailu supports, I have never used that.

If you are talking about SMTP and IMAP, I don't think there's a standard way to do this. You'd have to set up port forwarding on the VPS for the SMTP ports and IMAP port, and set up your home server to accept connections from any IP over the wireguard interface.

That's exceedingly horrible though and there's a better option for SMTP at least: set up an MTA (e.g. Postfix) on the VPS and have it forward mail to the real destination server. And for outgoing mail it never has to touch your home server (except your client copying it into the Sent inbox over IMAP), just send it out over the VPS directly. Or if you're using some builtin web client, I guess do set the MTA on your local server to send mail to the VPS's MTA.

[–] SeeJayEmm@lemmy.procrastinati.org 1 points 11 months ago

Since you mention nginx, I assume you’re talking about proxying HTTP and not SMTP/IMAP… For that, you have the X-Forwarded-For header which is exactly for that, retaining the real source IP through a reverse proxy.

I was using NGINX streams feature to proxy the various mail components (smtp, imap, etc...) but that was setting the source IP to the VPS.

I was told in another comment that Mailu can handle being proxied behind traffik. I'm not sure if NGINX has similar support for the "PROXY" protocol. I need to dig into that.

[–] Decronym@lemmy.decronym.xyz 1 points 11 months ago* (last edited 11 months ago)

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer Letters More Letters
CGNAT Carrier-Grade NAT
HTTP Hypertext Transfer Protocol, the Web
IMAP Internet Message Access Protocol for email
IP Internet Protocol
NAT Network Address Translation
SMTP Simple Mail Transfer Protocol
TCP Transmission Control Protocol, most often over IP
VPS Virtual Private Server (opposed to shared hosting)
nginx Popular HTTP server

9 acronyms in this thread; the most compressed thread commented on today has 15 acronyms.

[Thread #369 for this sub, first seen 23rd Dec 2023, 00:55] [FAQ] [Full list] [Contact] [Source code]

[–] restlessyet@discuss.tchncs.de 1 points 11 months ago* (last edited 11 months ago) (1 children)

I guess your OPNSense rule from Edit3 is not working because the source is not your mailu instance, because connections are initiated from the outside and mailu only answers (TCP ACK). So you have asynchornous routing.

You may get this working if you set the "reply-to" option to the wg gateway on the firewall rule that allows VPS -> wg -> mailu traffic.

However there is a much cleaner solution using the PROXY protocol, which mailu seems to support: https://mailu.io/master/reverse.html

They are using traefik, but nginx also supports the PROXY protocol.

[–] SeeJayEmm@lemmy.procrastinati.org 1 points 11 months ago

I guess your OPNSense rule from Edit3 is not working because the source is not your mailu instance, because connections are initiated from the outside and mailu only answers (TCP ACK). So you have asynchronous routing.

Ohhhh. Well this is just my ignorance then. I was attacking it like the rules applied to the packet not the conversation. Thanks! I really got lost down the rabbit hole of "why is my routing not working the way I want" and lost sight of the problem I was originally try to solve.

[–] Atemu@lemmy.ml -2 points 11 months ago (2 children)

No. That's quite literally the point of a proxy. If you don't want to be proxied, don't use a proxy.

I know this is the selfhosted community but if you're new to this, you really shouldn't be hosting email as it's one of the hardest services to get "right".
(Ideally no other public service either, they're a huge liability. Start hosting stuff for local network use.)

[–] mholiv@lemmy.world 2 points 11 months ago (1 children)

No need to be toxic here. You don’t need put people down. We’re all learning here together. Hey. We all are all learning more about how reverse proxies and forwarded headers work together right now, including you.

We should aim to be an open welcoming community.

[–] Atemu@lemmy.ml 1 points 11 months ago (1 children)

I did not intend to be toxic.

[–] mholiv@lemmy.world 1 points 11 months ago

I understand. But do you see what you wrote could be seen as toxic? Intent is nice, but what and how you write really determines the tone of a community.

[–] SeeJayEmm@lemmy.procrastinati.org 2 points 11 months ago (1 children)

I know this is the selfhosted community but if you’re new to this, you really shouldn’t be hosting email as it’s one of the hardest services to get “right”.

I've been self-hosting since I was on dial up but thanks for the assumption. I'm quite familiar with the nuances of hosting mail successfully and all my boxes are ticked. The monkey wrench was just that I didn't account for the source addressing and it screwing up my spam filter as I've never tried this specific scenario before.

No. That’s quite literally the point of a proxy. If you don’t want to be proxied, don’t use a proxy.

That was the thrust of my post. I just started with NPM because it was already there. I've been experimenting and researching since I posted this and I think my solution is to masquerade (NAT) the mailu host behind the VPS and explicitly forwarding the necessary ports. Unfortunately, iptables is one of my weak spots and the nuances of making iptables work in this situation is eluding me. That's really where I could use some guidance.

[–] Atemu@lemmy.ml 0 points 11 months ago (1 children)

I completely misread your post. Your issue isn't outbound connections appearing as if they came from your VPS, it's inbound connections to your local mailserver being proxied.

[–] SeeJayEmm@lemmy.procrastinati.org 2 points 11 months ago

Yeah. The spam filter see everything as being from the VPS ip, so spf and dns checks are all failing on inbound mail.

[–] tagginator@utter.online -3 points 11 months ago

New Lemmy Post: Retain source IP when proxying through VPS (https://lemmy.world/post/9879470)
Tagging: #SelfHosted

(Replying in the OP of this thread (NOT THIS BOT!) will appear as a comment in the lemmy discussion.)

I am a FOSS bot. Check my README: https://github.com/db0/lemmy-tagginator/blob/main/README.md