How to get data from nginx to Androind app through Wix

[jeffd1]

So I have a Raspberry Pi that collects data and sends it to a flask server. I want to get data over to android app that I built. I was able to do this on the local network through flask by sending a get request from the android app to the flask server.

I then followed this tutorial from digital ocean and set up a nginx server that would be publicly available. When it came time to enter a domain, I entered my personal domain. The problem is that that domain currently points to a Wix website.

Calling get requests to my domain simply loads the website. I’ve experimented with backend code on Wix and I tried fetch(request) but that returns undefined. Perhaps that’s an issue on the nginx side, but I don’t think so. The logs look ok. I’ve looked into all of the following and I don’t know which avenue could yield success. So I don’t know where to spend my time learning.

Rabbit holes I’ve gone down:

1. backend javascript in the website (e.g. fetch())
2. setting up an api
3. routing in Wix
4. creating a subdomain
5. using a different location on my subdomain (wix loads a page that says no page found or something similar)
6. setting up a proxy server
7. setting up a reverse proxy server
8. moving my domain away from Wix entirely.

Thank you for your time and any help towards ending this confusion would be greatly appreciated!

Edit: I should add that I did set up portforwarding on my router.

[madpenguin]

Hi Jeff, welcome to the forums!

Ok, so to start with, your requirement is fairly typical and there are multiple easy ways to do what you want to do. To get there however I need to add a little background.

There are essentially three components to what you want to do.

• Hosting the application,
• Connecting the application to the Internet
• Finding the application on the Internet

Starting with the last part as that’s probably the easiest. Typically a domain name is used to cover naming for your organisation, rather than just one resource, like a website. So typically people would use www.domainname for their website, where www is the resource in question, then they might have mail.domainname, wiki.domainname etc etc.

Now it has become common practice to drop the “www” for websites, however physically the naming mechanism hasn’t changed. Although there are different ways of setting things up, the “norm” in this case would be to create an “A” record with your DNS provider for something like “myapp.domainname”, which points to the address of whatever you are going to expose to the Internet. Then instead of doing a “get” against “domainname”, do a GET against “myapp.domainname”.

So in terms of hosting and connectivity, I’m working on the basis that your app is hosted on an RPi, and that Flask is also running on the RPi. In this case I would look to expose the RPi to the Internet via a tunnel, then point “myapp.domainname” at the public end of the tunnel.

Sounds hard / complicated, but it’s really not so long as you have the right tools and are prepared to use them as intended

So, I am actually doing something very similar to you, albeit on maybe a larger scale. I’m running a bunch of things “here”, exposing them via tunnels and pointing names at them. If you’d like to try an example, take a look at;

https://zerodocs.madpenguin.uk

This is a Python/Javascript application that’s running inside a “container” on a Raspberry Pi4 and as it happens, managed by “systemd” as per your DO article

So assuming you already have the Raspberry Pi / Flask set up and running, it’s the tunnel and domain name to play with.

Tunnels & Domains

For both of these I use CloudFlare. They run a “free tier” so the tunnel function isn’t something you need to pay for. In terms of tunnel management and domains, it’s relatively automated, but in order to do this, CloudFlare need to be able to manage the domain. So the first thing I’d recommend it to use CloudFlare as your registry, then just point the domain name at your WIX website for continuety.

Once you’ve done this, on the CloudFlare dashboard (once you have a free account) take a look at the “Zero Trust” menu option, then within this “Tunnels”, then “Create a tunnel”. Select the default connector and hit “next”. Give it a name then click “save”.

Now you need to set up the RPi end. Click on the button for your distribution (on the CloudFlare dash - I use debian) and then for your architecture, which will be arm64 for a RPi, then click on the two overlapping squares icon (which does a clipboard copy). Now go to a SSH or console session on your RPi and paste in what you just copied. That should install the tunnel and connect it up.

If you go back to the CloudFlare page and click “Next”, it will prompt you for a subdomain, so you will want whatever name (like “myapp”) you want to use. Then select your domain from the drop-down. Now you need to tell it what to do when it gets to your end of the tunnel, so select HTTP from the drop-down (I’m guessing you’ve not set up SSL?) and then “localhost:port” for the URL, where “port” is the port you have flask running on, so if it’s on port 80, then “localhost:80”.

Cloudflare should then automatically set up “myapp.domainname”, install an SSL certificate for you (on it’s end) and make “https://myapp.domainname” live in such a way that requests are serviced by your Flask application …

Again, once you start playing with it you realise CloudFlare are doing 99% of the work for you and all you really need to know is what you want to do and how to navigate their dashboards …

[madpenguin]

Note: (re your edit)

Unless you have a business broadband connection and are paying for a static IP address, OR you are using DDNS to dynamically map your broadband IP address to a name, port forwarding is going to be a time limited proposition. (by design, all broadband connections come by default with a dynamic IP address that will change over time) So if you point your DNS at your broadband router, it will only work until your router is issued a new address. Could be a day, week or month (or longer), depending on your provider.

Also worth noting; CloudFlare advertise that domain registration is provided “at cost”, so whoever you currently have it with, CF will be at least as cheap. I think I’m paying under £4 per year for .uk domains at present.

[jeffd1]

Wow THANK YOU! I am overwhelmed with relief that I have a clear path now. That was so nice of you to take the time to spell things out for me and I think it’s exactly what I needed.

Would you extend the kindness further and ELI5 this for me:

“So the first thing I’d recommend it to use CloudFlare as your registry, then just point the domain name at your WIX website”

I’m taking this to mean that I transfer my domain away from Wix, then go to CloudFlare and register my domain with them. then within cloudfare i point the domain to wix? Is that about right?

[madpenguin]

Erm, so I’ve used WIX a little in the past but not done anything with the domain name management. Hopefully they don’t insist the domain has to be hosted with them, in which case yes, start with;

• Set up an account with CloudFlare
• Transfer domain to CF
• Make sure you have an “A” record in place pointing back at your WIX site
  (at which point you should be all square, but CF should be able to manage the name with regards to tunnels and the like)

Typically when you transfer a domain between registrars they will bill you for a year’s worth of hosting (£4 ish in this case) but they should extend the renewal date to include the amount of time remaining on your domain renewal. So if you have six months until your domain renewal at present, when you transfer you should pay an annual fee, but the first renewal should be in 18 months. (unless something has changed)

Caveat; £4 is ~ what it costs for a standard .uk domain, other registries can charge more, but it should always be less than WIX were charging.

You can check this with “whois” for example;

# whois linux.uk

    Domain name:
        linux.uk

    Data validation:
        Nominet was able to match the registrants name

    Registrar:
        Cloudflare, Inc. [Tag = CLOUDFLARE]
        URL: https://cloudflare.com

    Relevant dates:
        Registered on: 17-Feb-2015
        Expiry date:  17-Feb-2025
        Last updated:  18-Jan-2024

… it also gives you access to lots of other free stuff CF provide , just be aware they also provide chargeable services too. They’re pretty good at labelling what comes with a charge, just be aware that because you can see it, doesn’t mean it’s necessarily a free feature

Note: if you take a screen grab of your WIX domain name records before you transfer, once it’s transferred, if it doesn’t move the records over for you (I can’t remember whether it does or not) you can just duplicate the screen grab on the CF domain dashboard.

[jeffd1]

I tried to post my whois but there are too many url links and this website blocks it as I am a new user. It does say it is registered with wix and domain status is: “clientTransferProhibited”. So that doesn’t sound good. But I’ll go over to CF first and see what happens from there.

[madpenguin]

Ok, so the “clientTransferProhibited” is a protection to stop someone accidentally (or nefariously) moving your domain elsewhere. It should just be a case of finding the “unlock” option on the control panel, but you will need to do this probably before starting a transfer. (it won’t transfer until you unlock it on the WIX panel)

[jeffd1]

gotcha, you are awesome! thank you

[jeffd1]

ok so i’m still working through this. I was finally able to get my domain transferred over to CloudFlare. I had to first transfer it over to Gandi, change the name servers on Gandi to CloudFlare, then I could transfer to Cloudflare. Wix doesn’t let you change the name servers.

After working through your detailed instructions…it didn’t take me long to get stuck. I set up the tunnel, and went to my domain and I get

Blockquote Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

I think it might have to do with my configuration file as I don’t really know how to handle the subdomain.

server {
listen 80;
server_name sumppump.jeffs-handyman.net www.jeffs-handyman.net;
location /spdata/ {
include uwsgi_params;
uwsgi_pass unix:/home/sp_server_beta/sp_monitor_beta4/sumPumpApp.sock;
}
}

The nginx journals don’t show any recent entries.
Any help is greatly appreciated!

[madpenguin]

Ok, so … sorry if I mentioned nginx, I use nginx on the end of my tunnel when I have a Wordpress instance sat on the end, because Wordpress is PHP and doesn’t like listening on sockets.

However if I have a Python application for an API, I don’t actually use NGINX, I just terminate on the application itself. i.e. run the python / flask and make it listen on port 80.

Before we go into NGINX configs, which aren’t necessarily trivial, is there any specific reason you want NGINX in the loop?

For an API, you typically don’t want to mess around with any of this, all you want is the request coming on port 80 (or the port of your choice) and being directly handled by Flask. (for testing, could you just kill nginx and run your application on port 80 and see what happens?)

… that aside, without checking my nginx docs, my expectation is that the uwsgi_pass should be within a “location” block, so for Wordpress for example;

location ~ \.php$ {
  fastcgi_split_path_info ^(.+\.php)(/.+)$;
  fastcgi_pass unix:/var/run/php/php-fpm.sock;
  fastcgi_index index.php;
  fastcgi_param SCRIPT_FILENAME $request_filename;
  include fastcgi_params;
}

But the response you are seeing seems to imply it’s delivering from the default filesystem location … maybe an issue with your “location” spec?

[jeffd1]

No I’m definitely not attached to nginx. I disabled it and ran flask on port 80, but now I get Error 1033 Argo Tunnel Error in my browser when I go to my subdomain. I ran “cloudflared tunnel list” and it returned my tunnel. The tunnel also says “healthy” on cloudflare’s website.

[madpenguin]

Ok, so worth double checking. Make sure the tunnel is configured for “localhost:80”, and make sure you can successfully hit localhost:80 from your local machine with curl or wget.

The tunnel being healthy means its up, the problem is that the circuit has a break somewhere.

[madpenguin]

The route will be browser → cloudflare → tunnel → configured port → your application.

Its failing on one of the last 2 parts.

[jeffd1]

Yep…I found it! I had the subdomain listed in the dns records. When I went to add the subdomain to the tunnel, it told me that it already existed. I took this to mean that I was good to go. I wasn’t.
I deleted the subdomain name from dns records, then added it to my tunnel and I was SOO HAPPY to see it work.

To do this I am running my flask server on route and outside an environment (only way to run Flask on port 80 so it seems). I’ve read that this is extremely risky and asking for trouble. However, is it all safe because cloudflare is handling all the security with the tunnel?

[madpenguin]

Ok, so … running as root, not ideal. There is a way to allow user processes to access privileged ports (those below 1024) , I cant remember off hand the incantation, will look it up when im back on a pc.

The easier way is just to run it on a higher port (8080 for example) , then change the tunnel endpoint from localhost:80 to localhost;8080 using the cloudflare tunnel config.

Running as root, to be realistic, anyone serious getting into your account is probably good enough to find a privilege escalation hole in your distro to acquire root … that said running as a user is an additional level of protection against casual hackers and script kiddies.

The tunnel itself if just a network filter. It doesn’t protect against bugs in your code or backdoors in the libraries you may be using. It would be possible for a python library (for example) to add a few flask paths to implement an api to allow a remote user to run a shell session inside your software via remote http api calls …

Zerodocs for example , in terms of isolation, runs as a user process on 8080, inside a container, which runs inside a virtual machine, which is sitting on a dedicated server, sitting on a DMZ network segment. (he says quickly hiding his tin foil hat … )

[madpenguin]

Ok, looked it up, the relevant magic is called “capabilities”. So to set a program so when it runs as a user it can use ports below 1024 you would do;

sudo setcap 'cap_net_bind_service=ep' /path/to/your/program

Unfortunately in this instance, your program would be the Python executable … it looks like there’s another newer option I’ve not come across before;

sysctl -w net.ipv4.ip_unprivileged_port_start=80

Which should allow any process to listen on ports 80 and above

[jeffd1]

You are a life saver! So now I am running as a user in a virtual environment on port 8080.

Should I look into adding a container? What does that mean?

[madpenguin]

So containers are a form of virtualisation, i.e. a mechanism for partitioning resources (and security) within a system. In the context of what you’re doing, it can be handy because each container will behave as if it were a separate machine in that it will have it’s own IP address and it will be able to run it’s own tunnel daemon.

So you could have one machine, then one or more containers within the machine, each with it’s own filesystem, process space, ip address, tunnel etc. So each container instance could run a unique application that would be partitioned off from each other, and from the host system.

On a raspberry Pi I run lxc. Although containers are a featue of the Operating System, the way containers are managed and presented to the user can vary. One option is docker, but for this particular type of functionality I use something called incus.

Incus

This used to be the LXD project which was sponsored bu Ubuntu, but Incus is the new Open Source fork by the original author who has moved away from Ubnuntu. My RPi4 dashboard looks like this (still running LXD on this particular box);

This is what it looks like on the Debian based workstation I’m currently typing on (which is an RPi5);

Whereas it’s relatively each to find a user escalation exploit to “get root” as a user, it’s way more difficult to find a way to escape from a container and access the host system.

[jeffd1]

Gotcha! Thank you, I may take that on soon. Thank You!!