question

5teve avatar image
5teve asked

[Feature Request] - Remote SSH ability

Hi Guys

For those of us running Venus OS devices on board boats and in remote locations that are not manned, it strikes me that a key missing part of the maintenance of these devices is SSH remotely. For those of us not blessed with a clever brain, I cant seem to find a way of accessing my PI via an LTE (4g) connection that resides on the boat due to CGnat. The LTE router that I have only does OPENVPN server, not client (which is useless behind CGnat) so unless i'm on the boat with the laptop I have no access.

Now the like of Freenas have a built in 'terminal' or Shell.. It strikes me that would be a mightily useful thing to include in either the remote console (via VRM) or via VRM itself? It would certainly make life a little easier for doing the things that require SSH..

I know I could just get a router that supports a VPN client, hook it up to my home VPN server and access that way.. but it just seems like a nice touch having remote SSH...

I'm sure being a heathen.. I am ignoring security concerns and a million other things.. but I live in a simplistic bubble.. my brain hurts less that way ;O)

Steve

Venus OSremote console
3 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Kevin Windrem avatar image Kevin Windrem commented ·

I just released a beta version of a wrapper to install and setup tailscale on GX devices:

https://github.com/kwindrem/TailscaleGX

The ReadMe.md hopefully provides all that you need to know. Including how to add the package to Package manager so you can install it from the main GUI (gui-v1).

Please have a look but ONLY if you have local ssh access to uninstall the package should something go wrong.


Please post comments and bug reports to the issues section of the GitHub repo.

@mvader (Victron Energy) I would appreciate your feedback too.

2 Likes 2 ·
Kevin Windrem avatar image Kevin Windrem commented ·

Following on the work by @mvader (Victron Energy) I have released TailscaleGX, a wrapper for tailscale (described in another answer here).

TailscaleGX includes a menu for controlling tailscale and provides the necessary status in order to add he GX device to your tailscale account and establish a connection.

https://github.com/kwindrem/TailscaleGX

TailscaleGX is a SetupHelper package and can be added from Inactive packages / new:

Package name: TailscaleGX

GitHub user: kwindrem

GitHub branch or tag: latest

You also need to set up a tailscale account at tailscale.com.

The ReadMe will hopefully provide enough information to set up a connection.

tailscale provides ssh and http access, and probably many more I haven't tested.

Please post comments, questions, bugs to the issues section of TailscaleGX repo.

Thanks, enjoy

2 Likes 2 ·
dutchsolarfreak avatar image dutchsolarfreak Kevin Windrem commented ·
Working VERY well. Thanks!!
0 Likes 0 ·
11 Answers
mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hey @XZv I looked into that a bit further, and tailscale looks like the best solution.


Here is why:

to make all of it work well, also when installed behind various types of firewalls, then you need some cloud hosted service / server. Where all devices connect to and where the user can connect to as well.

And security is critical.

To host & offer such service for free, support it, maintain it, have it pentested and so forth by ourselves is a step to far.

The other route, totally open source / developer would be a rc.Local script that enables Wireguard on GX device + a (dockerised?) server that is the pivot. Nice but nerdy and complex.

Enter tailscale. All of it done and they offer a free service.


So, hereby the invitation to you and the rest of Community: can someone work on a script to get tailscale up and running on a GX device + instructions?

And in the interim do it with a one time self compiled tailscale binary - use our SDK to do that. Should be pretty simple.

If that works well, one of our guys can take care of having the binary included as a standard plus having it as enableable daemon which is watched over by Daemontools like all other services on the GX.

And the configuration/provisioning will need to be solved. If all that is needed is some long key, then perhaps gui-v2 allows cut & paste so solves that problem as well.


Note to myself for when including this into the normal Venus OS builds: https://github.com/ChristophHandschuh/meta-tailscale/


And wrt the other solutions:

- wireguard is a protocol; with implementations; tailscale is built in top of that

- zerotier is closed source solution (so I prefer wireguard, but in the end its jusr a choice

- setting up port forwarding, or having to setup and maintain anything at all, is too complex for most of intended user group.

- ngrok is an alternative solution; tailscale seems to be the more popular one after reading around a bit.

8 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

xzv avatar image xzv commented ·

Thanks @mvader (Victron Energy) for the investigation and your thoughts. Today I spent half a day on comparing different solutions and approaches as well. I totally agree, Tailscale looks like the most promising solution right now when the priorities are security, ease of commissioning and reliability.

I expect to have a large fleet of devices that I can access for maintenance remotely once in a while via SSH, so I would highly value a package that will always ship with Venus OS and just needs some initial "machine key" to join your Tailscale network.

I will do some initial tests and report back. Would be good if you or your team could comment on the feasibility of integration into Venus and the GUI then.

0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ xzv commented ·

Ok I couldn't help myself, and had a play :)


Here is more or less what I did:

1. Go to tailscale.com and create an account; and installed it somewhere. Either on my computer or iphone or both.

2 Logged on on a GX device, downloaded the latest linux binary for arm https://pkgs.tailscale.com/stable/tailscale_1.60.0_arm.tgz

3. Un tarred the two files and put them in /data/tailscale/

4. Created a run-tailscale file, /data/tailscale/run-tailscale, with these contents:

#!/bin/sh

# Start tailscale daemon and use a dir on data partition
# to keep state (certificates and such). tailscaled will auto-
# create that dir if it doesn't exist.
#
# Logs are written to /var/lib/tailscale, which -on Venus OS-
# is volatile.
#
# To see further commandline options for the daemon, run
#
#     tailscaled -?
#
# I didn't find any docs for its options.

/data/tailscale/tailscaled -no-logs-no-support -statedir /data/tailscale/stat


4. Added the /data/rc.local file, that starts above, with these contents:

#!/bin/sh

/data/tailscale/run-tailscale


5. made both files executable

6. rebooted the GX, and checked that it runs:

root@einstein:~# uptime
 21:32:07 up 1 min,  load average: 2.82, 1.43, 0.54
root@einstein:~# ps | grep tailscale
 1187 root      2952 S    {run-tailscale} /bin/sh /data/tailscale/run-tailscal
 1189 root      541m S    /data/tailscale/tailscaled -no-logs-no-support -stat
 1933 root      2688 S    grep tailscale


7. started up tailscale for the first time, triggers this autoconf:

1708119257664.png


8. copied and pasted that url into a browser, logged in. It then simply says success and goes back to commandline. Quite nice how they did that! Logging in on that URL auto-provisions that device with proper keys and everything.

9. rebooted the GX again, to see if and how it initialised:

root@einstein:~# /data/tailscale/tailscale status
100.89.189.104  einstein-2           matthijsvader@ linux   -
100.111.71.54   mva-ubuntu2204       matthijsvader@ linux   -

# Health check:
#     - dns-os: writing to "/etc/resolv.pre-tailscale-backup.conf" in rename of "/etc/resolv.conf": open /etc/resolv.pre-tailscale-backup.conf: read-only file system
#     - router: setting up nat/ts-postrouting: running [/usr/sbin/iptables -t nat -N ts-postrouting --wait]: exit status 3: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

root@einstein:~# 


10: so that worked nicely, that GX is now connected to my tailscale network, and the connection will survive firmware updates and reboots; until ofcourse something changes that makes the binary incompatible with Venus OS.


Next steps would be:

1. add including tailscale into Venus OS builds, and look into tailscale lite: https://tailscale.com/kb/1207/small-tailscale

2. Catch that login url, and show it in the gui somewhere, extra nice is as a QR code, so users can open it to login that device easily.

3. Document the whole thing.

2 Likes 2 ·
1708119257664.png (12.3 KiB)
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ mvader (Victron Energy) ♦♦ commented ·
ps, after above process, I could connect to the console on the GX from (for example) my iphone super easily:

1. install the tailscale app, login.

2. once thats done, you'll see a list of online devices, which includes the GX.

3. install a ssh app, for example termius, and login to the ip address of the GX device, and voila.


1 Like 1 ·
Kevin Windrem avatar image Kevin Windrem mvader (Victron Energy) ♦♦ commented ·

The archive you linked apparently uses systemd to start tailscaled. Is there a reason you didn't start this as a daemontools service?

Thinking I might want to roll this into SetupHelper until Venus OS includes it.

Would the same arm binary run on all platforms (including RPI)?

0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ Kevin Windrem commented ·

Hi @Kevin Windrem , I only took the binaries from the archive. There is no systemd on venus os.

Its not a daemontools service because I didn’t get to that.

Yes, will work on pi’s as well I’d think!

Nice!

0 Likes 0 ·
Kevin Windrem avatar image Kevin Windrem mvader (Victron Energy) ♦♦ commented ·

Carrying on with @mvader (Victron Energy)'s work:

The linked archive above does run on RPI also. Yea!

Executable size is unfortunately large:

The size of the two executables in the archive link above is about 47 MB.

I built one with the --extra-small flag and it came in at 22 MB with a combined executable for both commands as suggested. (.tar.gz is 8.5 MB).

To build the small tailscale executable:

install 'go' environment per their instructions

clone the tailscale archive

GOOS=linux; export GOOS
GOARCH=arm; export GOARCH
cd <tailscale clone>
./build_dist.sh --extra-small -o tailscale.combined -tags ts_include_cli ./cmd/tailscaled
ln -s tailscale.combined tailscale
ln -s tailscale.combined tailscaled

I then used the commands mvader provided to start tailsale on the GX device.

The login URL appears stable across a reboot but haven't tried a Venus OS firmware update yet. I think mvader did that already though. I did delete the GX from the tailscale console and had to do

./tailscale login

and at that point got a new login URL

Note that a tailscale up or tailscale login will wait for the connection from the tailscale console to complete!!!

So to establish the connection, the GX device needs to set tailscale up or tailscale login and report the login URL on the GUI. The GX device then needs to be connected to the tailscale console from another host.

Next I'll work on a daemontools service to manage all this and put tailscale status and info on dbus and a GUI page.

1 Like 1 ·
Kevin Windrem avatar image Kevin Windrem mvader (Victron Energy) ♦♦ commented ·
Thanks for the info. I'll give it a go.
0 Likes 0 ·
xzv avatar image xzv mvader (Victron Energy) ♦♦ commented ·
Great work! Thanks for sharing! Appreciate it!
0 Likes 0 ·
mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hi, one available solution is zero tier:

https://community.victronenergy.com/questions/81273/remote-access-to-venusos-zerotier-one-installation.html

I will need some self compiling and such. It shouldn't take you more than a few hours of work.


And, for sure it would be great to have a solution natively in Venus OS; but its not something we'll be focussing on anytime soon I'm afraid.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Warwick Bruce Chapman avatar image
Warwick Bruce Chapman answered ·

Put a Mikrotik router in-front of the GX-device and use one of its tunnelling features. I setup a Wireguard server on a Hetzner cloud container that cost a couple Euro and tunnel from the Mikrotik to that server. I can then port forward the Mikrotik to the GX-device and SSH to the GX-device from the cloud container.

2 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Warwick Bruce Chapman avatar image Warwick Bruce Chapman commented ·

Actually, something like this looks nicer using Zerotier and a Raspberry Pi: https://iamkelv.in/blog/2017/06/zerotier.html

0 Likes 0 ·
nebulight avatar image nebulight commented ·
Mikrotik now natively supports ZeroTier right in their router OS 7. It's so easy to get up an running, it's amazing that it's free (for personal use). Zero firewall rules needed.
0 Likes 0 ·
Kevin Windrem avatar image
Kevin Windrem answered ·

A huge security hole, but you can forward port 22 to your Venus device to allow SSH from the outside world.

The VPN solution is much more secure and the one I'd recommend.

2 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

gone-sailing avatar image gone-sailing commented ·
The issue here is CGNAT, your router’s IP address is not addressable from the internet behind CGNAT. VPN is the only solution. Unfortunately I have to deal with CGNAT on the boat and at home.
0 Likes 0 ·
mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

This looks like a really promising solution to me /

https://community.victronenergy.com/questions/205323/venus-os-ngrok.html

3 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

iv4n avatar image iv4n commented ·

Pretty good indeed! I still hope that one day Victron adds the package wireguard-tools so we can setup wireguard on the gx device.

0 Likes 0 ·
iv4n avatar image iv4n iv4n commented ·
I was able to compile it from source! I will post these as an reply on the main thread.
0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ iv4n commented ·

- Per Venus OS v3.10 (or something) wireguard kernel module is included.

- Per Venus OS v3.20, wireguard tools are standard in our online repository of installable packages, and installable with opkg


And for anyone looking for the most simple solution, see Tailscale. I’m hoping for some volunteers to help document and do the leg work for that; perhaps start with simple venus-data-tailscale.tgz package.


See also further down below.

0 Likes 0 ·
pau1phi11ips avatar image
pau1phi11ips answered ·

Another option: I have a Teltonika 4G router in front of the Cerbo. You can tunnel SSH through their remote management (RMS) portal very easy.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

iv4n avatar image
iv4n answered ·

This is another solution, using Wireguard as a VPN.

I just put together this quick guide:

https://community.victronenergy.com/articles/211164/howto-venus-os-setting-up-wireguard.html

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

xzv avatar image
xzv answered ·

@mvader (Victron Energy) Any chance to put a Victron SSH port forwarding onto the midterm roadmap? Like the one you already have for your R&D for customers and can be enabled with the "Remote support" switch? That would be a great benefit and service for many. Thanks!

2 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Hey @XZv , the best solution for remote ssh access that I’ve seen is Tailscale.


See my posts above for how it works / how to get it working; and Kevin has prepared an installer for it.

Perhaps someday we’ll make it more natively installable/configurable.


I recommend to try out what there already is - it works great!


0 Likes 0 ·
dutchsolarfreak avatar image dutchsolarfreak mvader (Victron Energy) ♦♦ commented ·

Search for the tailscaleGX package made by kwindrem.

Works like a charm and low CPU load on the CerboGX

0 Likes 0 ·
Kees Oomen avatar image
Kees Oomen answered ·

Wow, keep on the good work to support Tailscale as embedded in VenusOS soon! Will definitely following this topic to see the progress.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Randy Goodman avatar image
Randy Goodman answered ·

to me this is a BIG feature for customers. I would use this all the time in my development and deploy\troubleshooting. When can we have it?

i am not a net guru and dont follow all being said above and on the threads linked to, so, the idea of a customer just needing to run a script is perfect! make sure you include windows as one of the customer platforms. THX!

1 comment
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Kevin Windrem avatar image Kevin Windrem commented ·
See my post from earlier today. TailscaleGX is a SetupHelper package that adds access through tailscale as discussed a while back and tested by mvader.


It's currently in beta (along with a beta of SetupHelper). I'll be releasing both after some test time

0 Likes 0 ·
iv4n avatar image
iv4n answered ·

@mvader (Victron Energy) Just came in to check this post and went to review the script I contributed for installing wireguard in Venus OS and it no longer exists. It says to contact the administrator if I believe it's a mistake: https://community.victronenergy.com/articles/211164/howto-venus-os-setting-up-wireguard.html


Would you know why was that post deleted? I find it sad that it would be deleted without any notice and also quite discouraging to contribute further things after putting effort and time putting the script and the post together for the community.


2 comments
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Hi @Iv4n , as far as I can see it is not deleted:


0 Likes 0 ·
img-1658.png (254.8 KiB)
iv4n avatar image iv4n mvader (Victron Energy) ♦♦ commented ·
I ended up sending a message to the community manager, seems like it got restored!
0 Likes 0 ·