question

rns avatar image
rns asked

venus OS : Devices always get disconnected

Since I added a second bms and second smartsolar controller my rapsberry pi3 started crashing (at least no network connection at all)

I bought a new raspberry pi4 4gb and now my devices get disconnected all the time.


Anyone having the same issues ?

How to solve it ?


My devices are connected via the DIY rs485 to usb connector cables but it is strange that they work when I reboot but get disconnected sometimes after a few minutes sometimes after some hours.



Connected devices ;

2 smartsolars

1 multiplus II

1673894004152.png



If I reboot the PI they are all live again at least for some time (from minutes to hours)

Sometimes they are all disconnected.

Venus OS
1673894004152.png (17.0 KiB)
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

8 Answers
peregrines avatar image
peregrines answered ·

Having the same issue with my 3 JK BMS Lifepo4 Batterys. They drop out several times a day. Also using Rpi4.

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

oofta avatar image oofta commented ·

What kind of USB serial cables are you using - generic/DIY or Victron? I noticed this other thread that suggests using ones with isolators is better:

https://community.victronenergy.com/questions/118932/rpi-3-venus-os-not-seeing-smart-shunt-vedirect-ove.html




0 Likes 0 ·
oofta avatar image
oofta answered ·

Having the same trouble, now and then one or more USB devices drops out and need the USB dongle unplugged and plugged back in to fix. Still trying to figure out why, but in the meantime I found the post below for resetting individual devices.

https://askubuntu.com/questions/645/how-do-you-reset-a-usb-device-from-the-command-line

Fortunately the Venus image comes with gcc installed, so compiling the short program is no problem. I tried resetting the devices individually with mixed success. What I found works better is to reset the USB hub (bus 1 device 2) .. then everything comes back.

root@raspberrypi2# lsusb -t
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 1: Dev 22, If 0, Class=Vendor Specific Class, Driver=smsc95xx, 480M
        |__ Port 2: Dev 23, If 0, Class=Vendor Specific Class, Driver=cp210x, 12M
        |__ Port 3: Dev 24, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M
        |__ Port 4: Dev 25, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M
        |__ Port 5: Dev 26, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M
root@raspberrypi2# ./usbreset /dev/bus/usb/001/002
Resetting USB device /dev/bus/usb/001/002
Reset successful
root@raspberrypi2#


fwiw, I'm running Venus 3.00~15 on a Pi 3B with generic USB serial dongles connected to a Smartshunt, MPPT 150/70 and Phoenix 48/1200

edit:

Beware however that the ethernet nic is also attached to the usb hub so if you do this over ssh you will briefly lose connectivity. So far I haven't had it *not* come back, but ymmv...

# lsusb
Bus 001 Device 041: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
Bus 001 Device 040: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
Bus 001 Device 039: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
Bus 001 Device 038: ID 10c4:ea60 Cygnal Integrated Products, Inc. CP2102/CP2109 UART Bridge Controller [CP210x family]
Bus 001 Device 037: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub


1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

von-baron avatar image von-baron commented ·

I made mention of this in some other post recently, where serial over USB can not be considered to be 100% reliable. I do not suspect it is specifically because you are using an RPi, but more that it is a fact of using serial over USB, where if the serial connection has a hiccup then it disconnects. You generally need to unplug and plug the USB device back in. It is one reason many industrial equipment manufactures say not to use USB for any permanent connection for control.

Maybe it is related to RFI or some other EMI being imposed on the USB data cables and then that causes grief in the USB interface. Bluetooth is RF as is WiFi.

Another thing might be that if you do not use some degree of electrical isolation between two devices at either end of the USB cable, then that again might cause issues with voltage differentials between the zero volt lines if they don't exactly share the same zero volt potential, whether that be ground or Earth or floating.

Instead, the best is to use native serial, TTL, RS-232 or RS-485 or Ethernet, they don't suffer from this specific USB issue.

But, using USB is very convenient :-)

0 Likes 0 ·
rns avatar image
rns answered ·

I had DIY cables so ordered official victron cables. When I started replacing them, I noticed one USB port on the PI looks not to work anymore or at least instable. As soon as I use it again. all USB devices disconnect. Now using only the other 3 and an USB hub and official cable. Works fine without any disconnect for almost a week.

What I also noticed is that the disconnet most of the time disconnected when all PV power went to 0. Not sure, maybe pure coincidence but could relate it to a PV power going to 0 during the day (winter + clouds ) or at the end of a day.


So multiple factors not sure of any of the conclusions except that the 4th porth of the PI looks dead now and since I use the hub and only the other 3, everything works stable.

4 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

von-baron avatar image von-baron commented ·

You mentioned:

"What I also noticed is that the disconnet most of the time disconnected when all PV power went to 0. Not sure, maybe pure coincidence but could relate it to a PV power going to 0 during the day (winter + clouds ) or at the end of a day."

That to me does seem like a voltage potential issue to me, although I think it would normally occur when there was charging current from charge controller or discharge current from inverter to then cause voltage drop on the zero volt line between the various devices as your RPi.

Using a communications cable with electrical isolation on the TTL side before the USB converter will avoid voltage potential problems.

Ethernet interfaces are isolated as part of the design spec.

RS-485 and RS-422 are differential and so don't need a zero volt reference line, although sometimes one is used in case one device is floating with respect to its supply.

The RPi is not a particularly good design in my opinion, it lacks a lot of what I would call good design for industrial rated. It was developed more as a teaching aid for schools, and so was designed cheaply in my opinion.

Good that you seem to have things solved at this point, hope it plays nice from now.

0 Likes 0 ·
oofta avatar image oofta commented ·
I'm cycling my setup with "PV power" coming from a bench supply and turning an AC load on and off with an SSR attached to the GPIO of a second Pi that I'm using to interrogate dbus on Venus over ssh.


So far I don't see any correlation between any of that and the problem occurring. I've also adjusted the output of my 5v supply (a DIN rail ps) to 5.1v just in case. Unplugged all the USB cables and checked for a difference in ground potential and found none.


Sometimes if fails several times in an hour, other times it goes for many hours between. Very intermittent but consistent regardless.


0 Likes 0 ·
von-baron avatar image von-baron oofta commented ·
Sounds like you understand many aspects of the system, but can understand how frustrating something like this can be.

Just imagine if it was based on that Microsoft OS, Yikes :-|

1 Like 1 ·
oofta avatar image oofta von-baron commented ·

Update - so I ended up replacing the 3 generic USB serial cables with Victron branded ones. Partly I was curious to see if that would help, but also the Victron cables had a couple bonus (like a decent strain relief on the VE.direct end, the dongle part is mid-cable so it doesn't crowd up on the RaspI, and they are longer and reach all the way to where I need them to go).

At any rate, the upshot is that ever since I swapped them out I haven't had a single failure. Totally solid. Go figure.

0 Likes 0 ·
oofta avatar image
oofta answered ·

I noticed this error in syslog when the USB connection craps out:

Jan 23 22:24:10 raspberrypi2 user.err kernel: [248741.182617] pl2303 ttyUSB1: usb_serial_generic_read_bulk_callback - urb stopped: -32

Google search returns a bunch of threads with people talking about the same problem, usually in the context of FTDI or PL2303 dongles (mine is PL2303), possibly involving a race condition. Bottom line though, I'm starting to think this is a Linux kernel bug that has since been fixed, specifically:

https://github.com/raspberrypi/linux/issues/5088

Venus appears to be based on RPi OS which is using Debian 11 Bullseye, which is an LTS release using Linux kernel 5.10.x. Current kernel version for Debian 11 is 5.10.163. On my Venus 3.00~15 the kernel is 5.10.110 which is from April 2022, not ancient but definitely predates the comments (and the fix mentioned) in the thread above.

This seems to be affecting several Venus users. Where does one post to alert the Venus dev team that they might need to bump their kernel version?


2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mike4nl avatar image
mike4nl answered ·

Stupid question: is there a power safe mode active on the USB ports?

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

oofta avatar image oofta commented ·
This is the Venus Linux OS image that Victron distributes so you wouldn't expect there to be, but fwiw I don't see anything that looks remotely like power management running or installed.




0 Likes 0 ·
maxx8888 avatar image
maxx8888 answered ·

I‘m facing exactly same Issue on my Setups.

Was there ever a Solution for that? Venus OS 3.01 used

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

maxx8888 avatar image
maxx8888 answered ·

Now Venus OS 3.10, still around once per 24h disconnect

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

fellpower avatar image
fellpower answered ·

later.....im at 3.40-2 beta - and have the same issue.....disconnecting my devices over time. im only using a 75/15 and the 500A shunt. If im connecting them alone, all is fine. if i connect both, they loosing connection over time...


how to fix that?

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Related Resources

Victron Venus OS Open Source intro page

Venus OS GitHub (please do not post to this)