Cerbo GX random reboots

Question

replit

This site is now in read-only archive mode. Please move all discussion, and create a new account at the new Victron Community site.

question

ektus asked Apr 13, '24

Cerbo GX random reboots

System setup: Cerbo GX MK1 with small touchscreen, wired ethernet connection
Power supply from 48V bus
Pylontech battery, 5x MPPT 150/35, 3~ Multiplus 2 5000, Lynx Shunt,
2x Shelly 3 EM (grid and AC Solar)

Operating system: Current Venus OS Large, ESS active (region: Germany), VRM active

Node-RED active with some flows to evalulate current production and consumtion and alter the grid setpoint in order to optimize daily production.
The old 9,5kWp AC-coupled solar generator may feed in and gets paid for it, the new 5,7kWp DC-coupled system is not allowed to feed in.

Symptoms: The system randomly reboots, sometimes leaving the grid setpoint at a large negative value.

dmesg -H only shows entries since reboot.

Reboot on connection loss in the VRM settings is off.

Sometimes, the reboot doesn't complete, the screen shows a small white square on black background located in the middle and remains unresponsive. The Node-RED flow and the VRM connection are active, but neither local nor remote console work. Power cycling the Cerbo resolves the problem, but sometimes needs two attempts.

Is there a log file somewhere that could give a hint at what might be going on?

Any ideas?

cerbo gx Venus OS Node-RED

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

142 People are following this question.

Modifications Space

Node Red Space

Answer 1 · 2024-04-13T17:42:12Z

Alexandra answered · Apr 13 at 05:42 PM

@ektus

Update the firmware, (the screen with the small white square problem) on of the recent ones was doing this exact thing on a few odd systems.

The grid set point not being where expected sound like a flow restart problem. So not a Victron os issue.

2

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ektus commented · Apr 13 at 06:33 PM

I'm already running version 3.30 build 20240319094732, all of the reported problems are with this version. Checking for updates was the first thing to do. Most components in the system (Cerbo GX, MPPT, Lynx shunt) are up to date, the three Multiplus 2 5000 are on 506.

0 ·

Alexandra ♦ ektus commented · Apr 14 at 06:48 PM

@ektus

I would factory reset and start again.

0 ·

Answer 2 · 2024-04-13T19:06:34Z

kevgermany answered · Apr 13 at 07:06 PM

Could be an intermittent interruption of the DC power to the Cerbo. This will cause an unclean shutdown and unpredictable restart. And that white square means you have to do another unclean shutdown.

3

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ektus commented · Apr 13 at 07:57 PM

The power supply uses the cable that came with the Cerbo and comes directly from the Lynx bus. That one should be stable enough with 8x Pylontech US5000 batteries and at the time of the problem 5x MPPT devices providing power.

0 ·

kevgermany ♦♦ ektus commented · Apr 14 at 01:52 AM

Yes it should be. But if there's a loose connection etc.

Also please check this link: https://community.victronenergy.com/storage/attachments/42304-2022-08-cerbo-gx-power-supply-issue-in-48v-systems.pdf

0 ·

ektus commented · Apr 14 at 06:15 AM

My Cerbo GX serial number starting with HQ2248 the power supply should be okay. I can add the extra capacitor and see if this would improve stability. The voltage at the USB hub is only 4.6V without its own power supply attached. Might this indicate a problem with the Cerbo's 5V rail? I'll measure again dirctly at the Cerbo's port, might also be a diode in the hub lowering the voltage.

0 ·

Answer 3 · 2024-04-14T06:47:14Z

Alex Pescaru answered · Apr 14 at 06:47 AM

What is "top" command saying? Memory and CPU load ?...

Could be a watchdog triggering?...

2

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ektus commented · Apr 14 at 07:30 AM

top reports anywhere between 40% and 75% CPU, memory currently sits at 504M free (about half). node-red at 14..18% and gui at 13%, python3 6% and a lot of smaller processes. This is after 1h uptime, I'll watch further.

If it were a watchdog triggering, would there be log entries somewhere? Debugging random reboots (once or twice a day, if that) without data isn't fun :-(

The 5V measures 5.03V directly at the Cerbo's USB connector, so all good there.

1 ·

Alex Pescaru ektus commented · Apr 14 at 09:19 AM

Pretty high usage of the CPU... Especially from the gui.

Mine doesn't go higher than 2%. The same with python.

All the logs are in the /data/log folder.

2 ·

Answer 4 · 2024-04-14T07:24:50Z

nickdb answered · Apr 14 at 07:24 AM

Try reflash the unit. There is a reset to factory procedure that you can find via google or local search.

Done bother with the cap it won’t help.

Causes can only be - software issue, temp, missed wiring issue or a hardware fault. Eliminate the SW by flashing it from scratch.

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 5 · 2024-04-14T12:05:54Z

okay, we're up to something here. How to reduce load and/or increase the threshold?

From the file /data/logs/messages:

Apr 14 10:27:53 einstein daemon.info connmand[890]: ntp: time slew +0.042245 s
Apr 14 11:27:52 einstein daemon.err watchdog[670]: loadavg 11 7 4 is higher than the given threshold 0 6 6!
Apr 14 11:27:53 einstein daemon.err watchdog[670]: repair binary /usr/sbin/store_watchdog_error.sh returned 253 = 'load average too high'
Apr 14 11:27:53 einstein daemon.alert watchdog[670]: shutting down the system because of error 253 = 'load average too high'
Apr 14 11:27:53 einstein daemon.err watchdog[670]: /usr/sbin/sendmail does not exist or is not executable (errno = 2)
Apr 14 11:28:03 einstein syslog.info syslogd exiting
Apr 14 11:30:18 einstein syslog.info syslogd started: BusyBox v1.31.1

The last screen of top running in the SSH session (might be from an earlier reboot):

Mem: 542428K used, 487656K free, 2972K shrd, 35304K buff, 135828K cached
CPU:  72% usr  10% sys   0% nic  14% idle   0% io   1% irq   1% sirq
Load average: 10.20 7.20 5.36 2/307 7090
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 1960  1100 nodered  S     228m  23%  15% node-red
 1975  1071 root     S     137m  14%  14% /opt/victronenergy/gui/gui -nomouse -display Multi: LinuxFb: VNC:size=800x480:depth=32:passwordFile=/data/conf/vncpassword.txt:0
 1139  1116 root     S    22004   2%   7% {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
  809   807 messageb S     4468   0%   6% dbus-daemon --system --nofork
 1088  1061 root     S    38364   4%   5% {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
 2049  2038 root     S    32272   3%   5% python /data/dbus-shelly-3em-inverter/dbus-shelly-3em-inverter.py
 2041  2039 root     S    32208   3%   5% python /data/dbus-shelly-3em-smartmeter/dbus-shelly-3em-smartmeter.py
 1129  1106 root     S    26032   3%   4% {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
 1161  1147 root     S    21200   2%   4% {dbus_generator.} /usr/bin/python3 -u /opt/victronenergy/dbus-generator-starter/dbus_generator.py
 2034  2028 root     S    55828   5%   3% /usr/bin/flashmq
 1120  1063 root     S    19756   2%   2% /opt/victronenergy/venus-platform/venus-platform
 1131  1110 root     S     8748   1%   2% /opt/victronenergy/hub4control/hub4control
 2033  2024 root     S    21124   2%   2% {vesmart_server.} /usr/bin/python3 -u /opt/victronenergy/vesmart-server/vesmart_server.py -i hci0
 1163  1149 root     S    11272   1%   2% /opt/victronenergy/dbus-fronius/dbus-fronius
 1694  1658 root     S     3668   0%   1% /opt/victronenergy/mk2-dbus/mk2-dbus --log-before 25 --log-after 25 --banner -w -s /dev/ttyS4 -i -t mk3 --settings /data/var/lib/mk2-dbus/m
 1277  1124 www-data S     6908   1%   1% nginx: worker process
 2535  2533 root     S     3480   0%   1% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB1
 1674  1672 root     S     3376   0%   1% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS5
 1683  1681 root     S     3424   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS7
 1664  1659 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS6
 1747  1737 root     S     3648   0%   0% /opt/victronenergy/vecan-dbus/vecan-dbus -c socketcan:can0 --banner --log-before 25 --log-after 25 -vv
 1839  1827 root     S     3408   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB0
13524 13496 root     R     2780   0%   0% top

Answer 6 · 2024-04-14T13:06:11Z

It has been modified, but only with things I need for operation. Notably the interface for my two shelly 3em smart meters and by activating node-red for tuning the grid setpoint. Removing those would lose functionality, especially as only part of the solar generators are allowed to feed to the grid. One of the shellies is needed for the grid metering in ESS, and with the standard ESS settings the battery would be full well before noon, reducing the production of my DC solar to zero during the rest of the day.

The second shelly monitors the AC solar (SMA inverter without usable data connection). This value is needed to dynamically set the grid setpoint.

As is now, I use the DC system for house and battery and sell everything from the AC system unless needed for charging the cars or days with low production. This is switched manually.

Currently, it's once again sitting at the small white square. In this state, everything but the GUI works and the system load looks okay. GUI seems to have used 40MB of RAM, as the free value now sits at 546MB (was 504MB earlier):

root@VictronCerboEinstein:/data/log# top
Mem: 483404K used, 546680K free, 1120K shrd, 24956K buff, 131004K cached
CPU:  32% usr  16% sys   0% nic  44% idle   0% io   4% irq   4% sirq
Load average: 0.87 2.24 2.53 1/305 19722
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  806   804 messageb S    31716   3%   7% dbus-daemon --system --nofork
 2037  2028 root     S    31696   3%   7% python /data/dbus-shelly-3em-smartmeter/dbus-shelly-3em-smartmeter.py
19722  7157 root     R     2688   0%   7% top
 1948  1091 nodered  S     224m  22%   4% node-red
 2038  2031 root     S    31760   3%   4% python /data/dbus-shelly-3em-inverter/dbus-shelly-3em-inverter.py
 1146  1107 root     S    21972   2%   4% {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
 1144  1134 root     S    21068   2%   4% {dbus_generator.} /usr/bin/python3 -u /opt/victronenergy/dbus-generator-starter/dbus_generator.py
 2026  2018 root     S    21044   2%   4% {vesmart_server.} /usr/bin/python3 -u /opt/victronenergy/vesmart-server/vesmart_server.py -i hci0
 1111  1056 root     S    19808   2%   4% /opt/victronenergy/venus-platform/venus-platform
 1153  1136 root     S    11272   1%   4% /opt/victronenergy/dbus-fronius/dbus-fronius
 1123  1101 root     S     8632   1%   4% /opt/victronenergy/hub4control/hub4control
 1680  1652 root     S     3668   0%   4% /opt/victronenergy/mk2-dbus/mk2-dbus --log-before 25 --log-after 25 --banner -w -s /dev/ttyS4 -i -t mk3 --settings /data/var/lib/mk2-dbus
 1697  1688 root     S     3376   0%   4% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS7
 1164  1145 root     S    64136   6%   0% python /data/SetupHelper/PackageManager.py
 1112  1064 root     S    63704   6%   0% /opt/victronenergy/gui/gui -nomouse -display Multi: LinuxFb: VNC:size=800x480:depth=32:passwordFile=/data/conf/vncpassword.txt:0
 2032  2020 root     S    55312   5%   0% /usr/bin/flashmq
 1149  1132 root     S    45300   4%   0% {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 2024  2016 root     S    40592   4%   0% {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1080  1054 root     S    38060   4%   0% {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
 1075  1060 root     S    34192   3%   0% {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
 1148  1109 root     S    27708   3%   0% {dbus_shelly.py} /usr/bin/python3 /opt/victronenergy/dbus-shelly/dbus_shelly.py
 1120  1097 root     S    26032   3%   0% {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
 1118  1095 root     S    23264   2%   0% {netmon} /usr/bin/python3 -u /opt/victronenergy/netmon/netmon
  880     1 root     S    22740   2%   0% php-fpm: master process (/etc/php-fpm.conf)
  881   880 www-data S    22740   2%   0% php-fpm: pool www
  882   880 www-data S    22740   2%   0% php-fpm: pool www
 1157  1138 root     S    21660   2%   0% {dbus_digitalinp} /usr/bin/python3 -u /opt/victronenergy/dbus-digitalinputs/dbus_digitalinputs.py --poll=poll /dev/gpio/digital_input_1 /
 1129  1103 root     S    19532   2%   0% {dbus_vebus_to_p} /usr/bin/python3 -u /opt/victronenergy/dbus-vebus-to-pvinverter/dbus_vebus_to_pvinverter.py
 1074  1066 simple-u S    12592   1%   0% /bin/simple-upnpd --xml /var/run/simple-upnpd.xml -d
  118     1 root     S    12068   1%   0% /sbin/udevd -d
 1072  1062 root     S    10220   1%   0% /opt/victronenergy/venus-access/venus-access
  937     1 root     S     9072   1%   0% /usr/sbin/wpa_supplicant -u -O /var/run/wpa_supplicant -s
 2897  2867 root     S     8308   1%   0% -sh
 7157  7105 root     S     8308   1%   0% -sh
 1938     1 root     S     8288   1%   0% -sh
  809     1 root     S     7592   1%   0% /usr/sbin/haveged -w 1024 -v 1
 1269  1115 www-data S     6908   1%   0% nginx: worker process
 1115  1093 root     S     6640   1%   0% nginx: master process /usr/sbin/nginx

Answer 7 · 2024-04-14T16:28:02Z

Kevin Windrem answered · Apr 14 at 04:28 PM

The white square in the middle of the local screen indicates a problem with the gui task. Check /data/log/start-gui/current. Might provide a clue.

3

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Kevin Windrem commented · Apr 14 at 05:58 PM

There should be more entries for a normal boot.

If you use tail -50 /data/log/start-gui/current | tai64nlocal, the times will be converted to a readable format and you can more easily tell if things have changed since the last look at the logs. See for example if the gui is restarting over and over again.

Another thing to look at is dbus-spy and see if there's a com.victronenergy.gui service listed. It may be coming and going if the gui is restarting.

If you have not done so already, it may be time to reinstall the firmware. Go to the Firmware menu, select the backup firmware and after it's rebooted, go to online updates and download the latest firmware again. If there are any mods you've made to the system they may need to be reinstalled, but wait to see if the system comes up before installing them. Might also be "interesting" to turn off node red to see if that is somehow the source of your problems.

1 ·

ektus Kevin Windrem commented · Apr 15 at 03:05 AM

@Kevin Windrem @Alexandra

Looks like there's a problem with version 3.30:

https://community.victronenergy.com/questions/270553/has-anyone-had-issues-with-cerbo-gx-remote-console.html

I'll try and revert to 3.22.

0 ·

Alexandra ♦ ektus commented · Apr 15 at 04:26 AM

Yeah it's a funny one.

I am running 3.3 with no issues.

I had to do a factory reset though on one of the recent versions. But stay with the one that worked for you most correctly for now.

0 ·

Answer 8 · 2024-04-14T17:25:20Z

root@VictronCerboEinstein:/data/log/start-gui# more current
@40000000661b9fdc1ee9b434 [Service] installing "/service/websockify-c"
@40000000661b9ffd14b5eb04 file:///opt/victronenergy/gui/qml/OverviewTiles.qml:148: TypeError: Result of expression 'stateTile.activeNotifications' [undefined] is not an object.
@40000000661b9ffd14b90014 file:///opt/victronenergy/gui/qml/OverviewTiles.qml:144: TypeError: Result of expression 'stateTile.activeNotifications' [undefined] is not an object.
@40000000661b9ffd14b91f54 file:///opt/victronenergy/gui/qml/OverviewTiles.qml:94: TypeError: Result of expression 'activeNotifications' [undefined] is not an object.
@40000000661b9ffd14b9e68c file:///opt/victronenergy/gui/qml/OverviewTiles.qml:90: ReferenceError: Can't find variable: NotificationCenter
@40000000661ba2230f78b75c ScreenSaver::timer
@40000000661ba2230f7dfeec ScreenSaver::screenOff
@40000000661bac862865b29c VePlatformVenus::wakeupEvent
@40000000661bac8628685a4c ScreenSaver::enable
@40000000661bac86286869ec ScreenSaver::screenOn
@40000000661bac8628734efc VeQuickView::mousePressEvent: wakeupEvent
@40000000661baede287f8bcc ScreenSaver::timer
@40000000661baede287fd604 ScreenSaver::screenOff
@40000000661bbe550e63096c *** CCGX booted (30253) ***
@40000000661bbe552cedbb5c *** starting start-gui ***
@40000000661bbe56017ada54 Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe5627b8636c Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe572fc2dc2c Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe5835579614 Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe5939e36f64 Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe5b043ec8cc Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe5c0bd4b59c Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe5d1234d5d4 Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe5e1e0251d4 Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe5f247fd57c Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe602a73aef4 Error org.freedesktop.DBus.Error.ServiceUnknown: The name com.victronenergy.settings was not provided by any .service files
@40000000661bbe6231b34634 method return time=1713094232.833523 sender=:1.43 -> destination=:1.46 serial=9 reply_serial=2
@40000000661bbe6231b36d44    int32 0
@40000000661bbe6312bd0cec *** headless device=0
@40000000661bbe6312bd2c2c *** Waiting for localsettings...
@40000000661bbe632317dd24 *** Localsettings is up, continuing...
@40000000661bbe642d55201c *** Starting gui, with VNC enabled (VncLocal=1 VncInternet=1)
@40000000661bbe65199ba80c QVNCServer created on port 5900
@40000000661bbe651d71970c Reloading input devices: "LinuxInput:/dev/input/event0 LinuxInput:/dev/input/event1 LinuxInput:/dev/input/event2"
@40000000661bbe651f4bd934 "using /etc/venus for runtime features"
@40000000661bbe651fa77e24 "running on einstein"
@40000000661bbe6523d01d1c Connecting to deprecated signal QDBusConnectionInterface::serviceOwnerChanged(QString,QString,QString)
@40000000661bbe6536ef609c Creating settings
@40000000661bbe661d81db1c [VeQItemExportedDbusServices] Registered service "debug.victronenergy.gui"
root@VictronCerboEinstein:/data/log/start-gui#

That's the current file with GUI not running.

Answer 9 · 2024-04-15T03:29:18Z

Reverted to 3.22, now top looks much nicer:

Mem: 514880K used, 515204K free, 3160K shrd, 20440K buff, 137632K cached
CPU:  46% usr   6% sys   0% nic  44% idle   0% io   1% irq   0% sirq
Load average: 1.02 2.21 1.24 1/307 3657
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 1969  1115 nodered  S     216m  21%  12% node-red
 1152  1131 root     S    21744   2%   5% {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
 2055  2052 root     S    31696   3%   4% python /data/dbus-shelly-3em-smartmeter/dbus-shelly-3em-smartmeter.py
  826   824 messageb S     4148   0%   4% dbus-daemon --system --nofork
 2056  2054 root     S    31760   3%   4% python /data/dbus-shelly-3em-inverter/dbus-shelly-3em-inverter.py
 1097  1076 root     S    38124   4%   3% {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
 1135  1087 root     S     144m  14%   3% /opt/victronenergy/gui/gui -nomouse -display Multi: LinuxFb: VNC:size=800x480:depth=32:passwordFile=/data/conf/vncpassword.txt:0
 1178  1160 root     S    21068   2%   2% {dbus_generator.} /usr/bin/python3 -u /opt/victronenergy/dbus-generator-starter/dbus_generator.py
 1148  1121 root     S    26032   3%   2% {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
 1146  1125 root     S     8632   1%   1% /opt/victronenergy/hub4control/hub4control
 2047  2039 root     S    55308   5%   1% /usr/bin/flashmq
 2053  2041 root     S    21104   2%   1% {vesmart_server.} /usr/bin/python3 -u /opt/victronenergy/vesmart-server/vesmart_server.py -i hci0
 1180  1162 root     S    11272   1%   1% /opt/victronenergy/dbus-fronius/dbus-fronius
 1711  1672 root     S     3664   0%   1% /opt/victronenergy/mk2-dbus/mk2-dbus --log-before 25 --log-after 25 --banner -w -s /dev/ttyS4 -i -t mk3 --settings /data/var/lib/mk2-dbus/mkxport-
 1099  1090 simple-u S    12592   1%   1% /bin/simple-upnpd --xml /var/run/simple-upnpd.xml -d
 2540  2538 root     S     3480   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB1
 1729  1718 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS7
 1877  1866 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB0
 1690  1688 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS5
 1682  1678 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS6
 1834  1826 root     S     3644   0%   0% /opt/victronenergy/vecan-dbus/vecan-dbus -c socketcan:can0 --banner --log-before 25 --log-after 25 -vv
 3509  3238 root     R     2780   0%   0% top
 1182  1158 root     S    45304   4%   0% {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 1154  1127 root     S    19532   2%   0% {dbus_vebus_to_p} /usr/bin/python3 -u /opt/victronenergy/dbus-vebus-to-pvinverter/dbus_vebus_to_pvinverter.py
 1828  1820 root     S     3176   0%   0% /opt/victronenergy/can-bus-bms/can-bus-bms --log-before 25 --log-after 25 -vv -c socketcan:can1 --banner
 1194  1170 root     S    64136   6%   0% python /data/SetupHelper/PackageManager.py
 1105  1078 root     S    13620   1%   0% /opt/victronenergy/venus-platform/venus-platform
 3217  1830 root     S     5524   1%   0% sshd: root@pts/0
 1192  1168 root     S     3484   0%   0% /opt/victronenergy/dbus-adc/dbus-adc --banner
 1138  1096 root     S     3044   0%   0% {serial-starter.} /bin/bash /opt/victronenergy/serial-starter/serial-starter.sh
   48     2 root     IW       0   0%   0% [kworker/u4:1-ev]
   59     2 root     IW       0   0%   0% [kworker/u4:3-ev]
 2051  2043 root     S    40324   4%   0% {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1108  1083 root     S    34200   3%   0% {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
 1179  1133 root     S    27688   3%   0% {dbus_shelly.py} /usr/bin/python3 /opt/victronenergy/dbus-shelly/dbus_shelly.py

Answer 10 · 2024-04-15T07:10:09Z

semlohnhoj answered · Apr 15 at 07:10 AM

This is interesting. I have the new Victron Energy Meter VM-3P75CT on ve.can and since then I've had regular random reboots. I put this down to the alleged "professional quality" ethernet cable I used and replaced it with a krone ethernet cable which did seem to reduce the occurrences but it is still happening maybe every other day now instead of several times a day. I'm using Node Red as well monitoring the meter. I didn't realise there were logs I could investigate so I'll look at that. I'll check them later when I get a moment.

This may well be unrelated but I thought I'd mention it.

2

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

semlohnhoj commented · Apr 15 at 09:41 AM

Also, in my case I've noticed it operates in Schrödinger's cat mode where if you never open the NodeRed dashboard it doesn't reboot. I just opened it now for the first time in a few days and within 30 seconds it has rebooted. I have noticed this before but just thought I was imagining it. This could just be load related pushing the CPU over the edge I suppose?

0 ·

ektus semlohnhoj commented · Apr 15 at 10:44 AM

Sounds plausible. For me with 3.22, the cpu jumps around 5% if I open the Node-red dashboard. Without it 63% to 68% and with the dashboard open more like 68% to 75%. This extra load might be too much in some cases.

It would be nice if the watchdog settings could be altered to trigger at a higher load. Or the efficiency of the system be improved to avoid the high loads altogether.

0 ·

Answer 11 · 2024-04-15T09:08:21Z

For reference: With version 3.22 and nothing changed in the config, after 2:40 hours top looks as follows:

Mem: 532984K used, 497100K free, 2940K shrd, 32216K buff, 136200K cached
CPU:  60% usr   6% sys   0% nic  30% idle   0% io   1% irq   0% sirq
Load average: 2.99 3.85 4.08 1/308 5973
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 1077  1063 root     R     137m  14%  14% /opt/victronenergy/gui/gui -nomouse -display Multi: LinuxFb: VNC:size=800x480:depth=32:passwordFile=/data/conf/vncpassword
 1954  1089 nodered  S     226m  22%  10% node-red
 1149  1125 root     S    21696   2%   7% {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
  803   801 messageb S     4912   0%   5% dbus-daemon --system --nofork
 1098  1053 root     S    38204   4%   5% {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
 2043  2040 root     S    31760   3%   4% python /data/dbus-shelly-3em-inverter/dbus-shelly-3em-inverter.py
 2041  2037 root     S    31696   3%   4% python /data/dbus-shelly-3em-smartmeter/dbus-shelly-3em-smartmeter.py
 1158  1133 root     S    21196   2%   3% {dbus_generator.} /usr/bin/python3 -u /opt/victronenergy/dbus-generator-starter/dbus_generator.py
 1115  1095 root     S    26032   3%   2% {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
 1161  1135 root     S    11272   1%   2% /opt/victronenergy/dbus-fronius/dbus-fronius
 1152  1113 root     S     8764   1%   2% /opt/victronenergy/hub4control/hub4control
 2034  2024 root     S    55824   5%   1% /usr/bin/flashmq
 2033  2028 root     S    21092   2%   1% {vesmart_server.} /usr/bin/python3 -u /opt/victronenergy/vesmart-server/vesmart_server.py -i hci0
 1685  1652 root     S     3664   0%   1% /opt/victronenergy/mk2-dbus/mk2-dbus --log-before 25 --log-after 25 --banner -w -s /dev/ttyS4 -i -t mk3 --settings /data/v
 1842  1834 root     S     3404   0%   1% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB0
 2511  2509 root     S     3404   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB1
 1655  1651 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS6
 1668  1666 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS5
 1701  1694 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS7
 1776  1771 root     S     3644   0%   0% /opt/victronenergy/vecan-dbus/vecan-dbus -c socketcan:can0 --banner --log-before 25 --log-after 25 -vv
 5748  5733 root     R     2780   0%   0% top
 1159  1131 root     S    45304   4%   0% {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 1145  1117 root     S    19532   2%   0% {dbus_vebus_to_p} /usr/bin/python3 -u /opt/victronenergy/dbus-vebus-to-pvinverter/dbus_vebus_to_pvinverter.py
 1168  1141 root     S     3484   0%   0% /opt/victronenergy/dbus-adc/dbus-adc --banner
 1169  1143 root     S    64136   6%   0% python /data/SetupHelper/PackageManager.py
 1808  1804 root     S     3176   0%   0% /opt/victronenergy/can-bus-bms/can-bus-bms --log-before 25 --log-after 25 -vv -c socketcan:can1 --banner
 1107  1093 root     S    23264   2%   0% {netmon} /usr/bin/python3 -u /opt/victronenergy/netmon/netmon
 5688  1810 root     S     5524   1%   0% sshd: root@pts/0
 1097  1072 root     S     3044   0%   0% {serial-starter.} /bin/bash /opt/victronenergy/serial-starter/serial-starter.sh
    7     2 root     IW       0   0%   0% [kworker/u4:0-ev]
  788     2 root     SW       0   0%   0% [RTW_CMD_THREAD]
 2039  2026 root     S    40324   4%   0% {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1075  1059 root     S    34200   3%   0% {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
 1150  1127 root     S    27688   3%   0% {dbus_shelly.py} /usr/bin/python3 /opt/victronenergy/dbus-shelly/dbus_shelly.py
  877     1 root     S    22740   2%   0% php-fpm: master process (/etc/php-fpm.conf)
  878   877 www-data S    22740   2%   0% php-fpm: pool www
  879   877 www-data S    22740   2%   0% php-fpm: pool www
 1162  1137 root     S    21660   2%   0% {dbus_digitalinp} /usr/bin/python3 -u /opt/victronenergy/dbus-digitalinputs/dbus_digitalinputs.py --poll=poll /dev/gpio/di
 1100  1055 root     S    13620   1%   0% /opt/victronenergy/venus-platform/venus-platform

Answer 12 · 2024-04-15T16:34:26Z

nunorgp answered · Apr 15 at 04:34 PM

Have the exact same issue with that firmware :

but lately the gui as been very very slow ... even before the update to the latest firmware.

1713198753492.png (14.9 KiB)

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 13 · 2024-04-18T14:31:22Z

I'll probably go and test tomorrow, but in the meantime, I've had another unexpected reboot, this time with the older 3.22. Looks like it was once again due to some overload. Last refresh of top:

Mem: 632912K used, 397172K free, 2984K shrd, 83980K buff, 147084K cached
CPU:  62% usr   6% sys   0% nic  28% idle   0% io   1% irq   0% sirq
Load average: 10.94 7.00 4.83 2/308 19822
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 1077  1063 root     R     146m  15%  15% /opt/victronenergy/gui/gui -nomouse -display Multi: LinuxFb: VNC:size=800x480:depth=32:passwordFile=/data/conf/vncpassword
 1954  1089 nodered  S     246m  24%  11% node-red
 1149  1125 root     S    21696   2%   6% {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
  803   801 messageb R     4912   0%   5% dbus-daemon --system --nofork
 1098  1053 root     S    38204   4%   5% {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
 2041  2037 root     S    32340   3%   5% python /data/dbus-shelly-3em-smartmeter/dbus-shelly-3em-smartmeter.py
 2043  2040 root     S    32272   3%   5% python /data/dbus-shelly-3em-inverter/dbus-shelly-3em-inverter.py
 1158  1133 root     S    21196   2%   3% {dbus_generator.} /usr/bin/python3 -u /opt/victronenergy/dbus-generator-starter/dbus_generator.py
 1161  1135 root     S    11272   1%   2% /opt/victronenergy/dbus-fronius/dbus-fronius
 1152  1113 root     S     8748   1%   2% /opt/victronenergy/hub4control/hub4control
 2033  2028 root     S    21092   2%   2% {vesmart_server.} /usr/bin/python3 -u /opt/victronenergy/vesmart-server/vesmart_server.py -i hci0
 1115  1095 root     S    26032   3%   2% {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
 2034  2024 root     S    55824   5%   2% /usr/bin/flashmq
 1685  1652 root     S     3664   0%   1% /opt/victronenergy/mk2-dbus/mk2-dbus --log-before 25 --log-after 25 --banner -w -s /dev/ttyS4 -i -t mk3 --settings /data/v
 1701  1694 root     S     3424   0%   1% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS7
 1842  1834 root     S     3404   0%   1% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB0
 1655  1651 root     S     3424   0%   1% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS6
 1668  1666 root     S     3424   0%   1% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS5
 2511  2509 root     S     3404   0%   1% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB1
 1808  1804 root     S     3176   0%   0% /opt/victronenergy/can-bus-bms/can-bus-bms --log-before 25 --log-after 25 -vv -c socketcan:can1 --banner
26343 26323 root     R     2780   0%   0% top
 1145  1117 root     S    19532   2%   0% {dbus_vebus_to_p} /usr/bin/python3 -u /opt/victronenergy/dbus-vebus-to-pvinverter/dbus_vebus_to_pvinverter.py
 1776  1771 root     S     3752   0%   0% /opt/victronenergy/vecan-dbus/vecan-dbus -c socketcan:can0 --banner --log-before 25 --log-after 25 -vv
 1169  1143 root     S    64136   6%   0% python /data/SetupHelper/PackageManager.py
 1159  1131 root     S    47608   5%   0% {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 1097  1072 root     S     3044   0%   0% {serial-starter.} /bin/bash /opt/victronenergy/serial-starter/serial-starter.sh
 1100  1055 root     S    13620   1%   0% /opt/victronenergy/venus-platform/venus-platform
 1168  1141 root     S     3484   0%   0% /opt/victronenergy/dbus-adc/dbus-adc --banner
14317     2 root     IW       0   0%   0% [kworker/u4:2-ev]
19363     2 root     IW       0   0%   0% [kworker/u4:1-ev]
 2039  2026 root     S    40324   4%   0% {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1075  1059 root     S    34200   3%   0% {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
 1150  1127 root     S    27688   3%   0% {dbus_shelly.py} /usr/bin/python3 /opt/victronenergy/dbus-shelly/dbus_shelly.py
 1107  1093 root     S    23264   2%   0% {netmon} /usr/bin/python3 -u /opt/victronenergy/netmon/netmon
  877     1 root     S    22740   2%   0% php-fpm: master process (/etc/php-fpm.conf)
  878   877 www-data S    22740   2%   0% php-fpm: pool www
  879   877 www-data S    22740   2%   0% php-fpm: pool www
 1162  1137 root     S    21660   2%   0% {dbus_digitalinp} /usr/bin/python3 -u /opt/victronenergy/dbus-digitalinputs/dbus_digitalinputs.py --poll=poll /dev/gpio/di
 1082  1066 simple-u S    12856   1%   0% /bin/simple-upnpd --xml /var/run/simple-upnpd.xml -d
  115     1 root     S    11944   1%   0% /sbin/udevd -d
Connection to 192.168.0.80 closed by remote host.ctronenergy/venus-access/venus-access
Connection to 192.168.0.80 closed.

Answer 14 · 2024-04-19T22:53:31Z

mvader (Victron Energy) answered · Apr 19 at 10:53 PM

Hi @ektus ,

I see nothing unusual in your dumps of top output. Also the gui being at 14% is quite normal, having it open at the overview page, with the “moving ants” does that.

Also, I don’t expect that the problem that we fixed in v3.31 has anything to do with what you’re seeing. And that you’re still experiencing reboots with v3.31~beta confirms that.

Looks to me that you simply have a quite busy Cerbo GX. There is nothing that we (Victron) are going to do about that any time soon I’m afraid.

There are no trivial improvements to cpu load; if they would exist we’d already have implemented them.

To resolve this, get a faster GX device: Ekrano GX.

Or check how often a second there are dbus messages sent by your Shelly system. Since that is a mod you’re running.

Hope this helps, and if nothing else then for making clear that its best not to hope that this is because of a bug soon solved.

Matthijs

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 15 · 2024-04-20T01:32:32Z

The Shelly 3EM are queried once every 750ms. They are in the low single digits CPU load. I've been watching top for quite some time, and the load average is slowly creeping up and down. While watching, I've seen as high as 5.7 and as low as 1.8 for the 5min average. And anything between 0.9 and 10 for the realtime value.

One trivial improvement would be to not run any fronius-related tasks if there is nothing from this vendor present in the system.

Now I've got the 5min limit set to 8 while the 15min limit remains at 6. Let's see what happens.

For reference, at 3 hours uptime in the middle of the night:

Mem: 546632K used, 483452K free, 2980K shrd, 36936K buff, 136696K cached
Mem: 546312K used, 483772K free, 2980K shrd, 37248K buff, 136688K cached
CPU:  57% usr   7% sys   0% nic  33% idle   0% io   1% irq   0% sirq
Load average: 0.96 1.81 2.57 2/309 17346
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 1113  1082 root     S     138m  14%  12% /opt/victronenergy/gui/gui -nomouse -display Multi: LinuxFb: VNC:size=800x480:depth=32:passwordFile=/data/conf/vncpassword.txt:0
 1946  1094 nodered  S     228m  23%   9% node-red
 1107  1072 root     S    38284   4%   8% {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
 1152  1145 root     S    21980   2%   5% {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
 2046  2041 root     S    32340   3%   4% python /data/dbus-shelly-3em-smartmeter/dbus-shelly-3em-smartmeter.py
 2045  2042 root     S    32272   3%   4% python /data/dbus-shelly-3em-inverter/dbus-shelly-3em-inverter.py
  822   820 messageb S     4548   0%   4% dbus-daemon --system --nofork
 1186  1159 root     S    21068   2%   2% {dbus_generator.} /usr/bin/python3 -u /opt/victronenergy/dbus-generator-starter/dbus_generator.py
 2038  2027 root     S    55852   5%   2% /usr/bin/flashmq
 1127  1120 root     S    26036   3%   2% {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
 2036  2029 root     S    21104   2%   1% {vesmart_server.} /usr/bin/python3 -u /opt/victronenergy/vesmart-server/vesmart_server.py -i hci0
 1104  1074 root     S    19792   2%   1% /opt/victronenergy/venus-platform/venus-platform
 1126  1124 root     S     8736   1%   1% /opt/victronenergy/hub4control/hub4control
 1172  1161 root     S    11272   1%   1% /opt/victronenergy/dbus-fronius/dbus-fronius
 1707  1674 root     S     3668   0%   1% /opt/victronenergy/mk2-dbus/mk2-dbus --log-before 25 --log-after 25 --banner -w -s /dev/ttyS4 -i -t mk3 --settings /data/var/lib/mk2-dbu
 1805  1793 root     S     3648   0%   0% /opt/victronenergy/vecan-dbus/vecan-dbus -c socketcan:can0 --banner --log-before 25 --log-after 25 -vv
 1686  1684 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS5
 1716  1712 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS7
 1873  1869 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB0
 1187  1180 root     S    64136   6%   0% python /data/SetupHelper/PackageManager.py
 2802  2798 root     S     3408   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB1
 1676  1669 root     S     3376   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS6
 1168  1157 root     S    45300   4%   0% {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 1191  1178 root     S     3484   0%   0% /opt/victronenergy/dbus-adc/dbus-adc --banner
 1142  1136 root     S    19532   2%   0% {dbus_vebus_to_p} /usr/bin/python3 -u /opt/victronenergy/dbus-vebus-to-pvinverter/dbus_vebus_to_pvinverter.py
 1861  1856 root     S     3176   0%   0% /opt/victronenergy/can-bus-bms/can-bus-bms --log-before 25 --log-after 25 -vv -c socketcan:can1 --banner
 1109  1090 root     S     3044   0%   0% {serial-starter.} /bin/bash /opt/victronenergy/serial-starter/serial-starter.sh
17324  9639 root     R     2780   0%   0% top
   48     2 root     IW       0   0%   0% [kworker/u4:1-ev]
  807     2 root     SW       0   0%   0% [RTW_CMD_THREAD]
13993     2 root     IW       0   0%   0% [kworker/u4:0-ev]
13699     2 root     IW       0   0%   0% [kworker/0:2-eve]
 2037  2025 root     S    40592   4%   0% {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1096  1078 root     S    34192   3%   0% {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
 1155  1147 root     S    27708   3%   0% {dbus_shelly.py} /usr/bin/python3 /opt/victronenergy/dbus-shelly/dbus_shelly.py
 1130  1118 root     S    23264   2%   0% {netmon} /usr/bin/python3 -u /opt/victronenergy/netmon/netmon
  896     1 root     S    22740   2%   0% php-fpm: master process (/etc/php-fpm.conf)
  897   896 www-data S    22740   2%   0% php-fpm: pool www
  898   896 www-data S    22740   2%   0% php-fpm: pool www
 1182  1174 root     S    21660   2%   0% {dbus_digitalinp} /usr/bin/python3 -u /opt/victronenergy/dbus-digitalinputs/dbus_digitalinputs.py --poll=poll /dev/gpio/digital_input_1 
 1101  1084 simple-u S    12592   1%   0% /bin/simple-upnpd --xml /var/run/simple-upnpd.xml -d
  116     1 root     S    11680   1%   0% /sbin/udevd -d
 1112  1080 root     S    10220   1%   0% /opt/victronenergy/venus-access/venus-access
  947     1 root     S     9072   1%   0% /usr/sbin/wpa_supplicant -u -O /var/run/wpa_supplicant -s
 2020     1 root     S     8288   1%   0% -sh
 9639  9603 root     S     8172   1%   0% -sh
  825     1 root     S     7592   1%   0% /usr/sbin/haveged -w 1024 -v 1

Answer 16 · 2024-04-20T07:14:51Z

Alex Pescaru answered · Apr 20 at 07:14 AM

Hi @ektus

You can always try to systematically kill the PID process you don't want and see if in that session the watchdog based reboot happens or not.

Then, once you find the culprit, invalidate its start at the boot time.

Although killing one process and reducing CPU load could mask another problem with another process.

Alex

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 17 · 2024-04-20T10:19:06Z

ektus answered · Apr 20 at 10:19 AM

Looks like raising the 5min threshold to 8 has helped avoiding at least one reboot today, as I've seen the 5min average reach 7.3 or so half an hour ago and no reboot yet. Now it's back down to 3.77. No processes stand out, so probably just a lot of simultaneous starts randomly occurring at the same time. I'll try and see if I can spread out the "once per second" tasks in Node-red some.

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 18 · 2024-04-21T08:06:55Z

ektus answered · Apr 21 at 08:06 AM

Still running. Uptime is now 34hrs and counting. Nothing changed besides the watchdog parameters. I've even seen averages above 8, but those have come down relatively quickly again.

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 19 · 2024-04-22T07:56:00Z

Dierk Grossfeld answered · Apr 22 at 07:56 AM

I had those unforseen reboots also quite a while ago.
That load average watchdog reboots the cerbo hard, so headshot if you want to call it like this.

I had that mostly due to the custom component for my go-e Charger wallbox.

The python script is fetching the data via curl from the charger and is based on the shelly script you run.
Since wireless is sometimes due to key exchanges and stuff a bit unreliable it comes to connections that could not be established.

Since the script misses to define a timeout for a connection, the script stalls and waits forever for a response.

That in turn leads to a daemonize state for that process which in turn causes the load to rise (sysload 1/5/15 expresses an amount of waiting processes more or less).
I cant remember 100% for sure if the cerbo runs the process multiple time then, but at least it causes the sysload based watchdog to reboot the device.

Also at least the goecharger script also causes much load due to heavily logging in a log file. File writes on a nand also cause high load die to the relatively low speed of that storage, so they are also expensive from a sysload perspective.

I havent checked the sources for the shelly script, but check for things like requests.get (curl)

request_data = requests.get(url = URL)

and try adding a timeout:

request_data = requests.get(url = URL, timeout=3)

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 20 · 2024-04-22T13:55:06Z

ektus answered · Apr 22 at 01:55 PM

There is only one instance of requests.get, and that one has timeout=5 as parameter. Line 134 in

https://github.com/fabian-lauer/dbus-shelly-3em-smartmeter/blob/main/dbus-shelly-3em-smartmeter.py

There are lengthy logfiles, but logging is set to ERROR. The bigger one is 12MB, the smaller one 3MB but that one hasn't been installed as long. I've just now renamed both of these to have a fresh start.

The Shelly scripts might contribute to the problem, but I haven't seen excess CPU usage from these. The system is still running, now at 2 days and 16 hours. That's with watchdog threshold raised to 0 8 6.

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Dierk Grossfeld commented · Apr 22 at 02:03 PM

so timeout 5 seconds.

Thats good and should at least avoid high load.
but nevertheless, there has to be a reason for such high load.

I have that goe charger script + a little bit of node red running to show a dashboard and control the goe for excess pv charging.

normal load on mine is around 1 to 1.5

root@einstein:~# uptime
 14:01:20 up 33 days,  7:02,  load average: 1.87, 1.41, 1.50

0 ·

Answer 21 · 2024-04-30T05:06:22Z

My Cerbo GX has been running for more than a week now. Tonight the watchdog did trigger once again. 15 minute average still below 6, so it's a short-term overload. Reason still unknown.

Mem: 660280K used, 369804K free, 3080K shrd, 85860K buff, 157336K cached
CPU:  74% usr   7% sys   0% nic  15% idle   0% io   1% irq   0% sirq
Load average: 11.77 9.15 5.74 10/316 18741
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 1946  1094 nodered  R     251m  25%  23% node-red
 1113  1082 root     R     147m  15%  12% /opt/victronenergy/gui/gui -nomouse -display Multi: LinuxFb: VNC:size=800x480:depth=32:passwordFile=/data/conf/vncpassw
 1152  1145 root     R    22028   2%   6% {dbus_systemcalc} /usr/bin/python3 -u /opt/victronenergy/dbus-systemcalc-py/dbus_systemcalc.py
  822   820 messageb R     4708   0%   5% dbus-daemon --system --nofork
 2046  2041 root     R    32340   3%   5% python /data/dbus-shelly-3em-smartmeter/dbus-shelly-3em-smartmeter.py
 2045  2042 root     S    32272   3%   5% python /data/dbus-shelly-3em-inverter/dbus-shelly-3em-inverter.py
 1107  1072 root     S    38284   4%   4% {vrmlogger.py} /usr/bin/python3 -u /opt/victronenergy/vrmlogger/vrmlogger.py
 1186  1159 root     S    21160   2%   3% {dbus_generator.} /usr/bin/python3 -u /opt/victronenergy/dbus-generator-starter/dbus_generator.py
 1104  1074 root     S    20036   2%   2% /opt/victronenergy/venus-platform/venus-platform
 1127  1120 root     S    26036   3%   2% {localsettings.p} /usr/bin/python3 -u /opt/victronenergy/localsettings/localsettings.py --path=/data/conf
 2036  2029 root     S    21104   2%   2% {vesmart_server.} /usr/bin/python3 -u /opt/victronenergy/vesmart-server/vesmart_server.py -i hci0
 1172  1161 root     S    11348   1%   2% /opt/victronenergy/dbus-fronius/dbus-fronius
 1126  1124 root     S     8824   1%   2% /opt/victronenergy/hub4control/hub4control
 2038  2027 root     S    55852   5%   1% /usr/bin/flashmq
 1707  1674 root     S     3732   0%   1% /opt/victronenergy/mk2-dbus/mk2-dbus --log-before 25 --log-after 25 --banner -w -s /dev/ttyS4 -i -t mk3 --settings /dat
 1276  1129 www-data S     7308   1%   1% nginx: worker process
 8110  8108 root     S     3480   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS6
 8076  8074 root     S     3424   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS5
 8531  8529 root     S     3424   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB1
 1805  1793 root     S     3752   0%   0% /opt/victronenergy/vecan-dbus/vecan-dbus -c socketcan:can0 --banner --log-before 25 --log-after 25 -vv
 8139  8137 root     S     3424   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 0 --banner -s /dev/ttyS7
 8489  8487 root     S     3416   0%   0% /opt/victronenergy/vedirect-interface/vedirect-dbus -v --log-before 25 --log-after 25 -t 3 --banner -s /dev/ttyUSB0
31223 31125 root     R     2780   0%   0% top
 1142  1136 root     S    19532   2%   0% {dbus_vebus_to_p} /usr/bin/python3 -u /opt/victronenergy/dbus-vebus-to-pvinverter/dbus_vebus_to_pvinverter.py
 1101  1084 simple-u S    12860   1%   0% /bin/simple-upnpd --xml /var/run/simple-upnpd.xml -d
 1187  1180 root     S    64136   6%   0% python /data/SetupHelper/PackageManager.py
 1168  1157 root     S    48052   5%   0% {dbus-modbus-cli} /usr/bin/python3 -u /opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py
 1109  1090 root     S     3044   0%   0% {serial-starter.} /bin/bash /opt/victronenergy/serial-starter/serial-starter.sh
 1191  1178 root     S     3484   0%   0% /opt/victronenergy/dbus-adc/dbus-adc --banner
 1861  1856 root     S     3176   0%   0% /opt/victronenergy/can-bus-bms/can-bus-bms --log-before 25 --log-after 25 -vv -c socketcan:can1 --banner
 6300     2 root     IW       0   0%   0% [kworker/u4:0-ev]
 7905     2 root     IW       0   0%   0% [kworker/u4:3-ev]
 2037  2025 root     S    42736   4%   0% {mqtt-rpc.py} /usr/bin/python3 -u /opt/victronenergy/mqtt-rpc/mqtt-rpc.py
 1096  1078 root     S    34192   3%   0% {venus-button-ha} /usr/bin/python3 -u /opt/victronenergy/venus-button-handler/venus-button-handler -D
Connection to 192.168.0.80 closed by remote host

messages:

Apr 30 00:14:43 einstein daemon.info connmand[904]: ntp: time slew +0.040978 s
Apr 30 01:22:59 einstein daemon.info connmand[904]: ntp: time slew +0.039981 s
Apr 30 02:05:01 einstein cron.info CROND[21907]: (root) CMD (/opt/victronenergy/swupdate-scripts/check-updates.sh -auto -delay)
Apr 30 02:05:01 einstein cron.info CROND[21904]: (root) CMDOUT (*** Checking for updates ***)
Apr 30 02:05:01 einstein cron.info CROND[21904]: (root) CMDOUT (arguments: -auto -delay)
Apr 30 02:05:01 einstein cron.info CROND[21904]: (root) CMDOUT (Sleeping for 2304 seconds)
Apr 30 02:31:15 einstein daemon.info connmand[904]: ntp: time slew +0.038960 s
Apr 30 02:43:25 einstein cron.info CROND[21904]: (root) CMDOUT (Retrieving latest version... (from https://updates.victronenergy.com/feeds/venus/release/images/&gt;
Apr 30 02:43:26 einstein cron.info CROND[21904]: (root) CMDOUT (installed: 20240417095121 v3.31~3)
Apr 30 02:43:26 einstein cron.info CROND[21904]: (root) CMDOUT (available: 20240422090512 v3.31)
Apr 30 03:39:31 einstein daemon.info connmand[904]: ntp: time slew +0.039397 s
Apr 30 04:12:46 einstein daemon.err watchdog[684]: loadavg 11 9 5 is higher than the given threshold 0 8 6!
Apr 30 04:12:47 einstein daemon.err watchdog[684]: repair binary /usr/sbin/store_watchdog_error.sh returned 253 = 'load average too high'
Apr 30 04:12:47 einstein daemon.alert watchdog[684]: shutting down the system because of error 253 = 'load average too high'
Apr 30 04:12:47 einstein daemon.err watchdog[684]: /usr/sbin/sendmail does not exist or is not executable (errno = 2)
Apr 30 04:12:57 einstein syslog.info syslogd exiting
Apr 30 04:15:19 einstein syslog.info syslogd started: BusyBox v1.31.1
Apr 30 04:15:19 einstein user.notice kernel: klogd started: BusyBox v1.31.1 (2024-04-12 20:52:11 UTC)
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] Booting Linux on physical CPU 0x0
Apr 30 04:15:19 einstein user.notice kernel: [    0.000000] Linux version 5.10.109-venus-15 (oe-user@oe-host) (arm-ve-linux-gnueabi-gcc (GCC) 9.5.0, GNU ld (GNU>
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] CPU: ARMv7 Processor [410fc074] revision 4 (ARMv7), cr=10c5387d
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] CPU: div instructions available: patching division code
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] OF: fdt: Machine model: Cerbo GX
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] Memory policy: Data cache writealloc
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] Reserved memory: created CMA memory pool at 0x4a000000, size 96 MiB
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] OF: reserved mem: initialized node default-pool, compatible id shared-dma-pool
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] Zone ranges:
Apr 30 04:15:19 einstein user.info kernel: [    0.000000]   Normal   [mem 0x0000000040000000-0x000000006fffffff]
Apr 30 04:15:19 einstein user.info kernel: [    0.000000]   HighMem  [mem 0x0000000070000000-0x000000007fffffff]
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] Movable zone start for each node
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] Early memory node ranges
Apr 30 04:15:19 einstein user.info kernel: [    0.000000]   node   0: [mem 0x0000000040000000-0x000000007fffffff]
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000007fffffff]
Apr 30 04:15:19 einstein user.debug kernel: [    0.000000] On node 0 totalpages: 262144
Apr 30 04:15:19 einstein user.debug kernel: [    0.000000]   Normal zone: 1536 pages used for memmap
Apr 30 04:15:19 einstein user.debug kernel: [    0.000000]   Normal zone: 0 pages reserved
Apr 30 04:15:19 einstein user.debug kernel: [    0.000000]   Normal zone: 196608 pages, LIFO batch:63
Apr 30 04:15:19 einstein user.debug kernel: [    0.000000]   HighMem zone: 65536 pages, LIFO batch:15
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] psci: probing for conduit method from DT.
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] psci: Using PSCI v0.1 Function IDs from DT
Apr 30 04:15:19 einstein user.info kernel: [    0.000000] percpu: Embedded 15 pages/cpu s31564 r8192 d21684 u61440
Apr 30 04:15:19 einstein user.debug kernel: [    0.000000] pcpu-alloc: s31564 r8192 d21684 u61440 alloc=15*4096

Answer 22 · 2024-05-09T07:10:36Z

semlohnhoj answered · May 09 at 07:10 AM

I'm still getting random reboots, I've tidied up my NodeRed code to reduce load and made sure my ethernet cables on my CAN bus are good which where causing issues with my energy meter, but still the reviews are happening.

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 23 · 2024-05-09T09:19:18Z

ektus answered · May 09 at 09:19 AM

After increasing the watchdog from 6 to 8, reboots became very scarce. Now I've increased even further, and can report an uptime of

09:08:56 up 9 days,  4:54,  load average: 3.24, 3.51, 3.34
09:19:53 up 9 days,  5:05,  load average: 2.67, 2.46, 2.80
09:22:27 up 9 days,  5:07,  load average: 8.39, 4.83, 3.62
09:23:27 up 9 days,  5:08,  load average: 7.10, 5.04, 3.76

Currently it's around 3, but I've seen as low as 2.5 and as high as 7.5 today. Would be nice to log the load average on vrm (just once a minute or even every five minutes) to get an idea of what's happening there.

Currently, my /etc/watchdog.conf looks like this:

log-dir = /var/volatile/log/watchdog
min-memory = 2500
max-load-5 = 10
max-load-15 = 8
repair-binary = /usr/sbin/store_watchdog_error.sh
test-prescaler = 60
retry-timeout = 0

This is a workaround and may have adverse effects on response time as more tasks are allowed in the queue, but the watchdog randomly rebooting the system because 6 is just a tad too low borders on denial of service and is even worse.

4

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

semlohnhoj commented · May 10 at 06:38 PM

I added some code to NodeRed to check the 1, 5 and 15 minute load average values every 10 seconds and record the highest value of each.

So far today they have peaked at: 10.94 : 8.89 : 6.83 !

So no suprise I was getting reboots on the defaults of 6 & 6. The new settings of 10 and 8 are only marginally inside these figures as well.

This is after removing a lot of NodeRed code I had for status pages and minimising the Victron Node reporting intervals to minimise activity.

2 ·

semlohnhoj commented · May 13 at 09:09 AM

Checked this morning and I'm up to 10.94 : 9.46 : 7.81 for peak values!

Based on that 10 and 8 are only just high enough to avoid a reboot.

My system does have a fair amount of stuff connected:

2 x Multi RS Solar

4 x 500A SmartShunt's

5 x SmartSolar MPPT 100/20's

1 x SmartSolar MPPT 150/35

1 x Lynx Shunt

1 x Energy Meter VM-3P75CT

1 x EV Charging Station

2 x Temperature Sensors

I'm going to up the limit for the 5 & 15 minute watchdogs to 12 and 12 as I can see I'm going to get a reboot eventually.

What are the upper limits of these load averages and what qualifies as 'Too High'?

1 ·

semlohnhoj commented · May 20 at 07:39 AM

Been over a week now, no reboots and top values of - 11.39 : 9.53 : 8.04.

On that basis, 10 and 8 would cover it but I'll probably leave it at 12 and 12 unless someone can tell me why that's a bad idea :)

1 ·

semlohnhoj commented · May 09 at 01:34 PM

I've modified my settings from 6 & 6 to 10 & 8 to see if this helps. I'll let you know how mine gets on..

My load averages from uptime are around 3.74, 2.94, 2.92 at the moment.

If I power off my Victron EV charger for a minute and then power it back on then load averages spike upto 6.17, 3.88, 3.25 for a while.

I guess I'm interested in the 5 & 15 minutes averages going over 6 though. I'll keep an eye on them. I'll get NodeRed to log them periodically.

0 ·

Answer 24 · 2024-05-13T16:44:36Z

ektus answered · May 13 at 04:44 PM

I honestly can't tell. As long as it comes down again, it should be okay. In my experience, the limit of 10 would trigger as soon as 11 is reached, 10.9 is still okay. The load averages seem to creep up and down slowly. Maybe try and spread out the tasks in Node-red a bit? Instead of every second, issue some at 0.9s, 1.0s, 1.1s and so forth?

I've got

5x MPPT 150/35

3x Multiplus 2 5000

1x Lynx Shunt

1x Pylontech US 5000 system with 8 batteries

2x Shelly 3 EM power meters

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 25 · 2024-05-13T18:18:59Z

Alex Pescaru answered · May 13 at 06:18 PM

Why is not OK 12? In fact why is not ok 25? Or 50?

The same as not used memory is a wasted memory.

It's about CPU loading. A idle processor is a wasted processor... It's designed to work.

Just my 2 cents...

Alex

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 26 · 2024-06-07T19:53:28Z

ektus answered · Jun 07 at 07:53 PM

The higher it gets, the slower the response times will become, which might be detrimental to the reliability of an ESS installation. But so are reboots.

Also, there may be a point where the values don't recover because the system just lacks the computing power to run the backlog in addition to the current tasks. I've not reached that point yet with my system and it's uptime now sits at 38 days, 15 hours.

Currently realtime VRM is disabled due to temporary Cerbo overload, but the load averages are coming nicely down again (were on 4.91 4.01 3.59, now 3.,16 2.52 2.95). Okay, and realtime is back, too.

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 27 · 2024-06-30T13:47:19Z

ektus answered · Jun 30 at 01:47 PM

Uptime now stands at 61 days, 9 hours. I'm quite happy with the system now.

In https://community.victronenergy.com/questions/299999/faulty-cerbo-gx.html there's an image https://community.victronenergy.com/storage/attachments/86704-cleanshot-2024-06-24-at-112638.jpg in the first answer.

Trying to add a custom widget in VRM, I can select the load averages, but not D-Bus roundtrip times from that diagram. Also, those load averages seem not to work (diagram shows "no data").

Is this supposed to work? I'm currently running firmware v3.31~3

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Answer 28 · 2024-07-06T18:50:02Z

ektus answered · Jul 06 at 06:50 PM

Update on the custom widget: Looking closer, I found the D-Bus property, and it's even working :-)

The load averages are not, though.

1720291701970.png (44.4 KiB)

1

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Guy Stewart (Victron Community Manager) ♦♦ commented · Jul 08 at 05:26 AM

The load averages data is not sent to the VRM by the GX, I have asked the VRM team to remove them from the menu because they are confusing, but it must be a very low priority.

Also remember to enable the Max/Min setting in the cog.

0 ·

Answer 29 · 2024-07-14T10:50:11Z

mvader (Victron Energy) answered · Jul 14 at 10:50 AM

Hey @ektus , thank you for all the effort that you spend on that. We'll be looking at the configuration of the watchdog and if we can/should relax that a little. Hopefully after the summer.

Can you post, in a comment, the values you're now using?

Thank you, Matthijs

9

2 |3000

Attachments: Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

semlohnhoj commented · Jul 14 at 11:13 AM

For info, I get the same reboot issue and I'm recording the 1, 5 and 10 minute load averages and the current highs I'm getting are: 12.44, 9.51 and 7.72 .

On that basis I'd need to use 9 and 7 for the watchdogs but I'm pretty sure I've seen 5 minute value closer to 10.

I currently have the watchdogs set to 12 and 12 for lack of knowing why that's a bad idea and not wanting my system to reboot anymore. Seems to never crash but I do get a lot of moaning in VRM about asking for to much data when I'm in the advanced pages but I'm assuming that's the VRM and nothing to do with the Cerbo?

0 ·

ektus commented · Jul 14 at 12:22 PM

@mvader (Victron Energy) I'm stil running 10 and 8, haven't changed that since April 20th, 2024. But I'm also still on the old v.31~3 Venus OS from back then, which doesn't have the V2 GUI present. Others have reported increased CPU load on more recent 3.34 versions, which might warrant even higher thresholds. Or investigations into the reason for that higher load, for that matter.

My load average is still fluctuating anywhere between 1 and 9, sometimes triggering the "device overload" warning in vrm. But it's never stayed high long enough to trigger a reboot, uptime currently standing at 75 days. The load averages are not being logged in vrm, so I don't have historical data on these. D-Bus round-trip for the past week:

0 ·

1720959649810.png (171.7 KiB)

mvader (Victron Energy) ♦♦ ektus commented · Jul 14 at 01:13 PM

@Semlohnhoj

> I do get a lot of moaning in VRM about asking for to much data when I'm in the advanced pages

Can you post a screenshot?

@both: thanks for the quick replies. That round trip time looks OK-ish to me, though 400ms is on the high side; anyway, we'll know more after really looking into this.

I've noted your comment wrt people complaining about increased CPU load using recent systems. I'm not sure what that would be.

0 ·

semlohnhoj mvader (Victron Energy) ♦♦ commented · Jul 14 at 05:16 PM

I realise this is probably unrelated to the GX issue but happens even if I only pinch and zoom twice.

By the way, I'm running v3.40~38 on the GX and I have 2 tablets around the house showing the guiv2 all the time so that may be adding to the general load as well as all the devices I have connected. The reboots have been happening for months though I couldn't tell you from which build.

1 ·

screenshot-20240714-180626-vrm.jpg (442.4 KiB)

mvader (Victron Energy) ♦♦ mvader (Victron Energy) ♦♦ commented · Jul 14 at 01:15 PM

ps @ektus for me it would be interesting if you could update to v3.40~39, and then compare dbus round trip time. We'll be releasing v3.40 any day now - its a stable version and representative for v3.40 official.

When you looking at the system using the top command, keep an eye out for increased cpu usage by dbus-modbus-client; thats a known issue which was found recently; and will be looked into shortly. That might save 10 - 20% cpu on some systems; most likely some network activity can cause it to consume more.

0 ·

Alex Pescaru mvader (Victron Energy) ♦♦ commented · Jul 14 at 03:48 PM

Probably this:

0 ·

moan.jpg (17.4 KiB)

semlohnhoj mvader (Victron Energy) ♦♦ commented · Jul 15 at 07:31 AM

Update on previous message, the error below I believe is the one related to the GX load issue in VRM:

0 ·

screenshot-20240715-082741-vrm.jpg (474.3 KiB)

mvader (Victron Energy) ♦♦ semlohnhoj commented · Jul 15 at 07:54 AM

Yes:

1) "Realtime connection disabled due to GX device overload" is a message in VRM, but issue on the GX (typically CPU)

2) "Too many requests" is shown when a rate limiter that protects the VRM database kicks in. (and this should not happen for any normal usage! if it does, then we need to look into something)

0 ·

ektus commented · Jul 14 at 02:02 PM

This is a production system, I'm reluctant changing it if it just works. The "increased CPU load" reference was in the thread https://community.victronenergy.com/questions/308702/cerbo-gx-still-rebooting-multiple-times-a-day-even.html

Running two instances of the not officially supported Shelly 3 EM won't help with determining version-related differences either, and those are essential for the operation of my system.

0 ·

question

Cerbo GX random reboots

Related Resources

question details

Modifications Space

Node Red Space

question

Cerbo GX random reboots

Related Resources

question details

Related Questions

Modifications Space

Node Red Space