question

robertc avatar image
robertc asked

Cerbo GX started rebooting itself after 2 years

After around 2 years my installation started having issue. I have a 3x Multiplus with 3 MPPTs and Pylontech batteries and all of that connected to Cerbo. There are no SW modifications to Cerbo, it's just stock as is. I started experiencing issues where I get a notification about having lost connection to BMS, MPPT etc, and I noticed that the system is offline for few minutes. In short - Cerbo is rebooting. Recently it started behaving sometimes that It will not come back to life and I have to unplug it from power. Reboot time takes around 8-10 minutes which is way too much. I'm on the newest FW 3.33.

I did a factory reset but that did not help, had few restarts a day. Then I did a clean reflash but after that I also have the same problems. What was weird is that after a reflash I started configuring the system via HDMI monitor and keyboard, Cerbo suddenly rebooted for no reason. The boot time is still very long like before. Before the flash I noticed in /var/log/messages few times an error message about Unexpected signal 11 (which should be a segmentation fault according to some unix distributions), but I do not see that in this log anymore after reflash. Restarts appear one or few times a day randomly.

Cerbo is powered from Pylontech batteries, it is not from a series that had issue with 48V unstable power - I checked the S/N. Reboot device when no contact VRM is turned off. CPU usage is low around 15-20%, average load ±!0.3, memory is 300MB used, 700MB free. The only thing that I do is I have a script on my home automation system that reads data from Cerbo Modbus, nothing else in terms or any other communication, no errors on BMS Can, no errors on VE.Can etc.

Can someone point me in any direction to find out whether I need a new unit or? Are then any logs in the system that could help me? Right now I'm leaning towards that, because I just think there'a problem with the board, memory or so. It's more than 2years old so there's no warranty anymore.

cerbo gxerrorconnection issuerebootrestart
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

9 Answers
Guy Stewart (Victron Community Manager) avatar image
Guy Stewart (Victron Community Manager) answered ·

Hi @RobertC

Could you try disabling Modbus (or anything else connecting the GX to the home automation system) ?

I had a look at your GX device and there are large spikes of CPU activity

Unfortunately the average load value misses these, but if you look at the dbus round trip time maximum values you can see them. I can't say for sure if this is the cause, or the effect, but ideally we isolate things until they stop, and hopefully that stops the reboots as well;

cleanshot-2024-06-24-at-112638.jpg

The Victron warranty for the Cerbo GX is 5 years, so you should still be covered if there is actually something wrong with the hardware, but I think we can do some more troubleshooting first.


2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

robertc avatar image
robertc answered ·

I just disabled that. I use it to control heating for the water boiler. However I don't think that it would explain that very long boot time right. We'll see if that changes anything. Those 2 spikes do not seem to reflect the reboots but rather the time I was doing factory reset and the second one for the clean reflash I think.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Guy Stewart (Victron Community Manager) avatar image
Guy Stewart (Victron Community Manager) answered ·

If CPU load has been ruled out as a cause, the next most likely cause in my experience is some issue with the power supply.


It's very important to first fully isolate the power supplies so there is no voltage and no potential for an accidental short circuit, failing to do this will almost certainly cause more issues.


A thorough check of the DC cables to the GX device, removing the DC input plug and inspecting the wires. If there is any doubt, stripping new insulation and reconnecting. Checking the inline fuse and holder, removing the fuse, checking the springs and contact area, and the fuse itself. It's only a very occasional interruption so the fault might not be visible.

Then the connection to the DC bus itself.


The other occasional cause is an issue with a USB accessory, specifically some kind of internal short in the USB cable. If you have any USB connections, disconnect them and see if that resolves it.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

robertc avatar image
robertc answered ·

I had 2 restarts since so Modbus does not seem to be the issue. I will double check the wiring, I might switch temporarily to a spare laptop adapter laying around. I'll report back if that does not help either.

I did not found any errors or anything weird in /var/log/message for these 2 reboots.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

robertc avatar image
robertc answered ·

This is the top command export, does it look ok?

1719382424291.png


1719382424291.png (249.4 KiB)
2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

robertc avatar image
robertc answered ·

I changed the power supply wire for a new one a routed it to 48V DC busbar direcly for 2 days and had few reboots and one total freeze where I had to unplug the power for the cerbo to boot again. Are there any other tests that I can try or do you think it's a problem with the Cerbo? BTW the boot time is still 8-10minutes.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

derrick thomas avatar image
derrick thomas answered ·

With such a long boot time you may have a failing data drive. Reboot the gx, open a terminal and see what happens with the io. (Second line of the top screen)

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

robertc avatar image
robertc answered ·

When I tried reboot command from terminal, the reboot took around 2-3 minutes now, I haven't seen any spike in top in IO so nothing conclusive in there. For last 24 hours I had like few reboots and now to be sure I just tested unplugging it from power and plugging it back and it booted within like 2-3 minutes. It's acting strange, before I had much longer boot times. However random restarts are still happening.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

robertc avatar image
robertc answered ·

I have replaced the Cerbo with the warranty and finally there have not been any issues for weeks.

2 |3000

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Related Resources