question

hotel-lima avatar image
hotel-lima asked

After Cerbo GX update to V3.31 I now get Error #67 BMS connection lost

Hello all,

I have a system with 8 Pylontech US3000C connected via BMS CAN to the Cerbo GX. It worked without issues for over 1 year now. After Updating the Cerbo GX to firmware version v3.31 I already saw the Error #67 BMS connection lost now for the second time within a few days. The issue happened early in the morning both times. The Multiplus 2 went to Pass Thru / Low Battery voltage and the solar chargers stopped and did not start to charge when the sun came out (float mode).

When I noticed the system to be in this state I wanted to check if a new firmware for the Cerbo GX was available and I clicked on "Check for updates" in the menu "Online Updates". It looked like the Cerbo could not contact the update server as it said "checking..." for quite a while. Normally it returns the answer within 1-2 seconds. Not sure if it was to do something with the Error #67, but it was the same both times I saw the issue. I then quit the menu and restarted the Cerbo GX.

When the cerbo GX started up it again beeped showhing "Error #67" and "Multiplus-II Low battery voltage" but then after some seconds the system started to work normally.

I have the strong feeling that it could be a software issue in the Cerbo GX firmware v3.31 (especially as it seems to have issues to contact the update server when it is in the error state). I now went back to v3.30 to check if the issue happens again.

Please let me know if anybody experienced the same issue when starting to use v3.31.

Thanks and regards

Martin

cerbo gxVenus OSBMSfirmware updateerror
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

6 Answers
mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

Hi again,

First of all please re-check the previous message; I amended it.

Then I had a brief look, and its not really clear what causes the issues. But I did find one problem, with the mod (....), that needs to be fixed.

The problem is that your data partition is full.

And the cause is a log in dbus-opendtu. I'll file an issue there for you.


To fix this, log in and remove that file.

root@einstein:/data/dbus-opendtu# ls -al
drwxrwxrwx    8 root     root          4096 May  3 13:08 .
drwxr-xr-x   15 root     root          4096 Apr  5 20:21 ..
drwxrwxrwx    4 root     root          4096 May 10  2023 .github
-rw-r--r--    1 root     root          1816 May 10  2023 .gitignore
-rw-r--r--    1 root     root         16522 May 10  2023 .pylintrc
drwxrwxrwx    2 root     root          4096 May 10  2023 .vscode
-rw-r--r--    1 root     root         13737 May 10  2023 README.md
drwxr-xr-x    2 root     root          4096 May 10  2023 __pycache__
-rw-r--r--    1 root     root          3722 Jun  9  2023 config.ini
-rw-r--r--    1 root     root           113 May 10  2023 constants.py
-rw-r--r--    1 root     root     1134657536 May 25 14:50 current.log
-rw-r--r--    1 root     root          4969 May 10  2023 dbus-opendtu.py
-rw-r--r--    1 root     root         28045 May 10  2023 dbus_service.py
drwxrwxrwx    2 root     root          4096 May 10  2023 docs
-rw-r--r--    1 root     root          3399 May 10  2023 helpers.py
drwxrwxrwx    2 root     root          4096 May 10  2023 img
-rwxr-xr-x    1 root     root           777 May 10  2023 install.sh
-rwxr--r--    1 root     root           175 May 10  2023 restart.sh
drwxrwxrwx    3 root     root          4096 May 20  2023 service
-rw-r--r--    1 root     root          4464 May 10  2023 tests.py
-rwxr--r--    1 root     root           255 May 10  2023 uninstall.sh
root@einstein:/data/dbus-opendtu#



1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hotel-lima avatar image hotel-lima commented ·

Thank you so much for pointing me to that issue. I will keep an eye on that log file. I removed and re-installed the opendtu service and now it looks like it does not generate error messages any more.

Thanks for your continued support, I really appreciate it. It was the same with the RS shade performance topic. Great support from your side also there and the new firmware works perfectly now.

Thanks again and have a nice weekend!

Regards,

Martin

3 Likes 3 ·
hotel-lima avatar image
hotel-lima answered ·

What I can say now is that is does not have to do anything with v3.31 as it happened also 2 times after downgrading to v3.30.

I have some more observations:

- looks like the Multiplus II in the VE Bus at first loose the BMS connection (happened some minutes after 0:00 AM in all cases.). VE.BUS System BMS lost: Alarm

- In the morning when sun comes out the solar charges then report: #67 - No BMS

- When the issue happens the ESS assistant has vanished in the Cerbo Menu.

- No new software check works any more. It stays stuck at "checking..."

- Connection remote console via VRM not working any more in most cases

- One time I was able to remotely reboot the Cerbo via the remote console. The cerbo then started to communicate with the VRM again, but the remote console was not reachable afterwards. When I returned at home in the evening the UI on the cerbo display seemed to have crashed during rebooting. I then singed in via SSH and rebooted it.

- Today I checked the CAN interface to the BMS while the issue was present. There were no issues shown. Message counter still were increasing. BMS communication was working. Also checked with dbus-spy. Communication with the BMS itself was working.

- I learnt that restarting VE.Bus via the Multiplus II menu cuts off power for a short moment. Ok, my fault. I was expecting a warning and asking for confirmation before it actually performs an action which causes a power cutoff. Could be a future optimization to add a confirmation.

Until now I only restarted the cerbo to recover the error. But today I completely shut down the system (turn off Battery, AC in and PV) and then restarted it to reset a potential wrong state in any of the devices.

What could this be?
Any hints helping to find out the root cause of this issue are highly appreciated!

Regards
Martin

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hotel-lima avatar image
hotel-lima answered ·

The issue (BMS lost) happened again about 1 hour ago. It was again some minutes after 00:00 AM. I'm pretty sure it is something on the Cerbo. Is there something running at that time? Maybe firmware version update check? A manual check via menu was not working, as before when the error occurred. It was stuck at "checking....". Also some menu points could not be selected when I went to through the menu. The start page of the cerbo also only showed empty boxes. ESS menu was empty.

Also I tried to delete my WIFI connection as I prefer to use Ethernet. If I select forget WIFI connection it disconnects, but after some minutes the connection is established again. This issue seems to be persistent, means it also does not work to forget a WIFI connection if there is no other issue.

I now disabled the firmware version check and Fronius automatic search (I anyway do not use something from Fronius). Maybe this automatic checks/searches can block something. Was there maybe something added at VRM which might lead to blocks in the Venus OS?

Is there anything I can check in the logs via SSH when the issue occurs?

Any hints are appreciated.

Regards,

Martin

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Alexandra avatar image
Alexandra answered ·

@hotel-lima

Is the inverter also up to date? First start with all components updated.

Then reprogram.

Change which pylontec is the master battery.

If all else fails with the Fronius connect it directly to the GX. Steps for that are in the manual this will eliminate comms issues through the network there.

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hotel-lima avatar image hotel-lima commented ·

Yes, the Multiplus are up to date (Firmware and ESS assistant) since a while already.
Regarding the Fronius there is a misunderstanding. I don't use any Fronius components. I just disabled the search for it to possibly reduce some CPU load from the cerbo.

The rest of your hints I will keep in mind.
Thank you.

0 Likes 0 ·
xmorpheus avatar image
xmorpheus answered ·

Hi,

i think i have exactly the same issue since 2 days.
Right now I am waiting for the issue to happen again so i can check if it relates to some of my modifications running on the cerbo e.g. opendtu <-> dbus interface.

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hotel-lima avatar image hotel-lima commented ·

I also use an opendtu to connect my Hoymiles inverters. I've installed henne49/dbus-opendtu on the cerbo. But this is running quite some while and the last update has been several weeks ago....

0 Likes 0 ·
mvader (Victron Energy) avatar image
mvader (Victron Energy) answered ·

hey @hotel-lima , if you enable remote support on your system, I can login and have a quick look.

I already found it the system in VRM, its called Home, right?

2 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

hotel-lima avatar image hotel-lima commented ·

Hi @mvader (Victron Energy).
Thanks for your answer. I have enabled remote support right now. System is called Home, your're right.

0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ hotel-lima commented ·

Hi, I’ll have a look when I have a moment.


Note that most likely its something to do with the mods you’re running; other wise we’d have far more complaints

Correction: those screenshots were from a different site - sorry! Load is no issue on your system, see the graph that I added to your advanced page.


Matthijs

Ps. A next time please immediately report that you’ve mods running - thanks!

3 Likes 3 ·