question

mvader (Victron Energy) avatar image
mvader (Victron Energy) asked

SOLVED - VRM Portal performance issues 2023-06-01

UPDATE 2023-06-05: issues are solved, see my comment of today below for details.

Good morning!

Or not: we are still having issues with the VRM Portal database.


Current status in detail (updated at 2023-06-05)

Everything works:

  • Ingress of data works
  • Real time data works
  • Seeing last received data works
  • Energy graphs (kWh, on the bottom of dashboard page) work
  • Graphs, both on dashboard and advanced page work
  • Certain fields in reports work.
  • Dashboard Controls work (On/Off settings and others).
  • Remote access into Remote Console, Node-RED & SignalK works.


Issues fixed:

  • Remote firmware & Remote VEConfigure issues. Fixed per 2023-06-01. Details in below comment.
  • Being logged out and having to re-login in frequently. Fixed per 2023-06-02.
  • Frequent overloads of database has been fixed per 2023-06-05.




Best regards, Matthijs


Ps. To add, there is no risk, and there has been no risk, of any data thats transmitted by the Victron installations, or uploaded manually, to get lost. There is a queue system in place for that; which is very efficient, simple, lightweight and high in its capacity. Therefor, even in situations where the database systems can’t keep up with regards to “ingress”, then still data will never be lost.


VRM
4 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·

Good day all, a short update:

- database cluster is being expanded now; which will improve the situation.

- past two or three weeks, seeing Remote firmware update- or Remote VEConfigure- progress could sometimes fail. Which didn’t leave the system in a bad state: any change to the system will always only be started after all required data is received by the GX Device, and thus no longer relying on the internet connection to execute the command. But even so, still annoying and worrying to the user. This issue has been solved per this today.

- past one or two weeks, VRM frequently logged you out and then required to login again. Which is annoying especially for users doing the right thing and having 2FA enabled. After a long search, the cause for that issue has been found this morning. Which was really good news. And its either fully or mostly solved by now. I’ll confirm the full status tomorrow. And note that even if fully solved, it might be that it asks to login again one last time.


And I’ve updated the detailed status list above.

6 Likes 6 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·
Good day all,


Finally: I have good news to share, we're in solid green now and all features have been re-enabled.

After various optimisations and investigation in the past few weeks, this weekend was spent upgrading our database cluster to have more nodes and CPUs. This was completed successfully.

We might have some related (but then I expect much shorter) hick-up in the near future; but for now nothing of the sorts like recently.

So, again my apologies for all recent issues - this is not at all how I, or anyone else at Victron for that matter, envisions VRM.

And: thank you for all encouragements, offers to help out and similar that I've received here in this and the previous thread, as well as via various other means. Really encouraging and it for sure helped the admins in their late hours to stay motivated and focussed to asap find a solution + until then minimise the impact to you, our customer.

All the best, Matthijs

5 Likes 5 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·

Hi all,


short update:
(1) we're expecting to have less issues after today

(2) after having fixed most of it yesterday, we found one more issue today and have 100% fixed the issues having to relogin all the time.

2 Likes 2 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·
Good day all, short update; with one rather unfortunate mistake..:


1) we have re-enabled the auto-refresh of data every minute, for users that have the advanced tab, and/or the dashboard open. This feature was disabled quite early on in the process over the issues.

2) just now, accidentally, all users were logged out again: everyone will have to login, again. I'm a bit speechless about that one, but please be assured that although with very unfortunate timing, its a one time human error, not a bug or similar, and the devs will also look into a way now to prevent that from happening ever again.

2 Likes 2 ·
17 Answers
cathal-h avatar image
cathal-h answered ·

Thank you for this information and keeping us up to date.

Best of luck Cathal

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

sharpener avatar image
sharpener answered ·

Fortunately the remote Mode Select facility still works, I have successfully switched my installation to Inverter Only in anticipation of a visit from the DNO to upgrade the intake fuse.

ATB @mvader (Victron Energy)

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·
Good to hear! Yes those dashboard controls are not affected. I added the status of that to the list, good one - thanks
0 Likes 0 ·
victron-blog avatar image
victron-blog answered ·

Thanks for the confirmation. I have seen the pop-ups on the apps.

Good luck in getting it sorted quickly.

Craig - New User

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

snowii avatar image
snowii answered ·

Hello,

could something please be done with the logging out of all the VRM sessions (browser, apps etc.)

It`s understandable there might be issues and some information doesn`t load etc. but having not to log in everytime would be great.

Thank you!

6 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ralf-ct avatar image ralf-ct commented ·
Saving the email and password should work?
0 Likes 0 ·
iammotorhomeless avatar image iammotorhomeless ralf-ct commented ·

not for me :(

I am using FireFox and I click the box that says. "keep me logged in" (or what ever it says)

but it never keeps me logged in
and the saved email and password only work after I have clicked the 'email box' and selected my saved email address

why does it not just auto-populate and log me in ??


1 Like 1 ·
markess avatar image markess iammotorhomeless commented ·
Yes VRM logs me out constantly now. And with 2FA on it takes ages to get back in
1 Like 1 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ markess commented ·
Yes its extremely annoying. But, mostly or maybe even fully solved now
1 Like 1 ·
ralf-ct avatar image ralf-ct iammotorhomeless commented ·
Under Privacy & Security delete the Cookies and Site Data for VRM, also check if any password entries are present for VRM, delete those as well. Subsequent logins should then be saved, that has been my experience on other sites.
0 Likes 0 ·
natteverf avatar image natteverf iammotorhomeless commented ·

I recommend using KeepassXC password manager, there is a firefox add-in that integrates it, you can securely storte your passwords and it will enable you to save website logins with one click and to use them when required also with one click, it's way better than relying on Firefox to store passwords. If you store the dayabase on OneDrive or DropBox or similar then it will remain if your hard drive crashes also or you get a new computer.

0 Likes 0 ·
larsea-dk avatar image
larsea-dk answered ·

What do you all do of VRM can not be reached?? I do run NodeRed, but also this must be started from VRM if the tunnel connection closes…which it does for me several times a day.

7 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

iammotorhomeless avatar image iammotorhomeless commented ·

I am new to node red... but are on your a local lan network?

https://venus.local:1881/


0 Likes 0 ·
markess avatar image markess iammotorhomeless commented ·
Yes how to access the Node-RED dashboard without using the VRM relay.... any ideas?
0 Likes 0 ·
markess avatar image markess markess commented ·
Router/firewall/port forwarding fun no doubt
0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ markess commented ·

this looks like a promising solution for accessing things remotely, without relying on VRM and also without doing port forwarding / opening a port in your router firewall (please don’t do that) to me:

https://community.victronenergy.com/questions/205323/venus-os-ngrok.html

1 Like 1 ·
markess avatar image markess mvader (Victron Energy) ♦♦ commented ·
Interesting, thanks!
1 Like 1 ·
sharpener avatar image sharpener mvader (Victron Energy) ♦♦ commented ·
But doesn't this mean your traffic goes via a ngrok server instead of Victron's?


What is wrong with port forwarding through your own router which is what I would prefer to do, and is there some guidance on which ports are required? I only want the Remote Console and this works fine over my LAN already.


0 Likes 0 ·
daza avatar image daza sharpener commented ·
People love to send bots out these days to ping ports, the more ports you have open is opening you up to be hacked, if you haven’t turn off ping on the router that sends an automated response once an open port has been tickled letting them know what port number can lead to only a couple of things using that port. You are far better with a range and then picking something in that range
0 Likes 0 ·
cramik avatar image
cramik answered ·

Strange message "Warning Unauthorized" received when I accessed VRM portal as logged in. Is there any intrusion in VRM servers? Is there any risk for connected Victron devices to be infected or controlled by malicious attacks? In case of doubt, I disconnected my installation physically from the outside world, waiting for better times. Losing a bit of statistics is not a big deal. Thanks for your efforts, up to now VRM portal was a great tool.

2 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·
Hi @cramik , no need to worry about that. There are no signs of anything like that; its an overload that happens during a (normally weekly) background process in the database; which took a while to find but were onto it now; I’m hoping/expecting to get to a solution soon.


Can you share a screenshot of the warning you are seeing?

0 Likes 0 ·
cramik avatar image cramik mvader (Victron Energy) ♦♦ commented ·
Unfortunately (for reporting), the message does not appear any more this morning. There is only the blue box on top and it looks like it works fine.
0 Likes 0 ·
markess avatar image
markess answered ·

VRM now just gives me a blue box saying 'Add Installation'. When I'm not on-site I need a way to access GX/Node-RED without VRM...

9 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·
Hi, that should not be happening; and I dont see a relation to the performance issue.


I’ll ask support to look into that for you tomorrow.

0 Likes 0 ·
markess avatar image markess mvader (Victron Energy) ♦♦ commented ·
I deleted the VRM app and reinstalled it and it works again, although comes and goes and still logs me out daily. Don't fancy trying to access Cerbo GX from WAN through my router but may take a look to remove dependence on VRM currently
0 Likes 0 ·
janieronen avatar image janieronen commented ·

Some other users have same issue that cannot contact site trough VRM-portal. Do you have VPN-tunnel to installation site or other safe way to connect your local network? I usually propose customer have safe tunnel to installation site when mission critical systems need to have access all the time.screenshot-2023-06-02-at-05347.png

0 Likes 0 ·
screenshot-2023-06-02-at-05347.png (9.4 KiB)
markess avatar image markess janieronen commented ·
No, I don't have a way to connect when not on-site
0 Likes 0 ·
Show more comments
Sarah avatar image
Sarah answered ·

Had to log in again when starting VRM but able to close and reopen VRM without logging in so this appears to have been fixed. Much appreciated.

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ralf-ct avatar image
ralf-ct answered ·

Only one of my graphs under 'Advanced' load, and that takes quite long, no 'loading' message.

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

David avatar image
David answered ·

Thank you for keeping us updated, but even more thanks for understanding the importance of such an ecosystem. I liken VRM to Apple's ecosystem - It works across all their devices, is accessible across generations of hardware and makes using any of their products seamless and relatively easy.

The VRM brings together all your products - Presents vital data to the user, wherever they are and continues to ADD VALUE to the products long after the sale has completed.

Its excellent - and congrats for recognising the importance of it to your business :-)

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Ed Jones avatar image Ed Jones commented ·
They are a victim of their own success :-). That's a good problem to have, but still a problem. Hopefully they are engineering a solution that can keep up with very high growth and not require a battalion of software engineers. If they haven't already, it's time to enter discussions with AWS, Azure, etc.


I left the world of IT two years ago, gratefully!, but it seems I can't totally escape it :-)

1 Like 1 ·
ralf-ct avatar image
ralf-ct answered ·

I'm unable to select anything but 'Today' under Economic reporting. Is that part of the current "performance issue", or am I missing something somewhere?48v-3000-multiplus-ii-vrm-portal-2023-06-02-175650.png


48v-3000-multiplus-ii-vrm-portal-2023-06-02-175650.png (138.6 KiB)
1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

ralf-ct avatar image ralf-ct commented ·
This seems to have been fixed. Thank you.
0 Likes 0 ·
Peter avatar image
Peter answered ·

From 1:01 2023-06-03 (CET) my program GbbVictronWeb (5.9.165.39) receive two errors: "Bad gataway" or "Too many requests. Please try again in a few minutes". Is that part of the current "performance issue", or something else? Did you setup any limits to connect to VRM? Or maybe my ip is blocked (why?)?

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

-ed avatar image
-ed answered ·

I haven't seen the blue SOC% line on VRM Dasboard's System Overview graph for a few days now. I've used a browser and the VRM app for iPhone but It's in neither. Iv'e tried selecting different time ranges, refreshed the bowser, deleted the VRM app and re-installed it from the App Store, but the blue SOC% line is still missing. However, I can see the Advanced graphs that show SOC%, and I can also see the Battery % in the top schematic visualization of the VRM Dashboard. It's only the System Overview graph that doesn't show it. The blue dot in the legend below the System Overview graph, which is normally used to indicate the line, is also missing.

4 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

multivictor avatar image multivictor commented ·
Same problem here since a day or two.
0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ multivictor commented ·
Hi both, thank you for the report. That that doesn’t work is on purpose. The db team has been doing upgrades all weekend long. And tomorrow morning is D-Day to re-enable the now disabled features. Everyone here has fingers crossed wrt it all being smooth then.
4 Likes 4 ·
multivictor avatar image multivictor mvader (Victron Energy) ♦♦ commented ·

Thx,

Since an hour ago everything on VRM is back to normal.
The battery graphs can also be viewed.

1 Like 1 ·
-ed avatar image -ed mvader (Victron Energy) ♦♦ commented ·
Awesome, thank you and good luck.
0 Likes 0 ·
vvv avatar image
vvv answered ·

Hi!
It's probably a bug. can it be fixed?
Thanks

capture.png


capture.png (79.5 KiB)
1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·
Yes, noted and will be looked into; thank you.
1 Like 1 ·
redrum06 avatar image
redrum06 answered ·

Hi. I seem to have 1 site, still no back online. Probably a dumb question, but I should go and do a firmware update screenshot-20230606-073102-vrm.jpg


screenshot-20230606-073102-vrm.jpg (319.9 KiB)
3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ commented ·

Hi Redrum, that looks like a connectivity issue, please see here for troubleshooting that:


https://www.victronenergy.com/media/pg/Cerbo_GX/en/vrm-portal.html#UUID-9d000e4f-dfd4-e0a1-ec69-a66278548d67


Especially step 3.


0 Likes 0 ·
kevgermany avatar image kevgermany ♦♦ commented ·
Mine did the same, reboot of the CCGX did the trick. I think it was caused by a failed update to Venus OS. The update is still failing, but the instance is online. I'll update manually tomorrow.
0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ kevgermany ♦♦ commented ·
I wonder how that could happen Kevin, updating Venus OS has nothing to do with the VRM Portal connections, other than making the ethernet/wifi link busy.


anyway: @both: just be aware that we have an optional auto-reboot function, which will reboot the GX is no data has been sent successfully to VRM for a while.

If you want 100% reliable VRM connection, then I recommend to enable that.

0 Likes 0 ·
pwfarnell avatar image
pwfarnell answered ·

Since the issues have been resolved, I have tried to download my data in Excel format for the month of May as I keep monthly files. I can make a short timescale download and the files is downloaded immediately. If I ask for the full month it gives the message that this will take longer and a link will be emailed. This is all as per normal from prior to the late May / June VRM issues. However, the email with the link to obtain the large Excel file just does not turn up. Checked the spam folder, not there.

If I have asked for the large monthly download and then immediately ask for a download of the previou8s 24 hours data I get a warning that states "There is already a data export running for this installation, please wait for it to complete" so the system appears to be doing the large download, just not sending an email.

Any ideas if this remains an issue to be solved.

3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ commented ·

Hi @pwfarnell

We have just pushed a fix for this issue.

It should now be possible to download long range data again.

1 Like 1 ·
pwfarnell avatar image pwfarnell commented ·
I have been able to do this 1 week at a time, perhaps the amount of data recovery was too much a month at a time.
0 Likes 0 ·
mvader (Victron Energy) avatar image mvader (Victron Energy) ♦♦ pwfarnell commented ·
Hi @pwfarnell , I'll check with the developers; thank you.
0 Likes 0 ·
ralf-ct avatar image
ralf-ct answered ·

I'm not sure if the following is linked to the current VRM problems, I only started noticing it today:

When viewing a graph maximized under 'Advanced', it will minimize on its own after a minute or so.

3 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Guy Stewart (Victron Community Manager) avatar image Guy Stewart (Victron Community Manager) ♦♦ commented ·
Thanks for reporting.

I can reproduce it, and will pass it on to the VRM team.

2 Likes 2 ·
cacer avatar image cacer commented ·
yes. i can see the same behavior.

after some seconds to a minute, it will minimize itself.

1 Like 1 ·
ralf-ct avatar image ralf-ct cacer commented ·
Since you're also experiencing it, it can't be any setting on my side.
0 Likes 0 ·