question

jvergeer avatar image
jvergeer asked

How to store more than 48 hours of backlog data on local storage (or rather upload to VRM)

Hello,

I'm running Venus OS 2.8x on RPi 3B+ (16GB sd card).

Without Internet access, local storage for the last 48 hours works as does 48+ hours on external USB drive. Love the VRM portal! :-)

Is there an elegant way to increase backlog data beyond 48 hours on local storage? There seems to be ample space on the /data file system.

Else perhaps a separate partition could be created on the local storage and mounted as an external "USB" drive. (?) How could this be done?

The main reasons for wanting to only use local storage are:

  • - simplicity, ie. one less component that can give trouble and
  • - power efficiency, ie. the USB stick does consume some extra power


Thanks

johnny

Raspberry Pi
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

5 Answers
nebulight avatar image
nebulight answered ·

I think the Venus OS will natively see a plugged in USB stick to your pi and write the data to that when it can't communicate to VRM. It's been a while since I've had a pi, but I'm pretty sure that's how I setup mine when camping in dead zones.

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jvergeer avatar image jvergeer commented ·

Hi nebulight, yes indeed the logs are written to the USB stick if present, and this works 100%. I'm just checking if the 48 hours limit writing to local storage is a "soft limit". Is there a way to change the default configuration to allow a longer duration of backlog records to be written on the local storage?

0 Likes 0 ·
gainestr avatar image
gainestr answered ·

I too am looking for this solution. The smallest SD card I currently have is 128GB and it seems a shame that I can’t use it to log more than two days of data on it. I full time RV and sometimes have to go weeks and months without good internet connection. Please let this be an option.

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Kevin Windrem avatar image
Kevin Windrem answered ·

For sure the RPI version of Venus OS uses a USB flash drive for VRM storage if one exists.

Without a card plugged in it uses available space on the /data partition. I just looked and my PI running on a 64 GB SD card and it reported over 21 GB available for VRM backup storage. That is WAY more than 48 hours -- more like a year.

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jvergeer avatar image
jvergeer answered ·

Hi Kevin Windrem, I had the opportunity to re-test the offline/backlog storage upload to the VRM portal. With the RPi out of WiFi range more than 48 hours, once it was in range, it uploaded only the most recent 48 hours of stored data, but no more. My /data file system has 4.9G of data.

This behaviour is accurate as per the documentation, but I'm still hoping to find that this is a soft limit and can be extended by some configuration setting.

Else is there a simple way to upload the missing data? Ie. I see a file called vrmlogger-backlog.sqlite3 in the /data/db directory that possibly contains the data not automatically uploaded to VRM?



2 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Kevin Windrem avatar image Kevin Windrem commented ·
There is a way to upload the file. It's in the main menu of VRM (where it lists all installations).

I have had several occasions where the real time data upload to VRM was delayed for several days and it all updated, but it does take a fair amount of time. I don't know what is causing your loss of data older than 48 hours.

0 Likes 0 ·
jvergeer avatar image jvergeer Kevin Windrem commented ·

Hello Kevin Windrem, I presume the file upload option is "Upload GX file" from the VRM main menu. I tried uploading the sqlite3 file from the /data/db directory, but got an error ... Invalid or Insufficient data.

Just for clarity, the system is working exactly as documented. Ie. when using local storage, only the last 48 hours are automatically synced to VRM when the Internet connectivity is re-established after a prolonged outage. I was just hoping that there was a way to extend that 48 hour limit via a code change or parameter setting somewhere.

1 Like 1 ·
iv4n avatar image
iv4n answered ·

I guess there hasn't been a solution to this? With a fair amount of raspberry pi out there running as a GX device/Venus OS, I hope this becomes an option, if it already isn't

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.