question

Jared Bovill avatar image
Jared Bovill asked

How do I run custom python scripts on Raspberry pi running Venus OS?

Speaking as a complete noob at this. I need to be able to run custom python scripts in general. Meaning I want to be able to have custom libraries, modules and scripts running at boot. Can anyone please provide me a detailed step by step explanation to this problem? Specifically how I can run scripts that need gpio access like relays or temp sensors(examples not looking for those specifically). Im not really looking for alternatives to python but I will take what will work. And lastly what do I put in the rc.local file to run scripts because I cannot figure that out for the life of me.

Thanks

python
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

6 Answers
von-baron avatar image
von-baron answered ·

What type of system are you using for the Python scripts, an independent GNU/Linux based system, Intel or ARM such as orange or apple or raspberry Pi ?

For all my Python development I always use systemd as the manager. I have never used rc.local as a way to do this.

Systemd can do everything I need, get programs running on boot and restart if needed. Plenty of online information on its use.

For systemd , you make a unit file (a text config file) that describes the actions.

But reading what you want to do, I would look at using a micro-controller, maybe even an Arduino type board and have it do all the analogue and digital processing. Then it can supply that to your system as a modbus slave, say modbus RTU or TCP.

Using a micro-controller board is more suited and in many ways easier for many situations where you need to deal with specific I/O that is often not so easy with an OS based system. For example, you say you want to read from temperature sensors, what type, analogue or a certain type of digital, and how would you get the GNU/Linux based board to read that (knowing that there are certain Python modules for such, but are quite technical and need a good amount of expertise to use) ?

Using a micro-controller, they are generally C/C++ or in some cases, micro-python.

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Jared Bovill avatar image Jared Bovill commented ·

Hi thanks for your reply. Sorry, I probably should have specified. Im just looking for python scripts to run on boot for a Venus OS system on a raspberry pi 4. rc.local is a file that is run on boot and is not erased by a system update or restart so I can use it to run a command to run any script I want at boot. I want a python script to run along side the Venus OS. I also want to know how to add libraries to the Venus OS directories and I want to know exactly what I need to put in the rc.local file to make a python script run on boot. I have not done a lot with single board computers so im a little confused. I prefer to do work with micro controllers but in a system like this its more cost effective to have a script and Venus OS running on one raspberry pi 4 rather then a raspberry pi 4 running Venus OS and a respberry pi pico running python scripts (while that's always an alternative if all else fails). I have seen people run custom scripts for new menus or having extra temp sensors and I just would like to have the same sort of freedom of customisation but with python.

Thanks

0 Likes 0 ·
von-baron avatar image
von-baron answered ·

Right, I understand better what you have now.

I would still look into using systemd, it is the modern way to handle processes, either root or non-root. You will have good control of the scripts then, check out the online documentation. Systemd is what is run on just about al modern GNU/Linux systems now, from everything to do with startup to managing all system processes, including user processes. It sticks, meaning it is there always, regardless of any form of updates to system or user parts.

If you want to use Python, make sure you use a virtual environment, where any specific modules can live independently from main system modules. That is the method I always use. But, I don't use or run Venus, I don't like it for many reasons, so I build my own system, all Python based.

If you really get into Python, then also look at helpers like Poetry as the package dependency manager, it makes life so much easier and you can just code away.

I am thinking you should really be looking at a micro-controller. How are you expecting to get temperature data from an external sensor into the system ? Do you have good knowledge of the electrical interface connection of such sensors and the electrical interface capabilities of your RPi ?

I'm not able to help much with RPi's, I choose not to use them, though I do have some in the parts box from a past job.

What type of sensor, analogue or digital ? These things are very simple with a micro-controller.

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Jared Bovill avatar image Jared Bovill commented ·

Thanks again. I sort of understand what your getting at, but for me its a lot more simple to run an already built system like Venus OS rather then build my own. I like what your doing building your own python based system, I think that's really cool. But ideally I want to have the Venus OS system running and a python script running in the background. I know a lot more about micro controllers then single board computers like the raspberry pi 4 and I have done a lot of projects with Arduino and raspberry pi pico (its one of my hobbies) but for this it seemed impractical to use one or more micro controllers when I can run several scripts at the same time on one raspberry pi that im already using anyway for a monitoring system such as Venus OS. Its less about what I want to see in the Venus software and on VRM and more about how the software is limiting my use of python and what I can do with the raspberry pi's potential. I've seen it done before with all sorts programming languages (including python) in the same system. I understand the sensors and everything im using but its less about the sensors and more about the system limitations. I just would like to know how to run general scripts (like any python based script) but its proving difficult due to the limited software. I'm sure someone has a fix but has not posted it to a forum because they never had the need to share it.

Thanks again

0 Likes 0 ·
von-baron avatar image
von-baron answered ·

Sounds like you really do wish to use the local user rc file over using the systemd, so I did a quick online search, there is plenty of talk about it. The link that might help you understand some more is this one how-to-enable-etc-rc-local-for-running-commands-on-linux-boot.

You should be able to make a singe Python application that does all that you want, not necessarily lots of separate scripts. Look at using asyncio to essentially multiplex between the different tasks with the one Python application. Again, using asyncio is a preferred method to concurrently handle multiple I/O bound tasks, such as handling requests from gpio.

I'm still curious on the sensors and how you will connect them and access them, are they analogue or digital sensors ? Are they maybe using an I2C or SPI interface, would be good to know ?

You say "more about how the software is limiting my use of python", what software is that ?

In your first post you mention "Meaning I want to be able to have custom libraries, modules and scripts running at boot." Do you mean you would like to have a single Python application set to run at some point in the boot process, and to keep running for ever ? Does this Python application print to screen or save data to a file or database or a network connection, how is the data being stored or saved or displayed ?

Yep, I bet there are many people who have tinkered and worked hard to make programs do what they want and are happy to share, we call it fun :)

1 comment
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

Jared Bovill avatar image Jared Bovill commented ·
Okay, thanks for the link that actually gave me some more info and I think that's sorted. I didn't want to use systemd as Venus OS is very picky about the stuff it runs. for example there is no pip (by default) in Venus OS. and I wanted to run a script for a case I bought but Venus OS didn't like it.

As for the sensors they were just an example. For raspberry pi its easy to find sensors that will work well and the info on setting them up is a lot more wide spread. I know how to write scripts for sensors and I've done a lot with them in the past so for me that's the easy part. But with some sensors they require some sort of library just like you would need to Import a time library into a python script. (* info on that issue below)

When I say software I mean Venus OS. Venus OS is not like the raspberry pi os image were there is a desktop and all that. Its just a terminal. But its more limited then the raspberry pi os lite for example (because that's also a terminal os). Venus OS is limiting my python use due to the library issues im having (more info below*)

If you could give me some more info on the Python singe application that would be appreciated. but im pretty sure its supposed to be a lot more simple than that.

I plan on using a 16x2 or 20x4 character display to display data but that's something that can be incorporated into the python script. I have already found a somewhat informative forum post about that so I can deal with the data.

(*) I dont need as much help writing the script more running it. I had some files I was using to test. Just a simple button and print. but Venus OS was having issues running it because I was trying to use a time and gpio library that it couldn't find or use. I found that odd because especially the time library because its supposed to be installed with python (or that's what I thought, correct me if im wrong). I asked around some of my mates from a ccna course I took last year but they couldn't seem to find an answer so I posted this forum question.

I also find it fun, thus the tinkering I been doing for the past 2 months. But I hit a dead end here. I have 2 raspberry pi's, one running an install of Venus OS that is in my caravan untouched just a plain Venus OS image. And the other one im running Venus OS as a test bench. so I dont care what I do to it, so I dont mind messing it up on this one, rather then the running and functioning one in the van. if something works I can put it in the van. What im trying to do here is research. So its not a huge deal if I can't figure this out but it would really help me as I have some ideas that I would like to try out but I need a proof of concept. So to start I need to figure out python working on Venus OS and that's were I've hit a dead end with the libraries(*).

0 Likes 0 ·
von-baron avatar image
von-baron answered ·

Okay, I am understanding more now, thanks for the write up to explain it better.

Python includes the time module, yes.

I have no real knowledge of the inner workings of VenusOS, and would assume it is all based on a conglomeration of shell scripts and Python, with, maybe a sprinkle of some compiled C/C++, I am not sure at all. I know they use the dbus as a method to communicate between processes. If you look you will find some info from Louisvdw who has developed some Python code for interfacing various BMS to VenusOS via dbus, that might provide you with some insight, by looking at the code on github.

But, if the VenusOS is specifically built it may well not have the standard Python modules that normally exist. Maybe someone else can answer that.

But, if you create a Python virtual environment, then you can bring in any module you want. The virtual environment will have pip, so it can pull in any module.

Using a virtual environment allows you full freedom and not interfere with the Python parts installed that might be used and specific for VenusOS.

Using a virtual environment allows you to easily delete the your project without effecting your main system. It provides very good isolation.

Again, look at using poetry to handle the complete project, from creating the virtual environment to other structures.

For asyncio, look at the Python documentation, spend a few minutes reading it together with some other online tutorials. But to get started, just leave out asyncio until you get a basic application going.

Make sure you are using a current version of python, such as 3.9 or greater.

Sounds like you have had experience with sensors and micro-controllers and that this is not an area that is going to be difficult for you.

After midnight here, sleep time :)

2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

jschz avatar image
jschz answered ·

Hi @Jared Bovill . I think I'm in a similar case to yours. I have a raspberry pi zero 2w running venus v2.90, and I want to run there a python script that adquire the data from the devices connected to the raspberry pi and make a data processing. I have my script written in my pc perfectly, and the problem is that in Venus I can't install the libraries I need for my script, as in Venus I don't have pip and I dont know how to get it. If you could help me I would really appreciate it

6 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

von-baron avatar image von-baron commented ·

@Jschz Did you not try to do an online search ?

Something like "Debian install pip" would give you plenty of reading.

The following is a standard method to install pip if your system does not have it.

sudo apt install python3-pip -y

Then to check the install and version of pip

pip3 --version
0 Likes 0 ·
jschz avatar image jschz von-baron commented ·
Hi @von baron. Thank you for your answer. The problem is that in Venus there isn't "apt" neither, so you can't do what you say.

I've done a online search and I didn't find anything helpful.

0 Likes 0 ·
von-baron avatar image von-baron jschz commented ·

Well, I don't know the VenusOS and what has been taken out.

But if I were tinkering with such a system I might use dpkg to install apt. Again, there is plenty of info online.

You can get apt from the debian site for the correct architecture for your RPi, what ever that is.

Debian site for admin bits and pieces, specifically apt: Debian stable apt

Then you could read point 2 from stackexchange site on how this might be done: Install apt

But, I guess they took apt out for a reason, so you would be on your own here. But the only risk is that you break the OS and need to re-install it. Keep notes of your steps can be helpful to understand later on.

1 Like 1 ·
jschz avatar image jschz von-baron commented ·
Thank you so much, @von baron
0 Likes 0 ·
Jared Bovill avatar image Jared Bovill commented ·
Hey @Jschz. Thanks for your relatability. I talked to a few mates I have from a Cisco networking class I took last year and they seemed to say the same thing. "install pip." But even when I did get pip it didn't seem to help. It might download one library but would fail eventually because the system wasn't designed for pip (I guess) and didn't know what to do with a library. (I think, I dont actually know what the issue was im just guessing here XD) So from what I could see pip didn't help. Then again I never tried to fix it after that. I almost gave up. This forum post was a last ditch effort to get an answer.


(This isn't really a info comment, I guess Im just saying I sympathise with the frustration and I beg of you to please share any info you might come across or even an answer.)

Thanks :)

0 Likes 0 ·
jschz avatar image jschz Jared Bovill commented ·

Hi @Jared Bovill :),

I think the reason could be what it says here: https://github.com/victronenergy/venus/wiki/commandline-introduction

In the part of: Opkg (for something to be opkg-installable, it needs to be pre-built by us and available in the online repository... ) I think that with pip is the same problem.

I will have to find a diferent solution. Thank you ;)

0 Likes 0 ·
Kevin Windrem avatar image
Kevin Windrem answered ·

Venus OS does not have systems but does have a services mechanism. Like systems, these services start at boot and will be restarted if the process terminates.

services can be any code with a wrapper placed in /service. Actually the service is placed in /opt/victronenergy/service due to the read-only root partition. Anything there is copied to /service (an overlay in RAM disk) at boot.

You can look at the stock services in /opt/victronenergy for ideas

There is much about setting up a development environment here:

https://www.victronenergy.com/live/ccgx:root_access

As of Venus OS v2.90, the root partition is mounted read-only but there is a shell script that remounts it read-write so you can add your scripts.

dbus provides access to pretty much all resources managed by Venus OS. Anything outside of it's control would of course need it's own "driver". Much on dbus here:

https://github.com/victronenergy/venus/wiki/dbus

There are libraries to access Victron stuff including dbus for python. Many stock services are python and have copies of the library.

As you probably already know, Venus OS replaces the root partition when updating firmware. This includes /service and /opt/v*/service. So you will need something to reinstall your service after a firmware update. This is where /data/rc.local usually comes into play since everything in /data survives a firmware update.

I have several "packages" that run python code as a service to extend Venus OS capability. You might take a look at these packages to give you some ideas on integrating your code with Venus OS.

SetupHelper includes some shell script utilities to help with installing code and reinstalling after a firmware update. It includes PackageManager which runs as a python service. So this might be a good reference for ideas.

https://github.com/kwindrem/SetupHelper


7 comments
2 |3000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 8 attachments (including images) can be used with a maximum of 190.8 MiB each and 286.6 MiB total.

von-baron avatar image von-baron commented ·
Thank you Kevin, I have learnt quite a bit from what you write here.

I often do see mention of the OS update needing people to then re-install custom or bit not in the OS due to it being overwritten.

It really is good that the decision to initially make it open source and that Victron try their best and are willing to support and have many people write additional code to integrate into the system.

I wonder how it might look in some years to come, quite exceptional I guess.


1 Like 1 ·
pdonner avatar image pdonner commented ·

Thank you so much @Kevin Windrem for the answer you wrote. You are stating a good example for procedures whereby an open, collaborative community of developers should proceed.

We need some kind of a mechanism, which would lower the initial threshold for newcomers entering into the community. The references you mentioned are some of the articles which developers need to be aware of to understand Venus OS architecture. Most of the essential material is already available, but it is scattered all over this space, in Victron manuals and Git. Apart from a reference list, we also need some general articles (like the one you wrote) which bind together the whole.

As @mvader (Victron Energy) has indicated, Victron doesn't have the resources to produce a coherent overall description of the OS. Could we make an effort to assemble that material somehow?

0 Likes 0 ·
jschz avatar image jschz commented ·
Thank you so much, @Kevin Windrem .
0 Likes 0 ·
Jared Bovill avatar image Jared Bovill commented ·

@Kevin Windrem Wow, ok that's a lot of info. Really helpful thank you.

I still have a couple issues:

First, I cannot access the gpio pins. For example: If I wanted to use a simple 5v led that turns on when something happens. In the script I would need to attach the led to a gpio pin and use python to activate the pin when the event is triggered. how would I do this with python on Venus OS PI? I tried a few GPIO libraries like GPIO.RPI (something like that I can't find the code now) all giving me the same error: library cannot be found. I guess my question is what library (import) do I use for gpio? (same issue with simple stuff like the "time" library)

Am I having this issue because the python script is not in /services? If so using Setup helper how do I "download/copy" the files into /services again after an update?


My other issue is where do I put the libraries? Say, for example, I needed a library for a lcd display. where would u put the library? and would I need to "download/copy" the library the same as I would the python script?

( @Jschz did you find a solution?)

Thanks so much for the info so far.

0 Likes 0 ·
Kevin Windrem avatar image Kevin Windrem Jared Bovill commented ·
Venus OS uses many of the GPIO pins and you would not be able to take over them from main app's functions. Venus OS does provide a number of digital inputs and outputs for control through the main app. This is controlled by /etc/venus/gpio_list. Look at my RpiGpioSetup package to get an idea of how to manipulate pins. But once a pin is referenced in gpio_list it will probably not be controllable outside of the main app. ShutdownManager does use a pin for it's own purposes outside of the main app so look at that package also.


Any libraries you use would need to go in the normal place which is overwritten by a firmware update. So you'd need to run installation again after the firmware update. There is no apt in Venus but it does have opkg. I'd try to avoid adding libraries to control the gpio pins as they ming conflict with what Venus OS does.


SetupHelper handles reinstallation. You build a "package" in /data with specific file and directory structure and use the SetupHelper utilities to build a "setup" script that installs, configures and uninstalls the package. SetupHelper then runs a script at boot and looks for packages that need to be reinstalled, e.g., following a firmware update. If so, the package's setup script is run.

For your purposes, a simpler mechanism should be easy to build into /data/rc.local. It amy be sufficient to copy the services control folder from /data to /services at boot time.


You would not want to execute your main python script from /data/rc.local since the OS expects /data/rc.local to return to it's caller in a timely manner.

0 Likes 0 ·
Jared Bovill avatar image Jared Bovill Kevin Windrem commented ·

@Kevin Windrem Thanks once again. I think im getting more of an idea of what I need to do. I have all (relevent) scripts from setuphelper on my Venus OS test image. Been great so far and would like to know more about the "packages" Is there a document or blog on how to set up one of these "packages"? it seems like the perfect thing. (or instructions on how to do the alternative you gave with rc.local)

Regarding where to put the libraries. Any chance you could give me the directory? Im still unsure of where the "normal place" is. (I am still a noob with raspberry pi and Venus OS)

I looked at your code for shutdownmonitor and but was a bit confused at how you used the GPIO pins. what import controlled GPIO? I can do some research if I know what you used. (or if there is any chance you could give me some example code so I can tinker and work up a solution that would be great. (say I wanted a light to power when a button is pressed. how would I do that using the same GPIO method you used.))

I know you do a lot with Venus OS. Your work here is really cool. I really appreciate you taking the time to help me out here. Apologies if im not getting any of this. raspberry pi/single board computers are a huge step up to what I usually work with. That being Micro controllers. (They are much easier in my opinion) So im very confused when stuff doesn't work like it does on a micro controller.

Thanks.

0 Likes 0 ·
Kevin Windrem avatar image Kevin Windrem Jared Bovill commented ·
There is a document in the SetupHelper repo code section that describes how to create a package.

The setup for ShutdownMonitor to use a GPIO pin is done in RpiGpioSetup.

The libraries I mentioned previously were for other things you might add to the system (like GPIO pin access in python). Where these go would be up to what you install so can't say.

0 Likes 0 ·

Related Resources

Additional resources still need to be added for this topic