Help with upgrade.sh - can I preserve my customizations?

[youzer-name]

TLDR: Is there a way to customize the influxdb.conf file or powerwall.yml file so that my customizations won't be overwritten when running upgrade.sh? I think powerwall-extend.yml may handle the yml, but I'm not clear on how that works and I don't see anything similar for the conf file.

Details:
I've been migrating a lot of my services between different Raspberry Pi's so I could upgrade my oldest one from 32 bit to 64 bit OS. I've been running a very customized version of Powerwall Dashboard for a long time, but I took the migration as an opportunity to try to get my basic installation back to the defaults to hopefully make future upgrades easier. I installed a completely standard Powerwall Dashboard using the install scripts and put all my customized dashboards and data connections in a separate Grafana instance and that seems to be working well.

I did have to make some customizations during the install, and when I ran upgrade.sh for the first time yesterday I found out that my changes were not preserved. I'm hoping someone more familiar with the scripts can tell me if there is a way I can set things up where I won't have to make manual changes after each upgrade.

The two things I ran into involved powerwall.yml and influxdb.conf:

1. I have a unique setup where I have two solar arrays, one an older Solar City/Tesla array with no local access to the inverter data and a second one with a SolarEdge inverter. My Powerwall gateway sees the total production, so I pull the individual production data from SolarEdge and Tesla via the cloud. That means I'm running both the local pypowerwall-based data feed and the "solar-only" web-based feed. After the initial installation I changed the tesla-history section of the powerwall.yml file to

        profiles:
            - default

instead of the original setting where tesla-history is tagged with the solar-only profile.

Reading through the forum, I think I can preserve this customization by creating a powerwall-extend.yml with that change, but that means that going forward I'll have to watch for changes to the default powerwall.yml and copy them into my extended file. Can anyone (tagging @BJReplay) confirm if that's how that works?

2. I had changed my influxdb.conf to turn off statistics logging (I opened an issue to see if this should be the project default: see Default influxdb.conf caused high CPU usage #407). The upgrade removed that customization from the conf file. I don't see anything similar to powerwall-extend.yml that would apply to influxdb.conf. Is there a similar process? If the project default ends up being changed (per issue Default influxdb.conf caused high CPU usage #407) then this becomes moot for me.

It wouldn't be too hard to just make a note to myself to make these changes manually after each upgrade, but I'd prefer to have them preserved if that's possible.

p.s. - I'm very happy I discovered that the DD-WRT firmware on my router can apply multiple DNS names per IP. I have three Pi docker hosts and a Windows PC with interconnected services. When moving a service from one machine to the other I can just update the router to send mysql.local, influxdb.local, mqtt.local, ntp.local, nodered.local etc to the new IP and everything just keeps working. If I had to go through and change the IP in every reference to a service when moving them I think I'd end up leaving everything in place until something broke and forced my hand. Once I went through and replaced IPs with DNS names everywhere, everything became portable.

[mcbirse]

I'm not sure how this can be done with influxdb.conf and will need to investigate this further.

But, with your Docker Compose containers you can modify only certain elements of the existing services by using the powerwall.extend.yml file. You don't have to redefine the entire service.

i.e. create a powerwall.extend.yml with the below content:

version: "3.5"

services:
    tesla-history:
        profiles:
            - default

And this would just override the "profiles" attribute of the tesla-history container. All other values would remain the same, including the container version etc. from the main powerwall.yml service definitions.

This way when there are upgrades to powerwall.yml your containers would still upgrade, but also keep your specific customisations/overrride values.

Another use could be pinning the version of a container so that it does not upgrade with the main project/changes.

e.g. creating powerwall.extend.yml with the below would pin pypowerwall to "0.6.2t28" and it would not upgrade even if the main service was changed:

version: "3.5"

services:
    pypowerwall:
        image: jasonacox/pypowerwall:0.6.2t28

Be careful with certain attributes however that contain lists, like ports, as it does not always "override" the value but instead will add your values to the list.

For InfluxDB config however, I will have a look into a way we could do something similar with user specific config settings.

[mcbirse]

For InfluxDB config however, I will have a look into a way we could do something similar with user specific config settings.

@youzer-name - I had a look into this and the InfluxDB docs mentioned config settings defined as environment variables override the equivalent option in the configuration file.

So I made an update to Powerwall-Dashboard (v3.0.7 or later) and you will now be able to define your custom InfluxDB configuration settings in the influxdb.env file, which will not be affected by upgrades.

# Environment variables for InfluxDB
#------------------------------------------------------------------------------#
# InfluxDB configuration settings can be specified as environment variables
# in the form INFLUXDB_$SECTION_$NAME (or INFLUXDB_$NAME if not in a section).
#
# Environment variables defined here will override current defaults found in
# influxdb.conf. Ensure dashes are replaced with underscores in variable names.
#
# Refer https://docs.influxdata.com/influxdb/v1.8/administration/config/
# for the full list of configuration settings.
#------------------------------------------------------------------------------#

# Uncomment to enable statistics recording, which may be useful to diagnose issues (increases CPU usage)
#INFLUXDB_MONITOR_STORE_ENABLED=true

[BJReplay]

Reading through the forum, I think I can preserve this customization by creating a powerwall-extend.yml with that change, but that means that going forward I'll have to watch for changes to the default powerwall.yml and copy them into my extended file. Can anyone (tagging @BJReplay) confirm if that's how that works?

Since you tagged me, even though @mcbirse has already replied, I'll add some commentary - the ./upgrade.sh and ./compose-dash.sh scripts both spot the existence of powerwall.extend.yml and handle its existence nicely.

So, yes, you can put your customisations in powerwall.extend.yml and even spin up additional services (such as, for example, a proxy). Those services can depend on services defined in the main powerwall.yml and it all Just Works™ - for example, I've got an ecowitt weather service which depends on influxdb defined in the main powerwall.yml

[youzer-name]

It looks like Jason agrees we should change the default settings in InfluxDB, so that issue is resolved. Thank you both for the explanation of how to use powerwall-extend.yml.