Weather History Import Tool #146
mcbirse
started this conversation in
Show and tell
Replies: 2 comments
-
|
Thanks @mcbirse ! This has been merged and is now available for use: https://github.com/jasonacox/Powerwall-Dashboard/tree/main/tools/weather-history |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I had been thinking for a while to write another script, similar to my Tesla History Import Tool, but for importing historical weather data from OpenWeatherMap. Actually had briefly investigated this months ago and mentioned it here during testing of the Tesla History script. 😊
Having finally had some free time recently to work on this, I believe it is ready for release/testing for anyone interested.
The script has been developed to replicate the data logged by Weather411, and similar to the Tesla History tool can be used to fill missing data / data gaps, for instance if Powerwall-Dashboard or the Weather411 container stopped working, or your internet connection was down for a while.
It could also be useful to use this script as a base to adapt or modify it for other API's as well, such as those provided by local weather stations perhaps (as discussed recently in #145).
Note: To use the script it does require you to sign up to the OpenWeatherMap "One Call by Call" plan, which is a paid subscription - however they allow for up to 1,000 API calls per day for free, so it is still possible to use for free if you wish.
Below is the README with further details and usage instructions that I will submit with the PR to add to the tools list.
If you find any issues or problems please let me know, or even just some feedback if you find this useful!
I wasn't sure how much interest there may be for this... but I have a feeling being a nerd for weather data seems to go hand in hand with our thirst for Powerwall + solar data & statistics! 😉
Weather History Import Tool
This is a command line tool to retrieve weather history data from OpenWeatherMap One Call API 3.0 and import into InfluxDB of Powerwall-Dashboard.
Features
The script can be used to import historical weather data by date/time period into Powerwall-Dashboard.
This could be useful for instance to import data from before you started using Powerwall-Dashboard or the Weather411 container. It can also be used to fill in missing data (data gaps) for periods where Powerwall-Dashboard or Weather411 stopped working (e.g. system down, or lost communication with OpenWeatherMap).
OpenWeatherMap Subscription
Historical weather data from OpenWeatherMap is available by subscribing to the "One Call by Call" plan. This is a paid subscription plan, however allows for up to 1,000 API calls per day for free.
To use this tool, you must subscribe to the "One Call by Call" plan, which can be done here: https://openweathermap.org/api
If you wish to use the service for free, or control your spend amount, make sure to edit your "calls per day" limit before use: https://home.openweathermap.org/subscriptions (set to 1,000 for free use)
Once subscribed to "One Call by Call", the same API key as used by Weather411 can be used with this tool (note, plan activation may take some time).
Data Observations and Notes
OpenWeatherMap provides historical weather data for 40+ years back.
This is generally available in 1-hour steps, however testing has shown requests for recent historical data (e.g. last few weeks/months) may be available in 10-30 minute intervals. Also, more recent historical data may return field values that are otherwise not available in the older historical data (e.g. wind gusts, visibility).
You can set the frequency with which to retrieve historical data during the script configuration. This value is also used as the "gap" threshold when searching InfluxDB for data gaps (NOTE: shorter intervals will increase the number of API calls, and reduce the range of historical data you can retrieve per day, and may be chargeable depending on your "calls per day" limit setting).
If your "calls per day" limit is reached, you may receive a "service usage alert" e-mail from OpenWeatherMap advising you have exceeded your account limit. The tool will continue to retrieve data for some time after this e-mail before the OpenWeatherMap API service catches up and responds with the account limit reached status code (this will be detected, at which point the retrieved data will be written to InfluxDB and the script will exit).
Although the number of API calls will have exceeded your "calls per day" limit - DON'T PANIC - OpenWeatherMap appears to cap the number of recorded API calls to your limit setting regardless, so you will never be charged for more than the limit you have set.
Finally, the data imported by this tool will not overwrite existing data. It will add data to the same field names as Weather411 which will update the following panels: "Energy Usage", "Temperature, Humidity and Clouds" and "Wind, Pressure and Precipitation".
Usage
To use the script:
Setup
On first use, it is recommended to use the
--setupoption. This will create the config file without retrieving history data.# Run setup to create or update configuration python3 weather-history.py --setupInitial configuration defaults will be sourced from the Weather411 config file, when found. If the location of this file is different, it can be specified with the
--w411confoption.Setup will then run in an interactive mode. The example below shows the config creation:
In most cases, the
[default]values will be correct and can be accepted by pressing Enter, however these can be changed if you have a custom setup.Saved config values can also be changed at any time by running with the
--setupoption again.Once the configuration is complete, you can then run the script to retrieve historical weather data.
Basic script usage and examples
To import history data from OpenWeatherMap for a given start/end period, use the
--startand--endoptions (date/time range is inclusive and in formatYYYY-MM-DD hh:mm:ss):By default, the script will search InfluxDB for data gaps and fill gaps only, so you do not need to identify time periods where you have missing data, as this will be done automatically. The "gap" threshold is configurable and is the same as the history interval chosen during the setup process.
You may want to run in test mode first for a short time period to ensure your OpenWeatherMap "One Call by Call" subscription is active. This could be done by using the
--testoption:Example output:
If your "calls per day" limit is reached during execution, you may receive a "service usage alert" e-mail from OpenWeatherMap advising you have exceeded your account limit - DON'T PANIC - you will not charged for API calls in excess of your "calls per day" limit setting.
The tool will continue to retrieve data for some time after the usage alert e-mail before the OpenWeatherMap API service catches up and responds with the account limit reached status code, at which point the retrieved data will be written to InfluxDB and the script will exit, as per the example below.
If you prefer, you can abort the script by pressing Ctrl-C and you will be given the option to quit and still write the data retrieved to that point to InfluxDB.
Some convenience date options are also available (e.g. these could be used via cron):
# Convenience date options (both options can be used in a single command if desired) python3 weather-history.py --today python3 weather-history.py --yesterdayFinally, if something goes wrong, the imported data can be removed from InfluxDB with the
--removeoption. Data logged to Powerwall-Dashboard by Weather411 will not be affected - only imported data will be removed:For more usage options, run without arguments or with the
--helpoption:# Show usage options python3 weather-history.py --helpAdvanced option notes
--debugcan be used to enable debug output. This will print the raw responses from OpenWeatherMap which might be helpful in some circumstances.--forceoption can be used to import data regardless of existing data (i.e. the search for data gaps is skipped). This should not be required normally, but could be useful for testing purposes.--removewill remove any previously imported data from InfluxDB for the date/time range, without affecting data logged to Powerwall-Dashboard by Weather411.If you have any questions or find problems with this script please let us know by posting here.
Beta Was this translation helpful? Give feedback.
All reactions