Local API with Protocol Buffers #124
Comments
|
Ok just realized its a floating point number! This calculator can do the conversion. Is there any interest in me posting samples or |
|
That's quite interesting! Would you mind sharing your firmware version numbers? |
|
I'm interested in this. Just got my SE10000H online yesterday. Need to wait two weeks for meter swap out though before it's producing power. I've got the inverter mirrored to another switch port, and an rPi that just does a file a day of tcpdump. If the inverters are changing their encryption keys a few times a day, being able to get all the same data your way would be much simpler! |
|
I don't see anywhere that explicitly says firmware version, but I'm assuming this is that you're looking for? I have probably 80% of the API response mapped which includes
|
|
Yes, that's what I was referring to, thanks! I think that's the first time I see a CPU version number starting with 4. Would be interesting to find out whether it offers more API functions than the one you discovered. More specifically, whether it would offer one that provides optimizer status! If we could get our hands on a recent firmware image we might be able to do some string search on it to find out? |
|
Just clicking around in chrome with Dev tools on, I see several different API endpoints being used, but the only optimizer specific information I see displayed in browser is the number online. I was looking at responses at night when it wasn't producing, so I'll try it again during the day in case the information is just in the API. The weirdest thing about this dashboard is it's totally unauthenticated and appears to allow you to change any setting you want on the inverter, including grid protection, etc. I haven't tried changing any setting since I don't know what I'm doing. I wonder if the installer enabled it by mistake? |
|
I believe that's by design. My installer used a connection this way to do all the setup and provisioning of the inverter. My versions match yours exactly. Fingers crossed this provides more data than the existing public api. |
|
SolarEdge was promoting this capability recently for installers https://marketing.solaredge.com/acton/rif/8801/s-144b-1902/-/l-0045:1133/l-0045/showPreparedMessage?sid=TV2:cfByisFlu. If this interface can provide information about individual optimizers it is very interesting because the lack of that capability was the original motivation for this project. Hopefully it does and can be available on older inverters with a firmware upgrade. |
|
Unfortunately I'm not seeing any details about optimizes (under then number communicating vs paired). Reading through the installer instructions, it has them pair via phone app, then go to the monitoring tool to setup the rest of the optimizer details. That makes me think the local API might not have these details implemented. On the plus side, I realized that if you leave the dev console open, it actually logs out the parsed API responses as you're browsing the dashboard, so it should be very easy to use the APIs. I'll try to publish details of the APIs over the weekend. |
|
I've been digging into this and am pulling out a lot of details. I'll post specifics soon. I've found some optimizer specific information via the maintenance API
Is that useful? I was hoping to see watts / power or something similar. |
|
That's very good. Optimizers don't directly report power, but the watts can be calculated by multiplying the input voltage by the input current. The only other thing that would be useful would be the optimizer temperature. |
|
Oh great! It does have the temperature in the API actually. I'm not sure where you want this information posted, so I created my own repo. Happy to merge into your project if you prefer. https://github.com/drobtravels/solaredge-local I went ahead and mapped what looked to me the most interesting data here |
|
Great job @drobtravels ! Thanks for sharing and for your effort. I made a pull request on your parallel repository with some addition on which inverters (US and European) are supported. |
|
Just FYI, here's the information I'm getting back per optimizer |
|
Any idea when this might be available? I can pull the data off with your example command but not get it into a csv or such to put into a database nor homeassistant. Very cool to get all the optimizer data on a local api though! |
|
I have it working on Home Assistant with a custom component. I'll hopefully be able to submit a PR this weekend. I don't plan to develop anything which outputs to CSV, as that would require keeping a service running to constantly call the API and store the results. I'm happy to offer guidance if you'ld like to develop that though. You can also leverage the python library. |
|
Awesome! Looking forward to the PR so I can implement it locally until it's merged. I had hoped there was an easy way to convert the protoc output to csv so I could keep the data stored apart from HA, but if it gets into HA than its easy to pull out of HA. Thanks! |
|
@AndyRPH There is no issue converting to CSV, but I assume you're looking for historical data in that case? The local API only gives you a snapshot of the current state. Typically CSVs contain lines of data by minute, hour, day, etc. To get historical data you'll need something to poll the API and constantly collect data. |
|
Yeah I just wanted to log it somewhere that was more robust than my rpi running home assistant. The maintenance page seems to include a time stamp in the data. I had hoped there was an easy way to convert the protocol buffer message output with the message definitions as the first row then all the data below, each query to the api being a row, since the "column" numbers should not change unless I change optimizer quantities. At the moment I just threw together a cronjob and bash script that outputs it to a unique log file every 15min. I can always just figure out a regex and clean the file up later on. It looks similar to but not quite json. I didn't see any good documentation on the --decode options for the protoc command line tool. If it gets reliably into HA I'll just replicate that section of the database to another device to maintain it all. |
|
Gotcha. Do you know any python? It should be relatively simple to write a python script using the library I made to create that. |
|
Nah, not a coder by trade but picked up enough knowledge to tinker, as my day job had me learn a relatively language for their hospital software. Once your stuff is available I'll probably give python a good faith try since I'm better if I can reverse something working as a template to learning. |
|
This API doesn't appear to be available for the LCD-based SE-10000H-US. I opened a ticket with SolarEdge and they pushed out the latest firmware on the same day but I'm not seeing a web interface or any major changes in behavior.
I may have lost access to the data using semonitor though. All messages are now coming through with |
|
I think the web interface is only on the non-LCD models. SE marketing failed to rename the models based on this key design change, so it makes it fun with threads like this. ;) I can personally confirm the same lack of web functionality on LCD Models SE6000H-USRGM AND SE3000H-USRGM. No access to the interface and refused message when attempting to connect. |
|
@jshank Model number is the same (with and without SetApp) but part number is different (see datasheet): SExxxxH-XXXXBXX4 |
|
The 4 in the 'cpu version' stands for the new software gen. The LCD only versions use '3' as it's software gen 3, as we see 'portia 3' throughout the codebases and firmwares. The gen 4 is called 'Portia-NG' as in net generation. These guys really need to work on naming and versioning, as the platform is really incomparable. I do wonder how the inside looks like, if they only changed the 'control' PCB. Also I wonder if this solution is cheaper then the LCD (more expensive development for the higher-level software). Initially, when searching for firmware files for the version 3, I stumbled upon the software for the version 4 and was very pleasantly surprised that it was running Linux (our version 3's are not). As such, I started to take it apart and analyze it. Sadly, I stumble upon this thread after you spend some painful amount to RE it. The protobuf protocol, and endpoints are available in the firmware quite visibly in python (well pyc, but that was easily resolved). :) Once I realized however that this firmware was not four our devices, but only for the non-LCD ones, I was sad in that I could not run Linux on my inverter and kinda gave up on the project. What was peculiar, is that they must have been working on this for a while, as they where using an extremely outdated and slow ARM chip ... However, all the RE data is still there, with most of my findings. P.S. If someone can point me to a new firmware, ideally in chronological order, I don't mind extracting the firmware and adding it to the repo, to nicely track the changes between versions. |
|
The current SetApp apk (v2.3.0) for Android does not contain any inverter firmware (the apk file is way to small for this anyway). Comparing the proto files from solaredge-local (apparently based on 4.4.67 or later) with the protobuf descriptors in 4.0.58 (from the Portia NG analysis by @oliv3r) shows that SolarEdge did not change essential parts. This may not be true in the future. We will need to find some other way to get to the protocol buffer message descriptors so that we can keep up with SolarEdge developments. |
|
If it's anything like the iOS app, you have to login to Setapp with an installer account and then it pulls new firmwares from their server. I just updated mine yesterday. |
|
Is there any evidence that updating firmware leaves files on the mobile device that can be recovered? (If so, this will probably be easier on Android than on iOS) |
|
The fact that it downloaded firmware to my phone before I connected it to my inverter means that it probably is there before. Maybe not after. I'm not sure how to get a look at what the iOS app may have downloaded since it's not jailbroken. Nor do I have an android device. |
|
I've been looking at the code as well; and the URL is reasonably easy, the problem it's behind a login, e.g. you need an authentication token. I didn't spend time yet to see if a) your username needs some special level and b) making it it work :) As for the file, it is guaranteed to be left on the device, for the simple reason, your inverter can't pull it itself (or you wouldn't use setapp) and thus the app first downloads it, and then uploads it to the device. So after figuring out the rather simple curl command with the authentication token; it should be easy to get new firmware versions automatically downloaded. That said, I've extracted the firmware from all 'old' versions (thank apkmirror) and will update the repo with the information where needed :) |
|
Thanks @kingfisher63. Do you know how debug mode is enabled? Would it be problematic to run in that mode continually? If security was one of their reasons for locking this down they still have a lot of work to do. Anyone with a smartphone can walk up to any of these inverters, scan a QR code and get full installer access in seconds. |
I have not looked into that, but probably not. This document says "Contact SolarEdge Support to obtain this password". I doubt that it is the same for all units.
As far as I know this is done with a special (digitally signed, tamper proof) spff file, so you cannot do this yourself. You may want to try and ask a friendly SolarEdge field engineer.
Probably not. Every inverter with firmware 4.6.24 or later that permits access, must be running in debug mode. I have not yet read any reports about adverse effects. Personally I would abandon the idea of monitoring an inverter using solaredge-local. SolarEdge has designed the interface for internal use (SetApp) with no public documentation. Therefore it will be subject to potentially breaking changes without notice. |
|
A while back I actually put a support ticket into Solaredge about this. They told me they did not block access to the data on the LAN port, had no plans to block it, but also had no plans to assist customers accessing or decoding it, but it was acceptable for us to navigate to its webpage on our LAN to view the data. Since my inverter hadn't blocked me the ticket ended there. Has someone who "lost" access to the web interface via LAN opened a support ticket and been officially told it was intentionally blocked? |
See my post above, I called in a week or so ago and got 2 different answers. 1 guy said it was being intentionally blocked to prevent non-installers from messing with settings. The other guy said it should still be accessible. So I don't know what their official stance is.
I agree, which is why months ago when I lost access I just switched to using their official API. I don't have time to babysit this thing every firmware release to bob and weave around whatever changes they decide to make. It would still be nice to have access for the odd occasion I want more detailed information, but it's not the end of the world. At least now I know how to use the installer's SetApp mobile phone app to get into the inverter if I really had to (see my post above). Too bad because @drobtravels has put a lot of effort into decoding this interface and the protobuf messages. |
I am getting the same level of detail from other APIs - inverter details every 5s now (via modbus), and optimizers every 5m (via solaredge web portal). So you don't need to lose any detailed information. |
|
It's too bad SolarEdge doesn't have an API that lets you access the optimizer data from their server. The last time I checked the only way to get it was by going to the web page and manually exporting data for each optimizer. Not practical with 34 optimizers. I haven't gone there for years, so maybe they have improved it. My inverters haven't connected to the SE server in over 3 years so I haven't gotten any firmware updates, so nothing has prevented me from getting the data. They keep producing power day after day so I'm not motivated to change anything.
|
|
Mine have 'never' connected to the server :) I did ask SE for firmware updates ages ago and have kept those; also https://www.solaredge.com/service/firmware can be used to download them as well. But I'll admit I have not yet updated mine in 3 years either. 'effort' and all that |
And now I can no longer access the API |
|
Same problem here. This afternoon no access by local api. What i noticed in the feedback from HA is that the max retries exceeded. Is there a limit on the amount of information you can request to the solar edge? edit 27-04: |
|
I just scanned my SE7K and this is the result: So it seems they've firewalled everything :( |
|
ALL INSTRUCTIONS BELOW ARE EXECUTED AT YOUR OWN RISK Hello, When you read through the SolarEdge's patchnotes, you will see the following: This means that we can execute shell scripts on the device, and it just so happens to give us full root access. Using a small webserver, we can log all post requests to a console, and the SolarEdge has curl installed, so we can execute scripts, and post their commands to our webserver. #!/usr/bin/env python3
"""
Very simple HTTP server in python for logging requests
Usage::
./server.py [<port>]
"""
from http.server import BaseHTTPRequestHandler, HTTPServer
import logging
class S(BaseHTTPRequestHandler):
def _set_response(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
def do_GET(self):
logging.info("GET request,\nPath: %s\nHeaders:\n%s\n", str(self.path), str(self.headers))
self._set_response()
self.wfile.write("GET request for {}".format(self.path).encode('utf-8'))
def do_POST(self):
content_length = int(self.headers['Content-Length'])
post_data = self.rfile.read(content_length)
#logging.info(" %s",post_data.decode('utf-8'))
print("%s"%(post_data.decode('utf-8')))
self._set_response()
self.wfile.write("".encode('utf-8'))
def log_message(self, format, *args):
pass
def run(server_class=HTTPServer, handler_class=S, port=8080):
logging.basicConfig(level=logging.INFO)
server_address = ('', port)
httpd = server_class(server_address, handler_class)
logging.info('Starting httpd...\n')
try:
httpd.serve_forever()
except KeyboardInterrupt:
pass
httpd.server_close()
logging.info('Stopping httpd...\n')
if __name__ == '__main__':
from sys import argv
if len(argv) == 2:
run(port=int(argv[1]))
else:
run()Entering a wifi password with the following value will give us the files and folders on / as output in our webserver. Opening port 80 and 8080 First, we want to connect a laptop to the solaredge's wifi network, this laptop will need to run the python3 webserver. We can check the current firewall rules using the following wifi password: They should look like this: We can open port 80 and port 8080 using the following wifi password: They should now look like this: When you restart the solar edge, you should find port 80 and 8080 working again. Block the solar edge from the internet |
|
This doesn't work for me on version 4.8.28. I get no requests on the http server. I am very sure that the firewall is open. I have also tried to redirect the shell responses to a server on the subnet on the LAN interface of the inverter. Also no success. I've also tried a number of methods of getting a reverse shell without success. I also tried the command for changing the iptables for accepting traffic on the 80/8080 ports. But this traffic is still dropped. This leads me to conclude that the commands are not being executed. I connected to the inverter (SE1500M) via the wifi ap interface, and then opened the setapp web interface at http://172.16.0.1/, went to Communication, selected wifi, and then entered the commands as password for a random network. @ironsm4sh: Is that the right way to enter the commands? |
Are you sure that you included the backticks before and after the commands? You also need to restart the solarEdge for it to reload the iptables config |
|
Very sure about the backticks. Will try a restart. |
That is indeed weird, are you sure that your http server was running on the machine that was connected to the SolarEdge and had an IP address of |
|
I think it may have been caused by the fact that I do not have an external antenna on the inverter, and the wifi range is about 20 cm from the device. Good enough for a phone, but not for a laptop. I was surprised to see mosquitto mqtt server running. I had a look at the messages but they do not seem to contain any solar data. That would have been too easy ;-). |
|
@ironsm4sh
Where do you need to enter that password? |
|
You create a WiFi network and use that as its password. You can setup ad-hoc networks on the LAN as long as it can reach the script that is in the password. |
|
I have SE3000H cant access IP. Was looking to setup as installer and take control that way, if I play around with that will my own installer loose access? TBH I have it setup with Home Assistant - its delayed to 15 min but perhaps I should be happy with that! |
|
Why not just use Modbus over TCP? https://github.com/binsentsu/home-assistant-solaredge-modbus |
Because Modbus over TCP provided by Solaredge doesn't support "per-module" data, i.e. power optimizer data. |
@kingfisher63 where you ever able to retrieve the password? |
|
First of all: do not mess with grid protection settings. These settings enjoy special protection for a reason. You may violate the terms of service for your local grid if you change them beyond grid limits. According to SolarEdge documentation (Viewing and Setting Inverter Grid Protection Values v1.3) you have to contact SolarEdge Support to obtain the password. A quick look into firmware 4.8.19 (the last release before SolarEdge started encrypting images) reveals that the grid protection password is taken from a file that is initialized with 22222222 if the file does not exist at system startup. This may or may not be true for later firmware versions. If SolarEdge has done their job right, they set the password to a non default value during inverter commissioning anyway, so the default password is unlikely to work. The current SetApp version does not support changing grid protection settings (I can only view them in v2.16), so the aforementioned SolarEdge documentation is apparently outdated. |
|
Hello, I am one of the out of the box disabled SE5000H owners. Via the inverter technician I have got access to the cloud based version. This version only updates every 15 minutes, which for me is very inconvenient. Has anyone tried the non-technical approach and tried to convince SE to open a read-only local API (through mass requests/spam/specialized forums/whatever/...) ? |
|
I found a solution and local access are now possible. See here drobtravels/solaredge-local#24 (comment) |
and others
I recently had a SE10000H-US installed and noticed when I visit the IP address of the inverter it has a status page with some high level information (although no specifics about each optimizer). I see its using an unsecured API by doing a GET on http://<inverter_ip>/web/v1/status, with the response in protocol buffers.
I would be interested in reverse engineering this response so I could monitor the status in realtime without setting up the additional hardware involved in the ethernet passive approach. Has anyone done this?
I parsed the response using
protoc --decode_rawand have a pretty good idea of what the structure of the data looks like, but am unclear how its actually converted to the proper number. For example it seems likely that0x481b4600corresponds to 159 kWh. I have no experience with protocal buffers, so I may be missing something obvious, but am not seeing how to get 159 or 159,000 from what I assume is the hex representation of 481b4600I apologize if this is out of scope for this particular package, but thought some people here might be interested. I can post more details and examples if people are interested.
The text was updated successfully, but these errors were encountered: