Merge pull request #1 from jbuehl/master

update from original

README.md

@@ -1,44 +1,245 @@
 SolarEdge Monitoring
 ====================
-This project is for directly monitoring the output of SolarEdge inverters by capturing
-the network traffic between the inverters and the SolarEdge server.  This approach was
-chosen in order to be able to access module level data and to reduce dependencies on 
-Solaredge.
-
-Solaredge supports the Sunspec Alliance logging protocols, however this data is only 
-accessible via an RS485 interface on the inverter and does not include module level 
-data.  Solaredge also publishes an API that allows access to their portal, however this 
+This project enables monitoring the performance data of SolarEdge inverters and optimizers.
+Solaredge supports the open Sunspec Alliance logging protocols, however this data is only 
+accessible via an RS485 interface on the inverter and does not include module level (optimizer) 
+data.  Solaredge publishes an API that allows access to their portal, however this 
 also does not include module level data.
 
-Hardware
---------
-I have two SE5000 inverters with 34 solar panels.  The inverters are connected together
-using RS485 with one configured as the master and the other as the slave.  The master is
-connected to the internet via hardwired ethernet.
-
-A Raspberry Pi model B running Arch linux is used as an ethernet bridge through which
-all traffic to and from the inverters must flow.  A second ethernet port on the RPi is
-enabled with a USB ethernet dongle.  The cable from the inverters is plugged directly
-into one port and the other port is connected to the network.
-
-Configuration
--------------
-The only configuration required is for the network bridge interface by placing the config 
-file br0 in /etc/netctl/.  The operating system will then automatically pass all traffic
-between the two ethernet interfaces, eth0 and eth1.
-
-Software
---------
-There are two services that are enabled, secapture and seconvert.  These services are
-restarted every day at midnight by a cron job.
-
-secapture uses the tcpdump utility to capture network traffic between the inverters and
-the SolarEdge server into a pcap file.  The file is named based on its creation time.
-
-seconvert reads the SolarEdge logging data in a pcap file and converts it.  The output
-is sent to one or more destinations:
-
-- log entries in a database
-- log entries in flat files
-- a file containing the current values formatted as json
+**semonitor.py** is a program that implements a subset of the SolarEdge protocol.  It can be 
+used to parse the performance data that has been previously captured to a file, or it can 
+interact directly with a SolarEdge inverter over the RS232, RS485, or ethernet interface.
+Performance data is output to a file in JSON format.
+
+**seextract.py** is a program that can extract the SolarEdge protocol messages from a PCAP
+file that contains data captured from the network.
+
+**se2state.py** follows a file containing performance data and outputs a JSON file
+that contains the current state of the inverter and optimizer values.
+
+**se2csv.py** reads a file containing performance data and outputs two separate comma delimited
+files that contain inverter and optimizer data that is suitable for input to a spreadsheet.
+
+semonitor.py
+------------
+
+SolarEdge inverter performance monitoring using the SolarEdge protocol.
+
+### Usage 
+    python semonitor.py [options] [datasource]
+
+### Arguments
+    datasource           Input filename or serial port.
+                         If no data source is specified, the program reads from 
+                         stdin, unless the data source type is network.
+                         If a file name is specified, the program processes the 
+                         data in that file and terminates, unless the -f option 
+                         is specified, in which case it waits for further data 
+                         to be written to the file.
+                         If the data source corresponds to a serial port, 
+                         process the data from that port.
+
+### Options
+    -a                   append to output file if the file exists
+    -b                   baud rate for serial data source (default: 115200)
+    -c cmd[/cmd/...]     send the specified command functions
+    -d debugfile         where to send debug messages (stdout|syslog|filename) 
+                         (default: syslog)
+    -f                   wait for appended data as the input file grows 
+                         (as in tail -f)
+    -m                   function as a RS485 master
+    -n interface         run DHCP and DNS network services on the specified 
+                         interface
+    -o outfile           write performance data to the specified file in 
+                         JSON format (default: stdout)
+    -r recfile           file to record all incoming and outgoing messages to
+    -s inv[,inv,...]     comma delimited list of SolarEdge slave inverter IDs
+    -t 2|4|n             data source type (2=RS232, 4=RS485, n=network)
+    -v                   verbose output
+    -x                   halt on data exception
+
+### Notes
+Data may be read from a file containing messages in the SolarEdge protocol that was previously created by 
+seextract from a pcap file, or the output from a previous run of semonitor.  It may also be
+read in real time from one of the RS232, RS485, or ethernet interfaces on a SolarEdge inverter.
+
+Debug messages are sent to the system log, unless the -d option is specified.  If an error occurs
+while processing data, the program will log a message and continue unless the -x option is
+specified.
+
+The level of debug messaging is controlled using the -v option, which may be specified up to
+4 times:
+    -v      log input parameters and file operations
+    -vv     log incoming and outgoing messages
+    -vvv    log the parsed data of incoming and outgoing messages
+    -vvvv   log the raw data of incoming and outgoing messages
+Messages logged at the -vv level and above contain the device or file sending or receiving the
+message, the direction it was sent, the message size, and an internal sequence number.  Separate
+sequences are kept for incoming and outgoing messages.
+
+The -t option is used to specify the data source type for non-file input.  If the data source is 
+a serial port, the -t option must be included with either the 2 or 4 value to specify
+whether it is connected to the RS232 or RS485 port.  If there is no data source specified and 
+-t n is specified, semonitor will listen on port 2222 for a connection from an inverter.
+
+To interact directly with an inverter over the network, semonitor must function as the SolarEdge
+monitoring server.  This means that the host running semonitor must be connected to the inverter
+over the ethernet interface.  In this configuration, semonitor may function as the DHCP server
+and the DNS server to provide an IP address to the inverter and to resolve the hostname of the
+SolarEdge server (usually prod.solaredge.com) to the IP address of the semonitor host.  This
+requires that semonitor be run with elevated (root) priveleges in order to access the standard
+DHCP and DNS ports.  The -n option specifies the name of the network interface that the inverter
+is connected to.
+
+The -c, -m, and -s options are not meaningful if input is from a file or stdin.
+
+The -m option is only valid if a serial port is specified, and one or more inverter IDs
+must be specified with the -s option.  If this option is specified, there cannot
+be another master device on the RS485 bus.  semonintor will repeatedly send commands to
+the specified inverters to request performance data.
+
+The -c option may be specified for a serial device or the network.  The option specifies one
+or more SolarEdge protocol command functions separated by a "/".  Each command function
+consists of a hex function code followed by zero or more comma separated hex parameters.
+Each parameter must begin with one of the letters "B", "H", or "L" to designate the
+length of the parameter:
+    B = 8 bits
+    H = 16 bits
+    L = 32 bits
+All function codes and parameters must be hexadecimal numbers, without the leading "0x".
+Exactly one inverter ID must be specified with the -s option.  After each command is sent,
+semonitor will wait for a response before sending the next command.  When all commands
+have been sent and responded to, the program terminates.  Use the -vvv option to view
+the responses.
+
+Commands initiated by semonitor as the result of the -c or -m options need to maintain a
+monotonically increasing sequence number.  A file named seseq.dat will be created to persist the
+value of this sequence number across multiple executions of semonitor.
+
+### Examples
+    python semonitor.py -o yyyymmdd.json yyyymmdd.dat
+
+Read from SE data file yyyymmdd.dat and write data to the json file yyyymmdd.json.
+
+    python seextract.py yyyymmdd.pcap | python semonitor.py -o yyyymmdd.json
+
+Extract SolarEdge data from the file yyyymmdd.pcap using seextract, process
+it with semonitor, and write data to the json file yyyymmdd.json.
+
+    python semonitor.py -o yyyymmdd.json -m -s RS232 7f101234,7f105678 -t 4 COM4
+
+Function as a RS485 master to request data from the inverters 7f101234 and 7f105678
+using serial port COM4.
+
+    python semonitor.py -c 0012,H0329 -s 7f101234 -d stdout -vvv -t 2 /dev/ttyUSB0
+
+Send a command to the inverter 7f101234 to request the value of parameter 0x0329
+using RS232 serial port /dev/ttyUSB0.  Display the messages on stdout.
+
+    python semonitor.py -c 0011,H329,L1/0012,H329/0030,H01f4,L0 -s 7f101234 -d stdout -vvv -t 2 /dev/ttyUSB0
+
+Send commands to the inverter 7f101234 to set the value of parameter 0x0329 to 1,
+followed by a command to reset the inverter using RS232 serial port /dev/ttyUSB0.
+Display the messages on stdout.
+
+    sudo python semonitor.py -o yyyymmdd.json -t n -n eth1
+
+Start the dhcp and dns services on network interface eth1.  Accept connections
+from inverters and function as a SolarEdge monitoring server.  Write performance
+data to file yyyymmdd.json.
+
+seextract.py
+------------
+
+Read a PCAP file that is a capture of the traffic between a inverter and the SolarEdge 
+monitoring server.  Filter out the TCP stream between the inverter to the server.
+
+### Usage 
+    python seextract.py [options] pcapFile
+
+### Arguments
+    pcapFile        pcap file or directory to read
+                    If a file is specified, the program processes the data in 
+                    that file and terminates, unless the -f option is specified, 
+                    in which case it waits for further data to be written to the 
+                    pcap file.
+                    If a directory is specified, all files in the directory are 
+                    processed.
+                    If a directory is specified and the -f option is specified, 
+                    only the file in the directory with the newest modified date 
+                    is processed and the program waits for further data in that 
+                    file.  If a new file is subsequently created in the 
+                    directory, the current file is closed and the new file 
+                    is opened. 
+### Options
+    -a              append to output files
+    -f              output appended data as the pcap file grows (as in tail -f)
+    -o outfile      output file to write
+    -s server       SolarEdge server hostname or IP address (default: 
+                    prod.solaredge.com)
+    -v              verbose output
+
+### Examples
+    python seextract.py -o yyyymmdd.dat yyyymmdd.pcap
+
+Convert the data in file yyyymmdd.pcap and write the output to file yyyymmdd.dat
+
+    python seextract.py -f pcap/
+
+Monitor PCAP files in directory pcap/ and write the output to stdout.
+
+    python seextract.py -o allfiles.pcap pcap/
+
+Convert all the pcap files found in directory pcap/ and write the output to files
+allfiles.pcap.
+
+se2state.py
+-----------
+Maintain a file containing the current state of SolarEdge inverters and optimizers.
+
+### Usage
+    python se2state.py options [inFile]
+
+### Arguments
+    inFile          File containing performance data in JSON format. (default:
+                    stdin)
+                    The program will follow (wait for new data to be written to)
+                    the file.
+
+### Options
+    -o stateFile    File containing the current (last read) data values for each
+                    inverter and optimizer values from the input file.  It will
+                    be overwritten every time that new data is read.
+                    
+### Examples
+    python semonitor.py -t n | tee yyyymmdd.json | python se2state.py -o solar.json
+
+Accept connections from inverters over the network.  Send performance data to
+the file yyyymmdd.json and also maintain the file solar.json with the current state.
+
+se2csv.py
+---------
+Convert SolarEdge inverter performance monitoring data from JSON to CSV.
+
+### Usage
+    python se2csv.py options [inFile]
+
+### Arguments
+    inFile          File containing performance data in JSON format. (default:
+                    stdin)
+
+### Options
+    -a              append to inverter and optimizer files
+    -d delim        inverter and optimizer file delimiter (default: ",")
+    -h              write column headers to inverter and optimizer files
+    -i invFile      file to write inverter data to
+    -o optFile      file to write optimizer data to
+
+### Examples
+    python se2csv.py -i yyyymmdd-inv.csv -o yyyymmdd-opt.csv -h yyyymmdd.json
+
+Read from SE data file yyyymmdd.json and write inverter and optimizer data to the csv files
+yyyymmdd-inv.csv and yyyymmdd-opt.csv with headers.
+
+
 

crontab

@@ -1,3 +1,3 @@
 0 0 * * * /usr/bin/systemctl restart secapture
-0 0 * * * /usr/bin/systemctl restart seconvert
+0 0 * * * /usr/bin/systemctl restart semonitor
 

seconvert → deprecated/seconvert

@@ -8,5 +8,5 @@ do
 done
 
 cd /root
-/usr/bin/python /root/solaredge/seconvert1.py -f -v pcap/ | python /root/solaredge/seconvert2.py -v -j /root/solar.json
+/usr/bin/python /root/solaredge/seextract.py -f -v pcap/ | python /root/solaredge/semonitor.py -v -j /root/solar.json -
 

seconvert1.py → deprecated/seconvert1.py

@@ -99,7 +99,7 @@ def readPcapRec(pcapFile):
     tcpHdr = readTcpHdr(pcapFile)
     # data is whatever is left
     dataLen = pcapRecLen - etherHdrLen - ipHdrLen - tcpHdrLen
-    if debugData: log("pcap", "pcapSeq", pcapSeq, "pcapRecLen", pcapRecLen, "dataLen", dataLen)
+    if debugData: log("pcap", "pcapSeq", pcapSeq, "srcIp", ip2str(ipHdr[3]), "dstIp", ip2str(ipHdr[4]), "pcapRecLen", pcapRecLen, "dataLen", dataLen)
     if dataLen > 0:
         if (seIpAddr == 0) or (ipHdr[4] == seIpAddr):   # only process records where IP dest is SE server
             if outFile:
@@ -170,6 +170,7 @@ def getOpts():
     try:
         if seHostName != "":    
             seIpAddr = struct.unpack("!I", socket.inet_aton(socket.gethostbyname_ex(seHostName)[2][0]))[0]    # 0xd9449842
+            log("serverIpAddr:", ip2str(seIpAddr))
     except:
         print "Unable to resolve hostname", seHostName        
 
@@ -235,6 +236,9 @@ def log(*args):
 #    print message
     syslog.syslog(message)
 
+def ip2str(ipAddr):
+    return "%d.%d.%d.%d" % struct.unpack("!BBBB", struct.pack("!L", ipAddr))
+
 def terminate(code, msg=""):
     print msg
     sys.exit(code)