-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[influxdb] Exception when storing integer values #8798
Comments
This also seems to be an issue for Items with OnOffType state which is stored as 0 and 1 values in the database. |
Thanks @wborn, one question, when you queried the InfluxDB do you remember if the integer values from the binding were stored as integers or floats in InfluxDB? |
When the influxdb add-on was able to store values the number of decimals was as expected. I'm just testing my migration path and now restored the original influxdb DB that I still use with my OH2 instance. It seems that now the OH3 influxdb add-on is able to store integer values in the existing series. But it could be that it is now working because in the meantime I had to reset the mapdb (#8794) which I have configured with a |
For my setup its either...
im pretty sure you can see whats happeneing... if the first value is a whole number its written as an integer, otherwise its a float irrespective of what has been written previously. |
I also see this issue in OH3. Looking at both source codes, I believe in OH 2 influxdb provider always translated decimaltype into float, independent of the scale. With the rewritten persistence addon, it is considering the scale and either writing integer or float. That results in existing series with float will not accept integer, at least on my influxdb instance. Maybe we are able to configure this behaviour in the rewritten influx addon to allow backwards compatibility? |
No it did not. The same optimization was already used in OH2. Do you also use mapdb to restore values on startup? If mapdb restores values with a wrong scale it can break the series in influxdb. |
@wborn When you say the number of decimals was as expected what do you mean? In your Influx database from OH 2.0 how are the integers stored inside Influx, as floats or integers? About your comment @LukasA83 it was my initial explication, but looking at code, at first sight, it doesn't seem the intent. I really think that the problem was with the old driver that is 4 years old: If I confirm that I think that the best decision is to use always floats. In fact for me is the best decision although we started from a fresh start because it's very dangerous to try to use integer while OH doesn't have an IntegerType. Because if an addon returns sometimes integers and sometimes floats it will break the persistence, or a case when an addon change the precision in an addon update |
That the number of decimals of the state reported by openHAB exactly matches that of the number of decimals stored in the influxdb. So in my case with the openweathermap binding humidity channel, both the old and new (after removing the series) store the values as integers in the influxdb series.
If the binding provides floats on that particular humidity channel that's of course possible while another binding can provide integers on a humidity channel. I'll have a look at the mapdb restore values on startup behavior as that persistence add-on was also rewritten. |
Sure @wborn, but what I wanted to say is that now that we know that InfluxDB has problems using several types for the same field is dangerous to use integers. I will try to debug when I've time because it's not clear to me still how the old addon avoids the different types problem. Because looking at drivers source code seems good: |
Yes it's an accident waiting to happen. I can also check what happens with the OH2 influxdb addon what happens if a float ends up in an integer series. |
@wborn you are right, I'm using MapDB to restore values on startup. |
I've test now sending an integer value from OH 2.0 and it's stored as float in InfluxDB:
Can someone confirm if he has some values stored as integers in a database only written by OH 2.0? |
Yes this is from the influxdb I am using currently using with OH 2.5.9:
|
It looks like the mapdb restore seems to work OK after fixing (openhab/openhab-core#1737) and adding/enabling debug logging:
These floats causing these issues could also be restored by the default rrd4j persistence service, which is nowadays automatically installed. See #8806. |
When I only use rrd4j to restore on startup the precision seems to get lost: Before restart:
After restart:
|
Then it seems not to be a problem on the persistence add-on per see, isn't it? One thing that I would like at least, is an option to use always floats when storing values, and I think that is the safest default for new users. But of course, not good for compatibility. |
It could of course always be a setting but that introduces complexity. It would also be strange to start storing on/off values as 0.0 and 1.0. Switching to floats may also break graphs or anything else querying InfluxDB. I will do some more testing to see if the issues I ran into really were due to the rrd4j restore on startup behavior. If that's the case it would be better to either fix rrd4j or not use it for restoring values on startup. Such issues may also impact the data stored by other persistence add-ons. We shouldn't be creating workarounds in every persistence add-on just because rrd4j is buggy. |
I don’t know if im helping or confusing things…. But..
I just tested..
Shutdown openhab
Removed restoreOnStartup I had defined in influxdb… (that’s the only persistence file I have)
Dropped influxdb openhab and recreated new one
Restarted openhab and I still get
org.influxdb.InfluxDBException$FieldTypeConflictException: partial write: field type conflict: input field "value" on measurement "GarageFreezer_Power" is type integer, already exists as type float dropped=1
so im not sure its related to restoreon startup unless its occurring somehow without me seeing it (is it configurable outside of the persistence.files in OH3) but since I have no other persistence addons installed…..
M
From: Wouter Born <[email protected]>
Sent: 20 October 2020 10:03
To: openhab/openhab-addons <[email protected]>
Cc: Mike McCaffery <[email protected]>; Comment <[email protected]>
Subject: Re: [openhab/openhab-addons] [influxdb] Exception when storing integer values (#8798)
It could of course always be a setting but that introduces complexity. It would also be strange to start storing on/off values as 0.0 and 1.0. Switching to floats may also break graphs or anything else querying InfluxDB.
I will do some more testing to see if the issues I ran into really were due to the rrd4j restore on startup behavior. If that's the case it would be better to either fix rrd4j or not use it for restoring values on startup. Such issues may also impact the data stored by other persistence add-ons. We shouldn't be creating workarounds in every persistence add-on just because rrd4j is buggy.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<#8798 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADSXK4UY5ZFI3END2BYNP53SLVG2NANCNFSM4SVM2YCQ>.
|
Its well hidden if it is…
168 x Active x 80 x 3.0.0.202010160310 x openHAB Core :: Bundles :: Model Persistence
169 x Active x 80 x 3.0.0.202010160310 x openHAB Core :: Bundles :: Model Persistence IDE
170 x Active x 80 x 3.0.0.202010160314 x openHAB Core :: Bundles :: Model Persistence Runtime
183 x Active x 80 x 3.0.0.202010160301 x openHAB Core :: Bundles :: Persistence
257 x Active x 80 x 3.0.0.202010200354 x openHAB Add-ons :: Bundles :: Persistence Service :: InfluxDB
M
From: Wouter Born <[email protected]>
Sent: 20 October 2020 11:35
To: openhab/openhab-addons <[email protected]>
Cc: Mike McCaffery <[email protected]>; Mention <[email protected]>
Subject: Re: [openhab/openhab-addons] [influxdb] Exception when storing integer values (#8798)
Can you check if you have rrd4j installed @mccaffm<https://github.com/mccaffm>? It is installed by default, restoring items on startup, see #8806<#8806>.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#8798 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADSXK4S3W6VC2GS2TMFBLZTSLVRXDANCNFSM4SVM2YCQ>.
|
I agree that it has to be fixed in rrd4j, but for me, the inherent problem has a broader scope. |
I did some more testing and also ran into the issue without having rrd4j installed. So I think it has to do with the new client library. It also seems that storing both integers and floats was never an issue with the library in OH 2.5.x. I was able to store both without any issue:
I also tested with removing the tags but that also did not fix the issue. Also when I remove the optimization in convertBigDecimalToNum to always use the |
That is good news, althought I can't understand completely yet: The documentation says that it can be two different types in the same shard. It may be creates two shards when the types are different? Or maybe the problems are if different values are in the same batch¿? I will try to isolate in a simple main and look at drivers code or ask a question on InfluxDB mailing list to be sure about the fix. |
Suggestion would be Nov 1. |
Maybe it only applies when using the new client? I used the exact same data and InfluxDB 1.8.3 with OH 2.5 and OH 3. I've replicated all the Docker containers+data used by my production OH 2.5 instance on my desktop to test my migration to OH 3. |
I will check tomorrow if I can to see the real cause, I only had time to do a little test but with new client it's easy to reproduce, writing only to points (a double and an integer) in a clean database and I caught the error. |
Finally, I found the mystery, the old addon really stores all numbers as double. The difference is not in the driver's version, it's in the deprecated method it uses. Old addon uses
Outputs:
Select in terminal using influx:
It seems that Chronograf is saying the truth here. About the Openhab part, as when the value is read is parsed directly in DecimalType constructor passing a string it doesn't matter if it's integer or float. In conclusion, I will remove the |
Great! Thanks for having a look at it and it indeed explains the mystery. It's nice to know the root cause and that we can fix the issue without having to add any configuration options while staying compatible with existing databases. 👍 |
Fixes openhab#8798 Improve documentation with more explicit information about Influx types used Implement toSTring to InfluxPoint to allow some trace info to be useful in case it's needed Signed-off-by: Joan Pujol <[email protected]>
…8831) * Update documentation with changed Influx2 RC port * Fix problem with non decimal numeric types Improve documentation with more explicit information about Influx types used Implement toString to InfluxPoint to allow some trace info to be useful in case it's needed Fixes #8697 Fixes #8798 Signed-off-by: Joan Pujol <[email protected]>
…penhab#8831) * Update documentation with changed Influx2 RC port * Fix problem with non decimal numeric types Improve documentation with more explicit information about Influx types used Implement toString to InfluxPoint to allow some trace info to be useful in case it's needed Fixes openhab#8697 Fixes openhab#8798 Signed-off-by: Joan Pujol <[email protected]>
…penhab#8831) * Update documentation with changed Influx2 RC port * Fix problem with non decimal numeric types Improve documentation with more explicit information about Influx types used Implement toString to InfluxPoint to allow some trace info to be useful in case it's needed Fixes openhab#8697 Fixes openhab#8798 Signed-off-by: Joan Pujol <[email protected]>
The following Exception is logged when integer values are stored by the OH3 influxdb persistence add-on:
This value is based on a humidity channel of the openweathermap binding (openweathermap:weather-and-forecast:account:local:current#humidity).
I queried the influxdb (1.8.3) and it looked like the Outside_Humidity series only had integer values. It does seem that the OH3 influxdb persistence now adds an item tag which previously wasn't stored. When I manually update the item to e.g. "91.0 %" it did store the value.
After I removed the series it was possible to store new integer values. However if I would then add a float it would error with:
I also saw the add-on is now using a new client library which may be causing this new behavior.
The text was updated successfully, but these errors were encountered: