Google Code skipped issue 21

[openhab-bot]

Skipping this issue number to maintain synchronization with Google Code issue IDs.

Original issue: http://code.google.com/p/openhab/issues/detail?id=21

[openhab-bot]

From [email protected] on August 22, 2011 04:59:36

Hi Kai,

About persistence (i prefer to move our conversation here), take a look at DB4O deployed in a OSGi environment. http://www.osgi.org/wiki/uploads/CommunityEvent2007/PersistenceForOSGi_CRosenberger.pdf I think that it could be a good idea to have such a service in OSGi and then, a specific bundle could manage the item/state info in openHab.

This way, other bundles could use persistence service and you could develop specific code for openHab items persistence requirements, and also for query this info.

Do you think it could be a good idea?

regards

[openhab-bot]

From kai.openhab on August 22, 2011 05:15:50

I didn't yet know about db4o for OSGi, this sounds indeed very interesting with a very low footprint.
Do you know by chance, if there is some out-of-the-box time series support? I.e. is it possible to store the same object multiple times without overwriting the last version, but by simply being able to query by timestamps? If not, we would probably have to wrap each item instance into something with a timestamp before storing it.

[openhab-bot]

From [email protected] on August 22, 2011 06:06:55

Yes, i think (we are starting with db4o, too). A couple of links: http://measuringmeasures.com/blog/2009/1/20/building-a-time-series-databse-in-db4o.html http://developer.db4o.com/Blogs/Product/tabid/167/entryid/131/Default.aspx I think that it's an already walked path, so it could be easy to deal with.

We will do some research about that so i will keep you update.

Regards

[openhab-bot]

From kai.openhab on August 24, 2011 12:46:54

This seems to be a better link as it includes some follow up on the initial posting, which speeds up performance by 100x: http://developer.db4o.com/Forums/tabid/98/aft/5193/Default.aspx Ok, it looks promising, so let me know the outcome of your research - I'd be more than happy to include a timeseries-capable state persistence based on db4o with your help!

[openhab-bot]

From [email protected] on August 25, 2011 05:36:35

Thanks Kai,

You could follow part II and III with this links: http://measuringmeasures.blogspot.com/2009/03/db4o-performance-tuning-part-ii.html http://measuringmeasures.blogspot.com/2009/03/db4o-performance-tuning-part-iii.html Well, we've prepared a bundle with db40 as a plugin, and some services exposed (basic things, get connection, crud...)

We will update this thread with news.

[openhab-bot]

From [email protected] on September 16, 2011 02:28:31

Hi again,

As i promised, i'm going to update this thread with our steps forward a persistence solution.

Attached, you will find a pdf file (in english) with the detail of the solution, and also a png (in spanish) where you could see images that could help to understand the code and final results.

There is a tricky way to select the items to log. We have used a specific group, so you must create the group and associate the items to log to that group.

We have just finished a working version, and it works fine.

It would be great to receive your feedback.

regards.

Attachment: Aviary srvpsat-gpm-int Picture 1.png DB4O(1).pdf

[openhab-bot]

From [email protected] on September 16, 2011 02:32:04

In the UI layer, we plan to use flot ( http://code.google.com/p/flot/ ) to draw different charts with this info.

Things to do in persistence layer:

• I think that it will be useful to delete data far away from X weeks, for example. This way the database must be cleaned by a job.
• We will rename some code, to use names more appropiate.
• There is no security in this layer. Is there any need for that?

[openhab-bot]

From kai.openhab on September 16, 2011 02:39:32

Wow, that looks cool - allow me the weekend to look over it and provide you feedback!

There is no security in this layer. Is there any need for that?

No, not for the moment as openHAB is lacking security features in general...

[openhab-bot]

From kai.openhab on September 17, 2011 04:10:21

Hi David,

As promised, here is some feedback:

1. db4o indeed seems to be a good choice, the integration and usage looks straight forward
2. The idea to use a group to "mark" the items to be persisted is good, although it seems as if it is a workaround in order to not having to change the item DSL syntax. I am not yet decided about the best solution here. One idea would be to add some binding configuration like { persist="events" } to it, but persistance is more a "core" feature than a simple binding (as persistence also needs access to the ItemRegistry). Out of this reason, I could also imagine to alter the DSL (and possibly the Item interface) to have persistance properties.
3. I would like to see other options than only persisting events. Something like "persist a value every x seconds", "persist ONLY the very latest status" (just to be able to recover from crashes), some RRD way "only keep 1000 values taken every second, 1000 every hour, 1000 every day, etc.." (this could solve your "cleaning old values" issue).
4. Besides item states/events, the persistence service should also support other kinds of data - e.g. something like logging information (see for example the "News" tab on the tablet prototype http://goo.gl/GpZ4H) )
5. I couldn't really see the Java API for accessing the persistence service from your documents. Imho, there should be an OSGi service for an openHAB specific persistence service (which does not expose the db4o interfaces), which can be used by other bundles. The interface for this service would be very important to get right.
6. I would like the servlet to be included into the REST API one day. The user could then request information in a format of his choice (XML, JSON, JSONP) and it would be an "official" part of the REST API.
7. How is the database configured? I.e. where does db4o write its data to?

That much for now, looking forward to your answers :-)
Kai

[openhab-bot]

From [email protected] on October 17, 2011 05:52:56

hi everyone, how is this going? seems promising!!

[openhab-bot]

From [email protected] on October 20, 2011 03:19:50

Hi there,

We will update this thread asap.

Thanks

[openhab-bot]

From kai.openhab on November 27, 2011 13:47:16

Asap is a very relative term ;-)
Could you give us an update? Cheers!

[openhab-bot]

From kai.openhab on December 06, 2011 13:02:32

If there is no response on this issue, I might look into it myself soon as I am really requiring this feature and would like to see some progress.
So David, if you are still on it, please be so kind and give an update - cheers!

[openhab-bot]

From [email protected] on December 13, 2011 08:26:17

Hi Kai,

First of all, sorry for the delay...

We will send our work as soon as possible, in fact, we want to release it today, but we are suffering some troubles with the development enviroment, probably due to the last changes comitted to the repository (drools, etc.)

We are trying to rebuild the enviroment from scratch and see if it's related to our code.

We will keep you update.

Regards

[openhab-bot]

From [email protected] on December 20, 2011 08:47:02

Hi Kai,

First of all, i want to send you a brief explanation of how it's working. It's not perfect, but could be a good start:

We have developed 5 bundles:

1. org.openhab.io.bbdd ==> Database. It contains the service interface: (CRUD operations and paginated select, get and close connection.
2. org.openhab.io.bbdd.db4o ==> DB4O Implementation. It contains all the services to work with DB4O + driver (database is written in $home/logs path)
3. org.openhab.io.bbdd.listener ==> It uses io.bbdd bundle. We called it EventTracer.
4. How it works: AbstractEventSubscriber implementation to listen the openhab event bus (commands and updates).
5. It receives a Item Registry to know which items will trace, as they have to belong to a specific group (it's a little tricky) to be traced by the system.
6. org.openhab.ui.querydata ==> It also uses io.bddd bundle. Returns all data in JSON or XML format.
7. It works as a servlet, but it will be better to work as REST service (not done).

To use the last bundle (to recover data), there are two services:

url = localhost:8080/query/alerts ==> It returns all traced events in the last week (doesn't matter which item) ordered by date (it could feed a "last alarms" service).
url = localhost:8080/query/events/itemname ==> It returns all traced events in the last week for one specific item (it could feed a flot graph, we will explain how to use it).

Kai, could you please tell me the usual way to send/share this code?

Thanks.
Regards

[openhab-bot]

From [email protected] on December 20, 2011 08:48:00

Sorry, i forgot to explain the 4th bundle. It's a MySQL implementation but it's not ready yet.

[openhab-bot]

From kai.openhab on December 20, 2011 12:01:17

Thanks, looking forward to dive into the code :-)

Kai, could you please tell me the usual way to send/share this code?

Simply create a clone of the repo at http://code.google.com/p/openhab/source/clones and push your changes to this repository.

Regards,
Kai

Status: Started

[openhab-bot]

From [email protected] on December 20, 2011 12:38:41

What about external persistence? Do you know pachube or thingspeak? It could be great to have these features in openhab.

As far as i know, thingspeak allows http put calls to send your measures (limited, i think) and pachube has json http calls (maybe a little harder).

What about using pachube or thingspeak in rules? This way you could check or uncheck which items do you want to trace... and the only thing you need is a function (pachube-trace, thingspeak-trace), and maybe a couple of lines in the conf file.

What do you think?

[openhab-bot]

From [email protected] on December 21, 2011 08:56:26

Hi,

You could get all changes from our clone: http://code.google.com/r/dmorales-persistence/ We have uploaded all bundles (unless mysql, number four) and also pom files, so you can check our code.

Regards

[openhab-bot]

From kai.openhab on December 21, 2011 09:02:16

Cool, thanks! I cannot promise that I can review it before christmas, but definitely still this year :-)

[openhab-bot]

From [email protected] on December 21, 2011 09:03:35

ok, have nice days¡

[openhab-bot]

From kai.openhab on December 30, 2011 13:31:19

I promised too much, the year end comes faster than expected... I already had a look at the code, but to give a decent feedback with reflected suggestions, I will need some more time - but you are not forgotten! Have a good start in 2012!

[openhab-bot]

From kai.openhab on February 02, 2012 13:23:20

I finally managed to get through the code - it already looks very useful indeed!
Still, most of my wishes from comment 9 remain; I'd like to continue the discussion on the mailing list, see https://groups.google.com/d/topic/openhab/uQ3WYYqYuYk/discussion

[openhab-bot]

From openhab.lb on March 02, 2012 04:01:33

i have all setup, the data is going to the database, how do use them, how to get out from there ? do you have widget for openhab ? how do you connect to the database ? i would like to have a graph from temperatures ... how is that possible ?

[openhab-bot]

From kai.openhab on March 02, 2012 04:40:07

Might I ask to wait with those questions until the feature is implemented?
The solution for openHAB 1.0.0 will look different to what David had posted here. See the discussion at https://groups.google.com/forum/?fromgroups#!topic/openhab/uQ3WYYqYuYk I hope to come up with some initial implementation in a few weeks. I'll post an update in the discussion group once it's there.

Labels: -Version-later Version-1.0.0

[openhab-bot]

From [email protected] on March 02, 2012 04:40:40

Please, Read comment #15 in this thread (how to recover data)

[openhab-bot]

From openhab.lb on March 02, 2012 05:59:39

Thanks a lot for answering

i could set it und the data is in the database ...

but with

To use the last bundle (to recover data), there are two services:

url = localhost:8080/query/alerts ==> It returns all traced events in the last week (doesn't matter which item) ordered by date (it could feed a "last alarms" service).
url = localhost:8080/query/events/itemname ==> It returns all traced events in the last week for one specific item (it could feed a flot graph, we will explain how to use it).

i don´t know how to do that...

HTTP ERROR 404

Problem accessing /query/alerts. Reason:

ProxyServlet: /query/alerts

[openhab-bot]

From openhab.lb on May 04, 2012 00:22:02

Hi Kay,

i have build the peristence branch. i copied the addons in the addon directory and they are loaded, but no database files are created and it seems that the persistence directory is not observed by the folderobserver. i used you default configuration with the line for the folderobserver

[openhab-bot]

From kai.openhab on May 04, 2012 00:28:04

The feature branch has been merged to the default branch (and only there the Maven build has been tested), so please use this.

Details on what has been implemented for this issue can be found here: https://groups.google.com/forum/?fromgroups#!topic/openhab/fecbooXCN_U

Status: Implemented