INDIGOplotD -- a plot tool for indigo

https://www.dropbox.com/s/r7opappb417g65o/indigoplot-2-8-1.zip v 2-8-1
will switch to NO SQL if database not found, should get you out of the no-sql found loop.

fixes:
SQL-Import: if there is sparse data for a device (i.e. no information for a bin) , SQL import took the average of the last bin and filled the next bin with the average of the previous instead of the last available info.
eg: if you head 90% "ON" time but the last value was OFF :1,1,1,1,1,1,1,1,1,0 = 90% , the next bins (if no information is available) where set to 90% instead of 0. That is fixed now.

Improvements:
better SQL import structure that will enable faster imports i.e. if old data was already imported only new data will be requested from SQL. On some smaller MACs with 2-4G memory and with > 2 GB SQL database the import might take several minutes. In the next release that will be reduced significantly. This step is the first intermediate step towards that.

future release (~ 2 weeks) will have:
-Design / create your own data, put it into a variable. or file e.g. (0;1;3|1;2;5|3;5;6|...|9;4;3) (as x;y1;y2|x;y1;y2|..) and INDIGOplotD can use this information to make a plot with 1.2,3,.. lines and nice fonts, texts scales.. Every time the variable / file gets changed/updated the plot will be refreshed.


Karl

1. do you have sql logger enabled?
2. could you switch to "Do not use SQL-Logger" in Configuration at the bottom of the page, see attached screen shot.
3. and if you are on OSX 10.9 could you try using matplot?

indigoplot to indigoplotD does not transfer any data, should be no interference, 2 completely independent plugins.


Once its running, could you go into the test plot and see if there are any lines defined.

thanks

Karl

I tried to enable sql logger but it did not help.
After I applied 2 and 3 it stated to work. So I think I am up and running again now. I have no preferences to use GNUplot instead of matplot. Thank you for your help!

https://www.dropbox.com/s/q4l59p8tresnyrf/indigoplotD-3-0-1.zip v 3-0-1


fixes:
smaller issues in rare circumstances fixed
- some more error controls when bad data is entered,

added:
- help text added in the menus if you hover over the fields
- addition to plot time series with data from devices and variables This version supports to plot data from a variable or file:
assume a variable "test2" contains:
#x;distance;direction|1;2;3|2;5;6|5;3;4|8;2;5|10;3;4

can be used to produce a plot (with added smooth options etc ) like:

variable plot-S1.png (6.34 KiB) Viewed 6157 times

You could fill the variable in an applescript or python script or create a file that contains the data to be plotted.

You can set the x-axis description, scale, format etc. and naturally all the option for formatting, smoothing, math etc are available too.

The detailed help is in the attached PDF file in the download.


I have tested this version for some days, naturally not in all the possible combinations.

Karl

Karl,

This is another cool plugin that I'm hoping to take advantage of. However, I'm running into a few problems setting it up. I can't seem to get the plugin to play nice with the SQL Logger (either mode). I was able to get it to behave when I switched the config away from SQL Logger.
If it helps, I'm running 10.9.3 Server. I'm also using Postgres and the db user has full access to the database (via phpPgAdmin for the setup).
While I've tried to figure this out on my own... I deleted the plugin along with all the files in my Documents folder; I even created a new Postgres database called indigo_data just to be sure it wasn't that. The log output below is from a fresh install of the plugin from the UC lib and not dropbox.

I trimmed the misc stuff out of the log and noted what I tried to do as such ****my actions****.

Please advise if I can provide any other info or trouble shooting steps.

Thanks!!

--Richard

Code: Select all

****Installed the plugin**** 
  Jun 17, 2014, 10:02:54 PM
  Installing and enabling plugin "INDIGOplotD" v2.6.1
  Loading plugin "INDIGOplotD 2.6.1"
  Starting plugin "INDIGOplotD 2.6.1" (pid 1894)
  Started plugin "INDIGOplotD 2.6.1"
  INDIGOplotD                     initializing  ... 2-6-1
  INDIGOplotD                     indigo version found: 6
  INDIGOplotD Error               Error in plugin execution startup:

Traceback (most recent call last):
  File "plugin.py", line 317, in startup
  File "plugin.py", line 3368, in getSQLdataDynamic
  File "plugin.py", line 3593, in testSQL
  File "plugin.py", line 3812, in readSQLLoggerPreferences
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'text'

  INDIGOplotD Error               ------------------------------------
  INDIGOplotD Error                not  configured yet, please select menue:  Indigo /plugins/indigoplot/Configure...
  INDIGOplotD Error               ------------------------------------


****Configuration change from Online to Batch - then saved & reloaded the plugin****
Jun 17, 2014, 10:12:17 PM
  INDIGOplotD                     Updating device/prop from SQL db: no device or variable defined.. skipping update   ...
  INDIGOplotD                     Updating device/prop from SQL db: no device or variable defined.. skipping update   ...

Jun 17, 2014, 10:12:44 PM
  Reloading plugin "INDIGOplotD 2.6.1"
  Stopping plugin "INDIGOplotD 2.6.1" (pid 1894)
  INDIGOplotD                     stopConcurrentThread  ...
  INDIGOplotD                      main loop stopped
  INDIGOplotD                     stopConcurrentThread  ...
  Stopped plugin "INDIGOplotD 2.6.1"
  Starting plugin "INDIGOplotD 2.6.1" (pid 2419)
  Started plugin "INDIGOplotD 2.6.1"
  INDIGOplotD                     initializing  ... 2-6-1
  INDIGOplotD                     indigo version found: 6
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/minute.dat lines: 2304
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/hour.dat lines: 912
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/day.dat lines: 390
  INDIGOplotD                     Updating device/prop from SQL db: no device or variable defined.. skipping update   ...
  INDIGOplotD                     Updating device/prop from SQL db: no device or variable defined.. skipping update   ...
  INDIGOplotD                     initializing  ...2
  INDIGOplotD Error               ------------------------------------
  INDIGOplotD Error                not  configured yet, please select menue:  Indigo /plugins/indigoplot/Configure...
  INDIGOplotD Error               ------------------------------------


****Configuration change from Batch to Online - then saved & reloaded the plugin****
Jun 17, 2014, 10:13:38 PM
  INDIGOplotD                     initialized
  Z-Wave                          received "Luminance, Porch" sensor update to 7 lux
  Z-Wave                          received "Humidity, Porch" sensor update to 80%
  Z-Wave                          received "Temperature, Porch" sensor update to 78.1 °F
  Reloading plugin "INDIGOplotD 2.6.1"
  Stopping plugin "INDIGOplotD 2.6.1" (pid 2419)
  INDIGOplotD                     stopConcurrentThread  ...
  INDIGOplotD                      main loop stopped
  INDIGOplotD                     stopConcurrentThread  ...
  Stopped plugin "INDIGOplotD 2.6.1"
  Starting plugin "INDIGOplotD 2.6.1" (pid 2492)
  Started plugin "INDIGOplotD 2.6.1"
  INDIGOplotD                     initializing  ... 2-6-1
  INDIGOplotD                     indigo version found: 6
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/minute.dat lines: 2304
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/hour.dat lines: 912
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/day.dat lines: 390
  INDIGOplotD Error               Error in plugin execution startup:

Traceback (most recent call last):
  File "plugin.py", line 317, in startup
  File "plugin.py", line 3368, in getSQLdataDynamic
  File "plugin.py", line 3593, in testSQL
  File "plugin.py", line 3812, in readSQLLoggerPreferences
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'text'

  INDIGOplotD Error               ------------------------------------
  INDIGOplotD Error                not  configured yet, please select menue:  Indigo /plugins/indigoplot/Configure...
  INDIGOplotD Error               ------------------------------------


****Configuration changed back to Batch - then saved & reloaded the plugin****
Jun 17, 2014, 10:14:38 PM
  INDIGOplotD                     Updating device/prop from SQL db: no device or variable defined.. skipping update   ...
  INDIGOplotD                     Updating device/prop from SQL db: no device or variable defined.. skipping update   ...
  Reloading plugin "INDIGOplotD 2.6.1"
  Stopping plugin "INDIGOplotD 2.6.1" (pid 2492)
  INDIGOplotD                     stopConcurrentThread  ...
  INDIGOplotD                      main loop stopped
  INDIGOplotD                     stopConcurrentThread  ...
  Stopped plugin "INDIGOplotD 2.6.1"
  Starting plugin "INDIGOplotD 2.6.1" (pid 2539)
  Started plugin "INDIGOplotD 2.6.1"
  INDIGOplotD                     initializing  ... 2-6-1
  INDIGOplotD                     indigo version found: 6
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/minute.dat lines: 2304
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/hour.dat lines: 912
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/day.dat lines: 390
  INDIGOplotD                     Updating device/prop from SQL db: no device or variable defined.. skipping update   ...
  INDIGOplotD                     Updating device/prop from SQL db: no device or variable defined.. skipping update   ...
  INDIGOplotD                     initializing  ...2
  INDIGOplotD Error               ------------------------------------
  INDIGOplotD Error                not  configured yet, please select menue:  Indigo /plugins/indigoplot/Configure...
  INDIGOplotD Error               ------------------------------------


****Configured my first data source point****
Jun 17, 2014, 10:25:16 PM
  INDIGOplotD                     sql commands launched  2014-06-17 22:25:16.601770  waiting for SQL tasks to end to read data into INDIGOplotD pid=3068
  INDIGOplotD                     sql commands launched  2014-06-17 22:25:22.852742  waiting for SQL tasks to end to read data into INDIGOplotD pid=3092
  INDIGOplotD                     sql commands launched  2014-06-17 22:25:29.126774  waiting for SQL tasks to end to read data into INDIGOplotD pid=3118
  INDIGOplotD                     sql commands launched  2014-06-17 22:25:35.400252  waiting for SQL tasks to end to read data into INDIGOplotD pid=3142
  INDIGOplotD                     sql commands launched  2014-06-17 22:25:41.672731  waiting for SQL tasks to end to read data into INDIGOplotD pid=3163
(Note the above continues endlessly)


****Created my plot then switched to Online SQL****
INDIGOplotD Error               Error in plugin execution runConcurrentThread:

Traceback (most recent call last):
  File "plugin.py", line 2705, in runConcurrentThread
  File "plugin.py", line 3368, in getSQLdataDynamic
  File "plugin.py", line 3593, in testSQL
  File "plugin.py", line 3812, in readSQLLoggerPreferences
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'text'

  INDIGOplotD Error               plugin runConcurrentThread function returned or failed (will attempt again in 10 seconds)


****Switched config back to Batch****
Jun 17, 2014, 10:37:22 PM
  INDIGOplotD                     sql commands launched  2014-06-17 22:37:22.683359  waiting for SQL tasks to end to read data into INDIGOplotD pid=5827
  INDIGOplotD                     sql commands launched  2014-06-17 22:37:28.945049  waiting for SQL tasks to end to read data into INDIGOplotD pid=5854
  INDIGOplotD                     sql commands launched  2014-06-17 22:37:35.213480  waiting for SQL tasks to end to read data into INDIGOplotD pid=5878
  INDIGOplotD                     sql commands launched  2014-06-17 22:37:41.490728  waiting for SQL tasks to end to read data into INDIGOplotD pid=5899
  INDIGOplotD                     sql commands launched  2014-06-17 22:37:47.771652  waiting for SQL tasks to end to read data into INDIGOplotD pid=5922
  Reloading plugin "INDIGOplotD 2.6.1"
  Stopping plugin "INDIGOplotD 2.6.1" (pid 2539)
  INDIGOplotD                     stopConcurrentThread  ...
  INDIGOplotD                      main loop stopped
  INDIGOplotD                     stopConcurrentThread  ...
  Stopped plugin "INDIGOplotD 2.6.1"
  Starting plugin "INDIGOplotD 2.6.1" (pid 5938)
  Started plugin "INDIGOplotD 2.6.1"
  INDIGOplotD                     initializing  ... 2-6-1
  INDIGOplotD                     indigo version found: 6
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/minute.dat lines: 2304
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/hour.dat lines: 912
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/day.dat lines: 390
  INDIGOplotD                     sql commands launched  2014-06-17 22:37:51.974984  waiting for SQL tasks to end to read data into INDIGOplotD pid=5948
  INDIGOplotD                     initializing  ...2
  INDIGOplotD                     sql commands launched  2014-06-17 22:37:58.367536  waiting for SQL tasks to end to read data into INDIGOplotD pid=5978
  INDIGOplotD                     sql commands launched  2014-06-17 22:38:07.743820  waiting for SQL tasks to end to read data into INDIGOplotD pid=6002
(Note the above continues endlessly)


****Switched the config to not use the SQL Logger (appears happy[initialized])****
un 17, 2014, 10:47:44 PM
  Reloading plugin "INDIGOplotD 2.6.1"
  Stopping plugin "INDIGOplotD 2.6.1" (pid 5938)
  INDIGOplotD                     stopConcurrentThread  ...
  INDIGOplotD                      main loop stopped
  INDIGOplotD                     stopConcurrentThread  ...
  Stopped plugin "INDIGOplotD 2.6.1"
  Starting plugin "INDIGOplotD 2.6.1" (pid 8147)
  Started plugin "INDIGOplotD 2.6.1"
  INDIGOplotD                     initializing  ... 2-6-1
  INDIGOplotD                     indigo version found: 6
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/minute.dat lines: 2304
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/hour.dat lines: 912
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/day.dat lines: 390
  INDIGOplotD                     initializing  ...2
  INDIGOplotD                     initialized


1. could you try 3.0.1 - see 2 postings before this one.
2. I have never tried to make it work with postgres - and the batch version will NOT work with it.
The dynamic may be, but I don't know. The error message is in the section where the connection to the database gets tested- The error indicates that it is not connecting properly
Also I don't know if you can have both (sql-logger and postgres) running at the same time.

The SQL code was "donated" by Adam (RogueProeliator), Adam, may be you could comment?

If you use it without SQL you will not have any historical data before you start picking devices / states in indigoplotd. It collects its own history. but otherwise its the same.

as for postgres, I would need some time to setup and test. -and I would lose all my sql-logger history ..

Karl

code section..

Code: Select all

   def readSQLLoggerPreferences(self, prefFilePath):
      configParams = dict()

      preferencesDom = xml.etree.ElementTree.parse(prefFilePath)
      prefRootNode = preferencesDom.getroot()
   
      databaseTypeStr = prefRootNode.find("databaseType").text
      if databaseTypeStr == "postgresql":
         configParams["dbType"] = indigosql.kDbType_postgres
         configParams["dbName"] = prefRootNode.find("postgresqlDatabase").text
         configParams["serverHost"] = prefRootNode.find("postgresqlHistoryDataost").text
         configParams["sqlUsername"] = prefRootNode.find("postgresqlUser").text
         configParams["sqlPassword"] = prefRootNode.find("postgresqlPassword").text
      else:
         configParams["dbType"] = indigosql.kDbType_sqlite
         configParams["dbName"] = prefRootNode.find("sqliteFilePath").text
         
      return configParams


Karl:

I believe your code there is correct for reading the configuration -- it seems to match mine in terms of configuring the database access parameter's dictionary. How are you creating the database connection after that -- do you check similar to:

Code: Select all

dbConfig = self.readSQLLoggerPreferences(sqlLoggerPrefPath)
      if dbConfig["dbType"] == indigosql.kDbType_sqlite:
         return indigosql.IndigoSqlite(dbConfig["dbName"], None, self.logMessage, None)
      elif dbConfig["dbType"] == indigosql.kDbType_postgres:
         return indigosql.IndigoPostgresql(dbConfig["serverHost"], dbConfig["sqlUsername"], dbConfig["sqlPassword"], dbConfig["dbName"], None, self.logMessage, None)
      else:
         return None

After that, I imagine it should be working fine -- I use and read SQL Logger history from Postgre all the time without any issues. Without knowing your plugin I can't say for sure, but it looks like it is actually making it to sending commands before erroring -- are you able to tell from that log what exactly piece of code/SQL access is bailing?

You'll find that this is a significantly more mature/powerful/full-featured database than SQLlite, so it is somewhat possible you are doing something that SQLlite allows that a fully typed and such database won't. However, for simple selects and what not normally you should be ok.

Adam

Adam,
the error message is in THIS section.
at this line:

configParams["serverHost"] = prefRootNode.find("postgresqlHistoryDataost").text

Its the first thing that happens -- well before any really query !
Could it be if there are both database files present it stumbles?

Karl

Karl & Adam,

I can gladly remote in and help with any type of changes and I'm not concerned with losing any historical data at this point.

Would it help if I modified something in the code to not do a look up but specified the host, database, UN, PW, etc.?

--Richard

configParams["serverHost"] = prefRootNode.find("postgresqlHistoryDataost").text

That should be:

Code: Select all

configParams["serverHost"] = prefRootNode.find("postgresqlHost").text

I missed that on your first posting...

Adam

Somehow in some replace all this must have been mixed up . Will fix later today. My charter internet is down now for > 2 hours.


Sent from my iPhone using Tapatalk

Karl - Since you don't have the Postgres database tied to the SQL Logger (and will lose your data), let me see what I can do. Just please know that my knowledge of coding is nowhere near yours! Will advise when possible.

As soon as my internet is back up I can post the fix. Charter is down 6 hours now .. Also TV. The whole area.


Sent from my iPhone using Tapatalk

Viola... That did the trick. Changed ("postgresqlHistoryDataost") to ("postgresqlHost") and it appears to be working correctly in ONLINE mode. As you eluded to earlier... Batch mode is a no go (same as before: INDIGOplotD sql commands launched 2014-06-18 14:15:51.306699 waiting for SQL tasks to end to read data into INDIGOplotD pid=49509).

One question left over... Once I started the service, it appeared that no data was being found in the database (no records returned from database, skipping). Is this normal? Trimmed Log below.

Thanks,
Richard


****Mod the plugin.py file as above and started the plugin****

Code: Select all

Jun 18, 2014, 2:12:01 PM
  Enabling plugin "INDIGOplotD 2.6.1"
  Starting plugin "INDIGOplotD 2.6.1" (pid 49304)
  Started plugin "INDIGOplotD 2.6.1"
  INDIGOplotD                     initializing  ... 2-6-1
  INDIGOplotD                     indigo version found: 6
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/minute.dat lines: 2304
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/hour.dat lines: 912
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/day.dat lines: 390
  INDIGOplotD                     initializing  ...2
  INDIGOplotD                     initialized

****Changed the config to online and reloaded the plugin****
Jun 18, 2014, 2:12:56 PM
  Reloading plugin "INDIGOplotD 2.6.1"
  Stopping plugin "INDIGOplotD 2.6.1" (pid 49304)
  INDIGOplotD                     stopConcurrentThread  ...
  INDIGOplotD                      main loop stopped
  INDIGOplotD                     stopConcurrentThread  ...
  Stopped plugin "INDIGOplotD 2.6.1"
  Starting plugin "INDIGOplotD 2.6.1" (pid 49348)
  Started plugin "INDIGOplotD 2.6.1"
  INDIGOplotD                     initializing  ... 2-6-1
  INDIGOplotD                     indigo version found: 6
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/minute.dat lines: 2304
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/hour.dat lines: 912
  INDIGOplotD                      read file /Users/richard/Documents/INDIGOPlotD/day.dat lines: 390
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.000040 Door, Front / zoneState.faulted/count   ...
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                                                          no records returned from database, skipping     ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.058197 Door, Back / zoneState.faulted/count   ...
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                                                          no records returned from database, skipping     ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.115773 Door, Patio / zoneState.faulted/count   ...
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                                                          no records returned from database, skipping     ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.172446 Nest Thermostat / temperatureInput1/max   ...
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                                                          no records returned from database, skipping     ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.234854 Nest Thermostat / temperatureInput1/min   ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.234902 Nest Thermostat / temperatureInput1/average   ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.234941 Nest Thermostat / humidityInput1/max   ...
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                                                          no records returned from database, skipping     ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.286450 Nest Thermostat / humidityInput1/min   ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.286493 Nest Thermostat / humidityInput1/average   ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.286529 Nest Thermostat / setpointCool/average   ...
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                                                          no records returned from database, skipping     ...
  INDIGOplotD                     Updating device/prop from SQL db:  0:00:00.343436 Nest Thermostat / hvacCoolerIsOn/average   ...
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                     connected to indigo_data as indigo on 127.0.0.1
  INDIGOplotD                                                          no records returned from database, skipping     ...
(The above continues on for all the devices I already had defined in the plugin after getting to work w/o the SQL Logger)


Then that's probably the SQL statement that works for SQL lite but not for postgre...
Internet still down. Need to get a wireless backup.


Sent from my iPhone using Tapatalk

https://www.dropbox.com/s/wjozhk9kxcdoluj/indigoplotd3-0-2.zip v 3-0-2

fixed the post- greSQL syntax error

will need some help in fixing the sql statement not returning data.

Does anyone know if there is a command prompt version to test sql statements?


Karl