So I have a wind anometer setup and I would like to start logging the data. I have a trigger setup to look at any changes to the Wind "device" (using RFXCOM plugin with an oregon scientific anometer) that populates a variable for the time stamp and populates another variable with current windspeed and direction. What I would like to do is start pushing that to some sort of log file... I can get it to load into the event log with the action collection plugin, but I would have to then parse all of that data out from previous logs which seems like it could be difficult... ideally I would like to just load the data into some text file that I could then import into excel to try to pull out some historical data to see the viability of wind power. Anything anyone can offer to get me started would be greatly appreciated.

Berkinet made a suggestion in another thread that would probably work; I was planning on trying this technique out to generate a tab separated file that I could then pull into excel

berkinet wrote:

automate wrote:...If I could just maintain a log of the DrivewayArrivingTimestamp Variable and DrivewayDepartingTimestamp I would be home free.

DrivewayArriving (used as a programming flag and is reset after a few seconds)
DrivewayArrivingTimestamp (uses the "insert timestamp into variable" from the Action Collection)

DrivewayDeparting (used as a programming flag and is reset after a few seconds)
DrivewayDepartingTimestamp (uses the "insert timestamp into variable" from the Action Collection)

Ok, since you are maintaining the data in variables, there is an easier way for you to approach this. Here is an example for the Arriving case:

1) Create a trigger that looks for any change in the value variable DrivewayArrivingTimestamp.
2) The trigger's Action is to Execute Script.
3) Set the script to Embedded - AppleScript.
4) Paste this code as the script

Code: Select all

set theLogText to "A car has arrived"
set theLogEntry to the value of variable "DrivewayArrivingTimestamp" & ":    " & theLogText
do shell script ("/bin/echo \"" & theLogEntry & " \">>$HOME/driveway.log")

5) Edit the text for theLogText to whatever you want.

Now, anytime a car arrives in your driveway (causing DrivewayArrivingTimestamp to change) an entry will be recorded in the file driveway.log in your home directory. Of course, you can change the file name and location as you wish.

For the Departing case, just duplicate these steps, but use the Departing time stamp in the trigger and the second line of the AppleScript and change the message text appropriately
.
The key line in the script is the "do shell script" command. This uses the built-in command echo to write the message to the end of the log file (>>). While cryptic, this is a lot easier than opening, writing, and closing the file.

Post back any questions and (hopefully) a report of your success.

Perfect! Thank you. This works great.

Have you tried to pull this file into excel as of yet? Didn't know the colon was a valid separator.

Dewster35 wrote:Perfect! Thank you. This works great.

Have you tried to pull this file into excel as of yet? Didn't know the colon was a valid separator.

Just found the post here at work earlier this morning and won't be able to get to the machine till I get home, but I wasn't planning on using the colon anyway; I was going to look up how to generate a tab character in Applescript, or failing that, just go lazy and use commas to separate everything and pull it in as a "csv" file, which I think SQL lite will also accept, although the main reason to go to Excel first is to do plotting, averaging, min, max...

Excel will also accept CSV... I think I'm going to change to a comma as well. Thanks again. I'm getting a data point every 30 seconds or so which does tend to clutter up my event log... but hopefully it won't cause too many problems.

I think there's a way to tell Indigo not to log a trigger...and BTW, you need to thank Berkinet for the script, not me. All I did was find it.

Just a checkbox under the trigger... it's working great. I know you didn't write the script, but you found it. I searched for quite a bit and couldn't find what I was looking for, so thank you for that.

Well it didn't take long for this not to scale! Any idea how to create a daily log? This file pretty quickly became unwieldy.

Code: Select all

set theLogDate to the value of variable "WindTime"
set theLogDirection to the value of variable "WindDirection"
set theLogValue to the value of variable "Wind" & "," & theLogDirection & "," & theLogDate
do shell script ("/bin/echo \"" & theLogValue & " \">>$HOME/wind.log")

You should check out the SQL Logger plugin in the latest version.

It makes storing historical device state (and variable value) changes in to either SQLite or PostgreSQL very easy.

matt (support) wrote:You should check out the SQL Logger plugin in the latest version.

It makes storing historical device state (and variable value) changes in to either SQLite or PostgreSQL very easy.

The problem with that is that (unless I am mistaken) you can only have one data base, and it logs EVERYTHING to it. I don't need to know when I turned the kitchen light on 2 months ago, but next year, I might still want to look up what the hottest day in July was this year and how long the AC ran during it... Dewster's post encouraged me to fix the simple file I put up once the RFX was operating. To create a separate csv file every day, I defined another two variables using the insert timestamp into variable from Jay's plugin with a custom format of %m-%d-%y and when I log data, the script checks to see if the date changed, and if it does, it moves the log file to a date stamped one and then creates a header before proceeding to continue logging. I haven't fully tested it, but it SHOULD result in a new unique file for each day...

Code: Select all

set theLogEntry to the value of variable "LogTime" & ",  " & the value of variable "AVGWind" & ",  " & the value of variable "OSIDir" & ",  " & the value of variable "OutTemp" & ",  " & the value of variable "RH" & ",  " & the value of variable "DayRain" & ",  " & the value of variable "OSIRainRate" & ",  " & the value of variable "ACDuctTemp" & ",  " & the value of variable "ACReturnTemp" & ",  " & the value of variable "FreezerTemp" & ",  " & the value of variable "TempDif"
if (value of variable "NowDate" is not value of variable "LastDate") then
   do shell script ("/bin/mv $HOME/Documents/OSIWeather/Log.csv $HOME/Documents/OSIWeather/WeatherData" & value of variable "LastDate" & ".csv")
   do shell script ("/bin/echo \"" & "Time, Wind, Dir, OutTemp, OutHum, Rain, RainRate, ACDuct, ACRet, Freezer, TempDelta" & " \">>$HOME/Documents/OSIWeather/Log.csv")
   set the value of variable "LastDate" to the value of variable "NowDate"
end if
do shell script ("/bin/echo \"" & theLogEntry & " \">>$HOME/Documents/OSIWeather/Log.csv")

If I've messed anything up, I'll let you know tomorrow.

It sounds like you are very close to a solution, but if I were starting from scratch what I would do is:

1) Use the SQL Logger plugin. Set the prune duration to 3 days.

2) Have a nightly python script that runs that queries the database (for the day that ended) for the variable value or device states you do want to keep, and with those values have it either: A) copy them in CSV format to a date stamped file, B) copy them to a different database file (I'd do this one).

Then I would have a historical database of the data I'm interested in that I can easily query (querying data from SQLite or PostgreSQL is easy from the command line, the plugin page has some examples).

There is quite a bit I'd like to add to the SQL Logger plugin, like the ability to select specific devices to log and GUI actions that do SQL queries on the database. We don't have time to add that now, but there is quite a bit that can be added to enhance it.

Actually, thinking about this a bit more, the SQL Logger could be hacked to have a skipPrune list. So any device or variable ID on that list would not follow the UI set pruning value...

John,
For the NowDate and LastDate... those are just the MM/DD/YYYY within the system?

Matt.... thanks but, way over my head. Struggling along with just applescript here

Dewster35 wrote:John,
For the NowDate and LastDate... those are just the MM/DD/YYYY within the system?

I set the NowDate variable as an additional action in the same "Copy Data" action group that sets the wind, temperature, humidity and rain variables, using the "Insert timestamp into variable" from Jay's "Actions" plugin. This can be configured with a custom format, which I chose as "%m-%d-%y", which loads the variable with "05-23-2012" (or whatever the current system date is). And I created a separate "Log Data" action group that just runs this script. Then I set up a schedule that runs once a minute and executes the "Copy Data" and "Log Data" action groups one after the other. The if statement in the "Log data" script compares the NowDate value with the LastDate value and if they do not match, it uses the system mv command to move (ie rename) the log.csv file to a different directory with a name that includes the LastDate variable (I initially loaded the LastDate variable with the string "Older" before I executed the "Log data" script the first time), then it creates a new log.csv file with a header and copies the value in the NowDate variable into the LastDate variable before grabbing all the weather variables and stuffing them into the new file. And as long as the "copy data" action that is putting the OSI device data into variables keeps happening on the same day, the if statement isn't executed and information just appends to the Log.csv. But the first execution after midnight, the NowDate and LastDate don't match and the log.csv gets renamed and it starts all over again. And the script as written did work; as soon as I saved the new Log Data script, it moved the log.csv file to WeatherDataOlder.csv, and at midnight moved the data to WeatherData05-22-2012.csv. One caveat; My unix is pretty rusty, but I seem to recall that the >> operator will fail if there are any spaces in the name you create.