Cacti polling of device status xml causes grief

I recently threw together a quick cacti poller script to hit indigo's device status feeds for the 2 thermostats I have, this lets me graph set point and temperature over time (I already do this with a sensorprobe unit for other areas of the house). The problem I have found is that with the constant polling (2 every 5 mins) the http interface of indigo stops responding. When trying to debug with the ios client, i have noticed that it will accept the commands, but will not reflect the changes in the client, the commands will eventually be executed (sometimes up to 60 seconds after issuing them).

Is there some type of memory leak, or object destruction problem in the pyweb instance?

This problem has occurred in the last 2 versions including 4.1.15

Thanks
-Dan

[matt (support)]

Hi Dan,

Can you try: letting the poller run a few times to get a stable memory footprint, use 'top' in the Terminal to see the memory usage, let it run until it bogs down, then use 'top' again?

I'd write down and compare the memory usage for your memory script, Python (running the IndigoWebServer), and IndigoServer. It sure sounds like some type of memory/resource leak, but I'm not sure from where it coming.

So, there doesnt seem to be any memory leak. The problem came back 2 days ago, and I have left it. Basically the device status requests for everything seems to have stopped/slowed. State changes from the iphone client are SLOW, like really slow (up to 2 mins sometimes).

Here is the ps output:


11:22am
administrator 33177 0.4 0.1 2744984 6592 ?? S 11:21AM 0:00.24 /usr/bin/osascript /Library/Application Support/Perceptive Automation/Indigo 4/Scripts/Background Tasks/Weather NOAA sync.scpt
administrator 33178 0.3 0.1 2735176 5424 ?? S 11:21AM 0:00.18 /usr/bin/osascript /Library/Application Support/Perceptive Automation/Indigo 4/Scripts/Background Tasks/sensorprobe.scpt
administrator 33176 0.0 0.7 2530564 44748 ?? S 11:21AM 0:00.71 python /Library/Application Support/Perceptive Automation/Indigo 4/IndigoWebServer/IndigoWebServer.py -i1176 -w8176 -cfn
administrator 33175 0.0 0.2 499256 10584 ?? S 11:21AM 0:00.47 /Library/Application Support/Perceptive Automation/Indigo 4/IndigoServer.app/Contents/MacOS/IndigoServer
administrator 33171 0.0 0.2 515612 15660 ?? S 11:21AM 0:00.91 /Applications/Indigo 4.app/Contents/MacOS/Indigo 4 -psn_0_1552763


1 day later: 2:49pm
administrator 35259 0.9 0.1 2745144 6524 ?? S Thu01PM 11:05.25 /usr/bin/osascript /Library/Application Support/Perceptive Automation/Indigo 4/Scripts/Background Tasks/Weather NOAA sync.scpt
administrator 35260 0.4 0.1 2735176 5568 ?? S Thu01PM 6:33.06 /usr/bin/osascript /Library/Application Support/Perceptive Automation/Indigo 4/Scripts/Background Tasks/sensorprobe.scpt
administrator 35258 0.0 0.8 2531876 48300 ?? S Thu01PM 0:43.98 python /Library/Application Support/Perceptive Automation/Indigo 4/IndigoWebServer/IndigoWebServer.py -i1176 -w8176 -cfn
administrator 35256 0.0 0.2 501600 14904 ?? S Thu01PM 1:19.52 /Library/Application Support/Perceptive Automation/Indigo 4/IndigoServer.app/Contents/MacOS/IndigoServer
administrator 33171 0.0 0.4 520272 22504 ?? S Thu11AM 4:59.28 /Applications/Indigo 4.app/Contents/MacOS/Indigo 4 -psn_0_1552763


6 days later: 10:30am (stopped reporting 2 days ago)
administrator 35259 3.1 0.1 2745144 6840 ?? S Thu01PM 154:20.53 /usr/bin/osascript /Library/Application Support/Perceptive Automation/Indigo 4/Scripts/Background Tasks/Weather NOAA sync.scpt
administrator 35260 0.4 0.1 2735176 5568 ?? R Thu01PM 36:18.37 /usr/bin/osascript /Library/Application Support/Perceptive Automation/Indigo 4/Scripts/Background Tasks/sensorprobe.scpt
administrator 35258 0.0 0.8 2531876 48632 ?? S Thu01PM 4:08.33 python /Library/Application Support/Perceptive Automation/Indigo 4/IndigoWebServer/IndigoWebServer.py -i1176 -w8176 -cfn
administrator 35256 0.0 0.2 501600 15044 ?? S Thu01PM 7:31.80 /Library/Application Support/Perceptive Automation/Indigo 4/IndigoServer.app/Contents/MacOS/IndigoServer
administrator 33171 0.0 0.5 525392 29552 ?? S Thu11AM 35:20.70 /Applications/Indigo 4.app/Contents/MacOS/Indigo 4 -psn_0_1552763

OOOhhh, just noticed something. Seems like the venstar insteon adapters have stopped responding.

Error "Heat: Living Room" mode request; send failed (no acknowledgment)
Error "Heat: Living Room" temperature request; send failed (no acknowledgment)
Error "Heat: Living Room" humidity request; send failed (no acknowledgment)
Error "Heat: Living Room" setpoint request; send failed (no acknowledgment)
Error "Heat: Kitchen" mode request; send failed (no acknowledgment)
Error "Heat: Kitchen" temperature request; send failed (no acknowledgment)
Error "Heat: Kitchen" humidity request; send failed (no acknowledgment)
Error "Heat: Kitchen" setpoint request; send failed (no acknowledgment)


I dis+reconnected them last week before this test, is it possible for non-responsive insteon devices to impact the performance of controlling other devices throughout?

[jay (support)]

Interesting - we've just recently had several other reports of thermostat adaptors causing problems. What PowerLinc model are you using?

Powerlinc controller 2414U

[jay (support)]

So much for it being an interaction issue - at least one other report was with a 2413 (which is quite different than the 2414).

[matt (support)]

So when you got the no-ack errors for the thermostats above, were the thermostats connected at that time or are they still disconnected? And can you get the status (via the Get Status button in the Main Window) of other INSTEON modules, or is everything failing?

And what is the poll interval in the Device dialog you have for your thermostats? I wouldn't recommend polling more than once every several minutes. It takes lots of INSTEON commands/traffic to extract out all of the data from the thermostat, which is going to increase the likelihood of collisions. That shouldn't cause them to fail to respond, but it might be related to the problem you are seeing.

Thermostats never get disconnected/turned off/loose power etc.

Get Status/Update Set Point etc fails from the main window, get status/update state works just fine for every other insteon device.

Oddly enough I havent done anything, and touch client is back to normal speed for lights etc. I was just about to test if main window was more responsive than the touch client.

The polling from cacti happens every 5 minutes, and just hits http://SERVER:8176/devices/Heat%3A%20Living%20Room.xml (etc) via a curl request. I dont see indigo sending insteon requests when that is hit, so the indigo requests _should_ only be happening every 30 mins (as per thermostat config), and every 2 hours (i have a full status sync).