Mac Catalina Python 3 code conflict with Indigo Python 2

As per recent Indigo Domotics notice, AppleScript is no longer supported in favor of just using Python scripts. I had been using a Trigger Action to execute script (file action) to log Thermostat ON/OFF times and other related Devices data with a Python script file with embedded AppleScript to update Mac Number App spreadsheet. Don't know if best work-a-round option, but I have the Trigger Action to send email to my Mac mail app that has a rule if email Subject contains a specific word to run AppleScript to update Numbers spreadsheet. So far, working OK.

[jay (support)]

Sorry, but the thread title has me a bit confused: what Python isn't working with AppleScript? Post your AppleScript that talks to Numbers and identify where it needs to get Indigo information and perhaps we can figure out a solution for you.

[MODERATOR NOTE]: moved to a more appropriate forum.

I had been using this code for a long time and then recently, without any code change, the AppleScript stop working. No error codes, I just assume that AppleScript was no longer supported. As you can see, it is not a good script, but it worked OK.

Code: Select all

#! Version 103
import subprocess, sys

heat = str(indigo.devices[16890506].states["hvacHeaterIsOn"]) # 03-Thermostat
cool = str(indigo.devices[16890506].states["hvacCoolerIsOn"]) # 03-Thermostat
hvacMode = str(indigo.devices[16890506].states["hvacOperationMode"]) # 03-Thermostat
outdoorTempF = indigo.devices[475937690].states["temperatureF"] # 02-Horry County Airport
supplyTempF = indigo.devices[1918428069].states["sensorValue"] # Supply Duct Sensor
returnTempF = indigo.devices[830102769].states["sensorValue"] # Return Duct Sensor
thermostatTempF = indigo.devices[16890506].states["temperatureInput1"] # Thermostat

if hvacMode == "1":
   hvacMode = "Cool"
elif hvacMode == "2":
   hvacMode = "Heat/Cool"
elif hvacMode == "3":
   hvacMode = "Heat"

# if heat or cool is true, then that mode is now working
file = open("/Users/WilliamBoatright/Documents/Indigo/HVAC-EnergyUsageText.txt","w")
file.write(str(heat)+"\n"+str(cool)+"\n"+str(outdoorTempF)+"\n"+str(hvacMode)+"\n"+str(supplyTempF)+"\n"+str(returnTempF)+"\n"+str(thermostatTempF))
file.close()

indigo.server.log("Heat - "+str(heat)+"  Cool  - "+str(cool)+"  Temp - "+str(outdoorTempF)+"  Mode - "+str(hvacMode)+
                  "  Supply - "+str(supplyTempF)+"  Return - "+str(returnTempF)+"  Thermostat - "+str(thermostatTempF))

applescript = '''
-- get date string

set rightNow to (current date)
set theTime to text -8 thru -1 of ((current date) as «class isot» as string)
set YR to ((year of rightNow) as text)
set MO to ((month of rightNow) as integer)
if MO < 10 then set MO to "0" & MO as text
set MO to MO as text
set DA to ((day of rightNow) as integer)
if DA < 10 then set DA to "0" & DA as text
set DA to DA as text

set rightNow to YR & "/" & MO & "/" & DA & " " & theTime
set rightNow to rightNow as text -- YYYY/MM/DD HH:MM:SS   in 24 hour mode

set filePath to "/Users/williamboatright/Documents/Indigo/HVAC-EnergyUsage-Heat&Cool-" & MO & "-" & YR & ".numbers" as string

set newFile to ("Macintosh HD:Users:WilliamBoatright:Documents:Indigo:HVAC-EnergyUsageText.txt")
set theFileContents to paragraphs of (read file newFile)
set theList to theFileContents
set heatStatus to item 1 of theList
set coolStatus to item 2 of theList
set outdoorTempF to item 3 of theList
set mode to item 4 of theList
set supplyTempF to item 5 of theList
set returnTempF to item 6 of theList
set thermostatTempF to item 7 of theList

if heatStatus is equal to "True" or coolStatus is equal to "True" then
   set status to "ON "
else
   set status to "OFF "
end if

try
        POSIX file filePath as alias
        true
on error
        --   false
        tell application "Finder"
                set master_file to POSIX file "/Users/williamboatright/Documents/Indigo/HVAC-EnergyUsage-Heat&Cool-Template-v2.0.numbers"
                set master_dir to (container of file master_file)
                set root_nm to the name of file master_file
                set ext to the name extension of file master_file
                set root_nm to characters 1 thru ((the offset of ext in root_nm) - 2) of root_nm as text
                set duplicated_file to duplicate master_file to master_dir
                set the name of duplicated_file to ("HVAC-EnergyUsage-Heat&Cool-" & MO & "-" & YR & ".numbers")
        end tell

        tell application "Numbers"
                activate
                open filePath
               
                tell document 1
                        tell sheet mode
                                tell table "HVAC-Energy-Usage"
                                        tell cell 6 of column "A"
                                                set value to rightNow
                                                set alignment to center
                                                set vertical alignment to center
                                        end tell

                                        tell cell 6 of column "C"
                                                set the value of cell "C6" to the value of the cell "F1"
                                                set alignment to center
                                                set vertical alignment to center
                                        end tell
                                end tell
                        end tell
                end tell
                save every document with saving
        end tell

end try



tell application "Numbers"
   activate
   open filePath
      
   tell table "HVAC-Energy-Usage" of sheet mode of document 1
      add row below last row
      tell column "A"
         set value of last cell to rightNow
         set alignment to center
         set vertical alignment to center
      end tell
      
      tell column "B"
         set value of last cell to status
         set alignment to center
         set vertical alignment to center
      end tell
      
      tell column "I"
         set value of last cell to outdoorTempF
         set alignment to center
         set vertical alignment to center
      end tell

      tell column "J"
         set value of last cell to mode
         set alignment to center
         set vertical alignment to center
      end tell

      tell column "K"
         set value of last cell to supplyTempF
         set alignment to center
         set vertical alignment to center
      end tell

      tell column "L"
         set value of last cell to returnTempF
         set alignment to center
         set vertical alignment to center
      end tell

      tell column "M"
         set value of last cell to thermostatTempF
         set alignment to center
         set vertical alignment to center
      end tell
      
   end tell
   
   save every document with saving
   --quit
end tell


'''
args = [item for x in [("-e",l.strip()) for l in applescript.split('\n') if l.strip() != ''] for item in x]
proc = subprocess.Popen(["osascript"] + args ,stdout=subprocess.PIPE )
progname = proc.stdout.read().strip()
sys.stdout.write(str(progname))


[jay (support)]

There's nothing I can see that would keep that script from running (note that there are no 'tell application IndigoServer' sections in your AppleScript - that's what would have stopped working). If you upgraded to Catalina it might be a permission error with the script trying to run the AppleScript via the popen system call. How are you executing this script?

Note however that this has nothing to do with Indigo no longer supporting AppleScript.

If I understand your question correctly, I execute this Python 3.7 generated script, with embedded AppleScript (Script Editor 2.7), in a Indigo 7.4 Trigger Action using file action in the type:" Execute Script (Script or file action)". This posted script was working on Catalina OS OK. If it was not something that changed on the Indigo server or app update, then I just don't know why the AppleScript part of this posted Python script just stop working.

I now use the file action to run only the Python script to access Local Server to get needed Thermostat data and write data to a text.txt file. 2nd action is to send email to trigger Mail app rule to run the AppleScript to append data to a Numbers app spreadsheet. This works OK and I will stay with this method.

New to posting in Forums, but have been using Indigo app for several years. As requested, where do I find instructions how to move this Topic Post to a more appropriate forum.

[jay (support)]

If I understand your question correctly, I execute this Python 3.7 generated script, with embedded AppleScript (Script Editor 2.7), in a Indigo 7.4 Trigger Action using file action in the type:" Execute Script (Script or file action)". This posted script was working on Catalina OS OK. If it was not something that changed on the Indigo server or app update, then I just don't know why the AppleScript part of this posted Python script just stop working.

Indigo uses Python 2 (always has), not Python 3 because it was only with Catalina that Apple started shipping Python 3 with macOS. I don't see anything in the Python code though that wouldn't work in Python 2 so that's unlikely to be the problem (particularly if it was working before). The AppleScript that's embedded in the Python script doesn't reference Indigo at all, so that's not the problem.

I wonder if the file is getting created correctly (from the Python part of the script)? It could be that some permission problem is causing a failure to write to the file. You aren't catching any exceptions that might be coming from the file.open() or file.write() commands, so if it couldn't complete those operations the script would just stop and you wouldn't see anything else. I guess one way to tell would be to see if you see the entry in the Event Log window that's written just after you do the file operations.

New to posting in Forums, but have been using Indigo app for several years. As requested, where do I find instructions how to move this Topic Post to a more appropriate forum.

I was just letting you know that I moved the topic to the right spot - you don't need to do anything else.

Thanks for the response and review.

This is a event log entry when working OK.
Feb 12, 2020 at 6:12:08 AM
Z-Wave received "03-Thermostat" is heating
Z-Wave received "03-Thermostat" fan is on
Trigger HVAC Energy Usage-ON
Script Heat - True Cool - False Temp - 39 Mode - Heat Supply - 73.5 Return - 66.9 Thermostat - 72.5

Also, all my other Home security Triggers using same method is not working. The Python part of the script works OK to produce the file. And, a separate standalone AppleScript with same code, works OK to read and execute functions with the file data.

Do you know of members who use Python script with embedded AppleScript, that does not call the Indigo Server, that works OK? If yes, and if you know of investigating tests, please advise. . If no, I will rewrite my scripts to work around this problem.

Try changing your AppleScript to do something small and easy to verify -- see if that executes. Are you running this script as an external file (i.e. the script action runs a file you select) or embedded into the Action itself?

Try changing your AppleScript to do something small and easy to verify -- see if that executes.

This is a Home Security script, with 3 AppleScript calls, is setup as Action embedded python script that monitors sensors and send alarms via Apple text message to me or local 911 operator.. I also tested an embedded python script with embedded AppleScript to activate Apple Script Editor to run a file that just has 1 line to log "Hello World". . Both had same non working results. It appears on my system, that any Python script referencing an AppleScript embedded or selected file, is not working.

Code: Select all

import subprocess, sys

doors = ["16-Front Door Sensor" , "17-Garage Door Sensor" , "18-Back Door Sensor" , "19-Breezeway Door Sensor" ,
"20-Z-wave Motion Sensor" , "04-Front Door Lock" , "05-Garage Door Opener"]
scan = 0
file = open("/Users/WilliamBoatright/Documents/Indigo/AwayArmed.txt","w")

for cnt , x in enumerate(doors):
        lamp = indigo.devices[x]   
        if hasattr(lamp, "onState"):
                isOn = lamp.onState
        if isOn == True and cnt+1 < 6:
                file.write(str(x))
                scan = 1
                break
        if isOn == False and cnt+1 > 5:
                file.write(str(x))
                scan = 1
                break

file.close()
indigo.server.log("cnt= " + str(cnt+1) + " Device= " + x + " Status=" + str(isOn) + " scan= " + str(scan))

if cnt + 1 < 6:
   applescript="""
      set newFile to ("Macintosh HD:Users:XXXXXXXXXXXXX:Documents:Indigo:AwayArmed.txt")
      set theFileContents to paragraphs of (read file newFile)
   
      tell application "Messages"
         set theNumber to buddy "+xxxxxxxxxx" of service "E:xxxxx@xxx.com"
         send "AWAY ARMED ALERT - " &  theFileContents & " Is Open" to theNumber
      end tell
      """
   args = [item for x in [("-e",l.strip()) for l in applescript.split("\n") if l.strip() != ""] for item in x]
   proc = subprocess.Popen(["osascript"] + args ,stdout=subprocess.PIPE )
   progname = proc.stdout.read().strip()
   sys.stdout.write(str(progname))

if cnt + 1 > 5:
   applescript="""
      set newFile to ("Macintosh HD:Users:xxxxxxxxxxxxx:Documents:Indigo:AwayArmed.txt")
      set theFileContents to paragraphs of (read file newFile)
   
      tell application "Messages"
         set theNumber to buddy "+xxxxxxxxxx" of service "E:xxxx@xxx.com"
         send "AWAY ARMED ALERT - " &  theFileContents & " Is Open" to theNumber
      end tell
      """
   args = [item for x in [("-e",l.strip()) for l in applescript.split("\n") if l.strip() != ""] for item in x]
   proc = subprocess.Popen(["osascript"] + args ,stdout=subprocess.PIPE )
   progname = proc.stdout.read().strip()
   sys.stdout.write(str(progname))

if scan == 0:
   applescript="""   
      tell application "Messages"
         set theNumber to buddy "+1xxxxxxx7" of service "E:xxxxx@xxx.com"
         send "AWAY ARMED CONFIRMED - All Doors are closed" to theNumber
      end tell
      """
   args = [item for x in [("-e",l.strip()) for l in applescript.split("\n") if l.strip() != ""] for item in x]
   proc = subprocess.Popen(["osascript"] + args ,stdout=subprocess.PIPE )
   progname = proc.stdout.read().strip()
   sys.stdout.write(str(progname))

'''
#if scan > 0:
   applescript="""   
      set newFile to ("Macintosh HD:Users:WilliamBoatright:Documents:Indigo:AwayArmed.txt")
      set theFileContents to paragraphs of (read file newFile)

      tell application "Messages"
         set theNumber2 to buddy "911" of service "E:xxxxx@xxx.com"
         send "POSSIBLE INTRUDER ALARM at xxxxxxxxxxxx xxx xxx xxx Road, xxxxx xx xxxxx  - Activated at " & theFileContents & ". Front Door entry code is xxxx - My Cell No: xxxxxxxxxxx7" to theNumber2
      end tell
      """
   args = [item for x in [("-e",l.strip()) for l in applescript.split("\n") if l.strip() != ""] for item in x]
   proc = subprocess.Popen(["osascript"] + args ,stdout=subprocess.PIPE )
   progname = proc.stdout.read().strip()
   sys.stdout.write(str(progname))
'''

Are you running this script as an external file (i.e. the script action runs a file you select) or embedded into the Action itself?

I use to use embedded into Action(self) and worked OK. But I got few time out errors because script took too long to run. I switched to run an external file you select and that worked OK until lately..

[jay (support)]

I just tried this Python script as an embedded Python script action, which has the simplest AppleScript I could think of:

Code: Select all

import subprocess, sys

applescript = 'say "testing"'
args = [item for x in [("-e",l.strip()) for l in applescript.split('\n') if l.strip() != ''] for item in x]
proc = subprocess.Popen(["osascript"] + args ,stdout=subprocess.PIPE )


I tried this on both Mojave and Catalina and it worked fine from Indigo 7.4.1. Now, here's the trick: I when you add a tell application "Numbers" to the script, that's where the trouble begins. For instance:

Code: Select all

import subprocess, sys

applescript = '''tell application "Numbers"
   say "testing from numbers"
end tell'''
args = [item for x in [("-e",l.strip()) for l in applescript.split('\n') if l.strip() != ''] for item in x]
proc = subprocess.Popen(["osascript"] + args ,stdout=subprocess.PIPE )

You should be prompted by Catalina to allow Indigo Server to control the application "Numbers":

grab6.png (58.71 KiB) Viewed 2238 times

When prompted, you should approve it. This is because of the new Catalina "security" features which are further locking down the OS. Unfortunately, we've seen several cases where the OS doesn't actually prompt you but just fail silently. I think we *might* have a hackaround for this in the next Indigo release (7.4.2), but no ETA on that. Anyway, if you get that dialog and you accept it, maybe your scripts will start working again.

I'd propose that you try a slightly different approach: have your script continue to write the file. But rather than try to execute the AppleScript from there, create a folder action in Automator that executes the AppleScript. So Indigo would only be responsible for writing the file. Automator (via the folder action) would be responsible for executing the AppleScript, where you'd be a lot less likely to run up against the (silly) Catalina "security" features.

Thanks very much for your help. This is what I like about coding, you're always learning.

I've found the culprit. It is a part of the AppleScript code that works OK on Python 3, but not on Python 2.
Last question. As you commented, Python 2 in Apple products is on its way out,. Do you plan to convert to Python 3 or later version?

[matt (support)]

See this thread for a discussion on it.

[jay (support)]

I've found the culprit. It is a part of the AppleScript code that works OK on Python 3, but not on Python 2.

Can you identify the exact lines of code? I'm trying to collect up Python 2 vs 3 code differences so that we'll have a good head start on helping users avoid pitfalls when we do upgrade. Thanks.

I know now I was mistaken about Indigo's support of AppleScript. As stated it was a Python version issue. The first code I posted was not the last version of the code I was using. I had modified the code, using Python 3, to populate custom formulae to a Numbers spreadsheet. As this code works with Python 3, and as, I found out, it did not work with Python 2. In Python 3, I couldn't use the formula copied from spreadsheet.as a string. Example: "=IF(B44= "OFF" ,A44−A43, "" )". The quotation marks in original formula, as is, would not work in Python 3. I found online method to use quotation marks within a string. As the formula would be populated to a new last row in the spreadsheet, the resulting formula was "=IF(B" & rowCount & "=" & OFF & "," & "A" & rowCount & "−" & "A" & rowcount2 & "," & END1 & ")".

Code: Select all

set OFF to " \"OFF\" "-- for "OFF" in the formula
set END1 to " \"\" " -- for "" in the formula

This is the last embedded AppleScript draft I believe Python 2 could not interrupt the forward slash "\" with quotation marks "".
The code adds a row to spreadsheet each time the HVAC unit turns ON or turns OFF.

Code: Select all

ell application "Numbers"
   activate
   open filePath
   
   tell table "HVAC-Energy-Usage" of sheet "Heat" of document 1
      add row below last row
      --set rowCount to last row's address
      set rowCount to row count
      set rowcount2 to rowCount - 1
      set OFF to " \"OFF\" "
      set ON1 to "\"ON\" "
      set END1 to " \"\" "
      
      tell column "A"
         set value of last cell to rightNow
      end tell
      
      tell column "B"
         set value of last cell to status
      end tell
      
      tell column "C"
         set value of last cell to "=IF(B" & rowCount & "=" & OFF & "," & "A" & rowCount & "−" & "A" & rowcount2 & "," & END1 & ")"
      end tell
      
      tell column "D"
         set value of last cell to "=IF(B" & rowCount & "=" & ON1 & "," & "A" & rowCount & "−" & "A" & rowcount2 & "," & END1 & ")"
      end tell
      
      tell column "E"
         set value of last cell to "=IF(C" & rowCount & "=" & END1 & "," & END1 & ",DUR2HOURS(C" & rowCount & "))"
      end tell
      
      tell column "F"
         set value of last cell to "=IF(C" & rowCount & "=" & END1 & "," & END1 & ",E" & rowCount & "×$B$1)"
      end tell
      
      tell column "G"
         set value of last cell to "=IF(C" & rowCount & "=" & END1 & "," & END1 & ",F" & rowCount & "×$B$2)"
      end tell
      
      tell column "H"
         set value of last cell to "=IF(C" & rowCount & "=" & END1 & "," & END1 & ",(($D$1×0.75)×E" & rowCount & ")×B$2)"
      end tell
      
      tell column "I"
         set value of last cell to outdoorTempF
      end tell
      
      tell column "J"
         set value of last cell to mode
      end tell
      
      tell column "K"
         set value of last cell to supplyTempF
      end tell
      
      tell column "L"
         set value of last cell to returnTempF
      end tell
      
      tell column "M" # set "Delta-T tempF" formula
         set value of last cell to "=IF(B" & rowCount & "=" & OFF & "," & "K" & rowCount & "−" & "L" & rowcount2 & "," & END1 & ")"
      end tell
      
      tell column "N" # set "Thermostat tempF"
         set value of last cell to thermostatTempF
      end tell
      
      tell row rowCount
         set alignment to center
         set vertical alignment to center
      end tell
      
   end tell
   
   save every document with saving
   --quit
end tell

[jay (support)]

The quotation mark issue is an issue for any version of Python. One simple workaround is to quote the entire string in something else. For instance, all of these are the same as shown in Python 2:

Code: Select all

Mac-mini:Volumes jay$ python
Python 2.7.16 (default, Jan 27 2020, 04:46:15)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> using_single_quotes = '=IF(B44= "OFF" ,A44−A43, "" )'
>>> using_multiline_single_quotes = '''=IF(B44= "OFF" ,A44−A43, "" )'''
>>> using_multiline_double_quotes = """=IF(B44= "OFF" ,A44−A43, "" )"""
>>> escaping_double_quotes = "=IF(B44= \"OFF\" ,A44−A43, \"\" )"
>>> escaping_double_quotes == using_multiline_double_quotes == using_multiline_single_quotes == using_single_quotes
True
>>>

and in Python 3:

Code: Select all

Mac-mini:Volumes jay$ python3
Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> using_single_quotes = '=IF(B44= "OFF" ,A44−A43, "" )'
>>> using_multiline_single_quotes = '''=IF(B44= "OFF" ,A44−A43, "" )'''
>>> using_multiline_double_quotes = """=IF(B44= "OFF" ,A44−A43, "" )"""
>>> escaping_double_quotes = "=IF(B44= \"OFF\" ,A44−A43, \"\" )"
>>> escaping_double_quotes == using_multiline_double_quotes == using_multiline_single_quotes == using_single_quotes
True
>>>

But neither will accept the string as you originally showed, doesn't work in either because of the mismatch in quotes caused by non-escaping or by the incorrect surrounding quotes:

Code: Select all

Mac-mini:Volumes jay$ python
Python 2.7.16 (default, Jan 27 2020, 04:46:15)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> original_string = "=IF(B44= "OFF" ,A44−A43, "" )"
  File "<stdin>", line 1
    original_string = "=IF(B44= "OFF" ,A44−A43, "" )"
                                   ^
SyntaxError: invalid syntax
>>> ^D
Mac-mini:Volumes jay$ python3
Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> original_string = "=IF(B44= "OFF" ,A44−A43, "" )"
  File "<stdin>", line 1
    original_string = "=IF(B44= "OFF" ,A44−A43, "" )"
                                   ^
SyntaxError: invalid syntax
>>>

So that's just a straight-up quoting issue which isn't Python version specific.

The issue seems to be that both Python and AppleScript escape quotes in strings using the backslash to precede the escaped character. So when you embed that AppleScript inside a Python script that has escaped quotes (as shown on the lines that set OFF, ON1, and END1), Python escapes the quotes before it gets to AppleScript so the AppleScript fails because it has erroneous quotes.

The easiest solution is to just not embed the AppleScript in your Python script at all, but rather save it as a compiled script and have the Python script use the osascript to execute the AppleScript file rather than attempting to pass the AppleScript to it. Or, as I suggested, separate them entirely and have Python write the file, and have a folder watcher run the AppleScript to process it.