View unanswered posts | View active topics It is currently Sat Apr 19, 2014 1:34 pm



Reply to topic  [ 16 posts ]  Go to page: 1, 2  Next
 Proliphix Web Scraper Help 
Author Message

Joined: Nov 07, 2003
Posts: 209
Post Proliphix Web Scraper Help
I have two Proliphix thermostats that are in a remote location. Both report to the Proliphix Remote Management web site. I am trying to figure out how to scrape the data off that web site into Indigo variables. The issue I'm faced with is figuring out how to log in to the Proliphix site. The page, "https://access.proliphix.com/Login.php" requires a username/password response. I cannot figure out how to get curl to work in this case. Any help/ideas much appreciated. Thank you.


Sun Sep 20, 2009 4:29 pm
Profile
Site Admin
User avatar

Joined: Mar 19, 2008
Posts: 8356
Location: Austin, Texas
Post Re: Proliphix Web Scraper Help
HFTobeason wrote:I have two Proliphix thermostats that are in a remote location. Both report to the Proliphix Remote Management web site. I am trying to figure out how to scrape the data off that web site into Indigo variables. The issue I'm faced with is figuring out how to log in to the Proliphix site. The page, "https://access.proliphix.com/Login.php" requires a username/password response. I cannot figure out how to get curl to work in this case. Any help/ideas much appreciated. Thank you.


I'm not positive this will work, but in many apps that take a url with authentication you use the URL in this format:

https://username:password@hostname.com/whatever

_________________
Jay (Indigo Support)
Image


Sun Sep 20, 2009 5:56 pm
Profile WWW
User avatar

Joined: Nov 18, 2008
Posts: 2252
Location: Berkeley, CA, USA
Post 
Check the curl man page.


Sun Sep 20, 2009 9:48 pm
Profile

Joined: Nov 07, 2003
Posts: 209
Post 
Thank you both for your replies. I have tried the form
Code: Select all
curl -k -u UserName:PassWord https://access.proliphix.com/Login.php
with no success - curl returns the HTML of the login page, and does not seem to submit the UserName:PassWord credentials to get to the next page. From the reading I've done, it seems as if I may have multiple layers of problems here - even if I figure out how to get past the login page, the Proliphix site may require cookies, and the page I ultimately want to scrape is presented in a frame. Any further guidance most appreciated.


Mon Sep 21, 2009 8:26 am
Profile
User avatar

Joined: Nov 18, 2008
Posts: 2252
Location: Berkeley, CA, USA
Post 
HFTobeason wrote:...From the reading I've done, it seems as if I may have multiple layers of problems here - even if I figure out how to get past the login page, the Proliphix site may require cookies, and the page I ultimately want to scrape is presented in a frame. Any further guidance most appreciated.

Sorry, I didn't realize the application was that complex. I believe curl can do what you want, but you probably need someone who has a lot more experience with curl than you are likely to find here (on the Indigo forums). I suggest you try the curl mailing lists. Note there are also commercial providers of curl support. They are listed under support on the curl web site.

Please do share your your results.


Mon Sep 21, 2009 11:03 am
Profile

Joined: Mar 19, 2008
Posts: 489
Location: Saratoga, CA
Post 
This guy has a program for polling and graphing the data from a pholiphix: http://www.anders.com/projects/thermostat-graph/

I just had a brief look at it - he appears to be using a URL which polls the values by OID (in a direct way - not screen-scraping).

You should be able to modify his script, strip out all the graphing stuff, and just have it dump the values you're interested in to STDOUT. Or maybe use to help you figure out how to craft the right query with curl.


Mon Sep 21, 2009 11:49 am
Profile

Joined: Mar 19, 2008
Posts: 489
Location: Saratoga, CA
Post 
check this out - includes curl examples: http://rtc.rubyforge.org/svn/trunk/docs ... _R1_11.pdf


Mon Sep 21, 2009 11:59 am
Profile
User avatar

Joined: Nov 18, 2008
Posts: 2252
Location: Berkeley, CA, USA
Post 
seanadams wrote:This guy has a program for polling and graphing the data from a pholiphix: http://www.anders.com/projects/thermostat-graph/

I just had a brief look at it - he appears to be using a URL which polls the values by OID (in a direct way - not screen-scraping)...

I think HFTobeason's problem is a little different. He is not on the same network as his thermostats and needs to get the data via a web reflector (Proliphix Remote Management web site), thus the need for curl to log into the reflector web site first.

If he could punch a hole in his firewall then he could probably read the data directly, as you suggest. But, I assume he has discarded that idea for one reason or another.


Mon Sep 21, 2009 12:01 pm
Profile

Joined: Mar 19, 2008
Posts: 489
Location: Saratoga, CA
Post 
berkinet wrote:I think HFTobeason's problem is a little different. He is not on the same network as his thermostats and needs to get the data via a web reflector (Proliphix Remote Management web site), thus the need for curl to log into the reflector web site first.


Ah, ok. Well in that case it IS possible to handle cookie-based authentication using curl, with two successive commands. http://www.google.com/search?q=cookie+login+curl


Mon Sep 21, 2009 12:08 pm
Profile

Joined: Jul 09, 2008
Posts: 19
Post Why not use the API
Why not getting values directly from the thermostat ? I know they've got a web interface and an http API which you can direcltly call. The API is not public, but you can ask the support to send them to you. Ther're free for personal use.
Antoio


Wed Nov 04, 2009 2:33 pm
Profile

Joined: Sep 24, 2008
Posts: 2
Location: Oakland, CA
Post 
HFTobeason, did you ever find a way to get your data?

Sam

_________________
Sam Boutros
-----------------------
http://inthrma.com/


Thu Nov 05, 2009 3:38 pm
Profile WWW

Joined: Nov 07, 2003
Posts: 209
Post 
No, I've totally failed to figure this out.

For the record, I can't get directly to the thermostats because they're behind a modem/router which DOES NOT have a static IP, and there is no computer at that location to run a DynDNS-type client, nor does the modem/router have built-in DDNS capability.

I'm at a loss.


Thu Nov 05, 2009 8:30 pm
Profile
User avatar

Joined: Nov 18, 2008
Posts: 2252
Location: Berkeley, CA, USA
Post 
Don't give up hope just yet... You might try taking a look at iMacros.
The iOpus web site wrote:iMacros lets you record and replay repetitious work. iMacros programmatically interacts with any and all websites. It fills out forms and automates the download and upload of text, images, files and web pages. You can import or export data to/from using CSV & XML files, databases or any other source to and from web applications.

They claim to have a Mac version in beta (scroll to the bottom of the page). They have a free Firefox Add-In. I gave it a quick try and it did learn to log in to the Proliphix remote access web site. I didn't try, but I'd guess you could create a macro to login, select your thermostat and save the contents of the web page (or better, the source) to parse later.

If you used the Firefox Add-In you would probably need to figure out how to call the macro from an AppleScript - it could then continue to parse the saved HTML data after the macro finished.


Thu Nov 05, 2009 9:39 pm
Profile
User avatar

Joined: Nov 18, 2008
Posts: 2252
Location: Berkeley, CA, USA
Post 
BTW. If you don't know already, there is a Yahoo Group for Proliphix owners.

Since I also have an interest in this issue, I posted your query there. I'll let you know if there is any response.


Thu Nov 05, 2009 10:00 pm
Profile
Site Admin
User avatar

Joined: Jan 27, 2003
Posts: 13210
Location: Texas
Post 
I use the iMacros Firefox plug-in. It works pretty well. Every month I have it download various PDF statements from different Web sites. It won't be an issue with the Proliphix, but the only problem I have is Website pages changing and breaking my scripts (no fault of iMacros).

_________________
Image


Thu Nov 05, 2009 10:46 pm
Profile WWW
Display posts from previous:  Sort by  
Reply to topic   [ 16 posts ]  Go to page: 1, 2  Next

Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.   Template designed by STSoftware.