get data from website - which authentication for requests?

Afternoon (in this part of the world)

What method of requests authentication do i need please? Is it dependant on the website?

I'm trying to get data on the current temperature etc of one of my kilns. The controller sends the information to a web server and can be accessed online. There is no api for the controller. The data appears in the source code for the website but i can't log in using "requests". I am not sure which authentication to use, or how to see what authentication the web site needs (very new at this type of thing). Once I can get the source code, i can extract the data that i need from the text and save into variables - happy with that side of things.

The username and password are real as its only reading data (not my usual password - honest).

This is what I have to get the source code. Any assistance to a python beginner would be great thanks.

Normal log in is here https://theplant.herokuapp.com/kiln_controllers/

Code: Select all

import requests

from requests.auth import HTTPBasicAuth

r = requests.get('https://theplant.herokuapp.com/kiln_controllers/3720', auth=HTTPBasicAuth('mat@planware.co.uk' , '123456789012'))

print(r)

print(r.content)

Yeah, that's not going to work. The built-in authentication requests you're trying to use is only for when the server itself has enabled authentication. Not for when the website has implemented it's own login screen and authentication pages.

You're going to need to do real page scraping to get past that login page. This is not a simple operation. The best library for doing this is BeautifulSoup. Here's a how-to on using it, but I warn you, it's still quite complicated.

https://codeyogi.co.uk/2020/01/21/how-t ... th-python/

Thanks. Ok, i'm up for a challenge so will at least give it a read. I did wonder if that would be the case - these things are never simple

Thanks Joe, got the variables extracted, just go to integrate now. Down hill from here.

Cheers


Mat

Good deal.