day 3 ] Requests

1. 공공데이터 api 요청

2. 데이터를 params or data(Dictionary)형태로 보냄
1) get 방식, 
import requests
resp = requests.request("get", "http://httpbin.org/get", params={"key":"value"})

2) post 방식
import requests
resp = requests.request("post", "http://httpbin.org/post", data={"key":"value2"})


3. JSON
1) json 형태로 바꾸는 방법
import json
result = json.loads(resp.content)

2) 특정 값 뽑아오는 방

3) 다운로드 함수
- resp.raise_for_status() 를 사용해야 오류가 났을 때 except이 발생한다.
import requests

header = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}

def download(url, params={}, retries=3):
    resp = None
    
    try:
        resp = requests.get(url, params=params, headers=header)
        resp.raise_for_status()
    except requests.exceptions.HTTPError as e:
        if 500 <= e.response.status_code < 600 and retries > 0:
            print(retries)
            resp = download(url, params, retries-1)            
        else:
            print(e.response.status_code)
            print(e.response.reason)
            print(e.request.headers)            
    return resp

4. 공공데이터포털 API 사용하기
- 키 값이 두번 quote되기 때문에 한번 unquote해서 넘기기
url = "http://openapi.airkorea.or.kr/openapi/services/rest/ArpltnInforInqireSvc/getMsrstnAcctoRltmMesureDnsty"
params = {
"serviceKey":"u1rFHz3u6gdgSXAKSNbTm7jGXSXM1Tcs2CgDniQaahCajzFoFqvXk5UMdf1if9d0mbR8Lje%2FC0PLOygux%2BzqXQ%3D%3D",    
"numOfRows":"5",
"pageNo":"1",
"stationName":"종로구",
"dataTerm":"DAILY",
"ver":"1.3",
}

params['serviceKey'] = requests.utils.unquote(params['serviceKey'])
html = download(url, params)
html.text


5. 응용
- post 방식에서는 form-data 형태로 보내면 결과를 얻을 수 있다.

1) post형태의 downloads함수
- params가 data로 바뀌고 get이 post로 바뀌었다.
import requests

def download2(url, params={}, retries=3):
    header = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'}
    resp = None
    
    try:
        resp = requests.post(url, data=params, headers=header)
        resp.raise_for_status()
    except requests.exceptions.HTTPError as e:
        if 500 <= e.response.status_code < 600 and retries > 0:
            print(retries)
            resp = download(url, params, retries-1)            
        else:
            print(e.response.status_code)
            print(e.response.reason)
            print(e.request.headers)            
    return resp

2) 결과
[input]
url = "http://pythonscraping.com/pages/files/processing.php"
params = {"firstname":"sion", "lastname":"test"}
result_form = download2(url, params)
print(result_form.text)
print(result_form.request.body)

[output]

Hello there, sion test!

firstname=sion&lastname=test

 

1. 공공데이터 api 요청 

2. 데이터를 params or data(Dictionary)형태로 보냄 
1) get 방식,  
import requests 
resp = requests.request("get", "http://httpbin.org/get", params={"key":"value"}) 

2) post 방식 
import requests 
resp = requests.request("post", "http://httpbin.org/post", data={"key":"value2"}) 


3. JSON 
1) json 형태로 바꾸는 방법 
import json 
result = json.loads(resp.content) 

2) 특정 값 뽑아오는 방 

3) 다운로드 함수 
- resp.raise_for_status() 를 사용해야 오류가 났을 때 except이 발생한다. 
import requests 

header = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'} 

def download(url, params={}, retries=3): 
    resp = None 
     
    try: 
        resp = requests.get(url, params=params, headers=header) 
        resp.raise_for_status() 
    except requests.exceptions.HTTPError as e: 
        if 500 <= e.response.status_code < 600 and retries > 0: 
            print(retries) 
            resp = download(url, params, retries-1)             
        else: 
            print(e.response.status_code) 
            print(e.response.reason) 
            print(e.request.headers)             
    return resp 

4. 공공데이터포털 API 사용하기 
- 키 값이 두번 quote되기 때문에 한번 unquote해서 넘기기 
url = "http://openapi.airkorea.or.kr/openapi/services/rest/ArpltnInforInqireSvc/getMsrstnAcctoRltmMesureDnsty" 
params = { 
"serviceKey":"u1rFHz3u6gdgSXAKSNbTm7jGXSXM1Tcs2CgDniQaahCajzFoFqvXk5UMdf1if9d0mbR8Lje%2FC0PLOygux%2BzqXQ%3D%3D",     
"numOfRows":"5", 
"pageNo":"1", 
"stationName":"종로구", 
"dataTerm":"DAILY", 
"ver":"1.3", 
} 

params['serviceKey'] = requests.utils.unquote(params['serviceKey']) 
html = download(url, params) 
html.text 

4. 쿠키
1) 쿠키 로그인
- 쿠키도 dictionay형태로 넘기면 된다.
[input]
url = "http://pythonscraping.com/pages/cookies/welcome.php"
params = {"username":"1234", "password":"password"}
result = requests.get(url, cookies={'loggedin': '1', 'username': '1234'})
result.text

[output]
'\n

Welcome to the Website!

\nYou have logged in successfully! 
profile.php">Check out your profile!'

 

2) Session

-  쿠키로 데이터를 포함시킨 후 아래를 호출하면 Session으로 쿠키가 저장된다.

session = requests.Session()
html = session.post(url)
print(html.request.body)

print(html.text)

3) 네이버 로그인 힌트
#네이버 로그인 힌트
#md5 => rsa => 공개키 => 등등등, 리다이렉션을 멈추고 작업을 한다. javascript에서 암호화 하는 부분을 찾아서 


5. 응용 
- post 방식에서는 form-data 형태로 보내면 결과를 얻을 수 있다. 

1) post형태의 downloads함수 
- params가 data로 바뀌고 get이 post로 바뀌었다. 
import requests 

def download2(url, params={}, retries=3): 
    header = {'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36'} 
    resp = None 
     
    try: 
        resp = requests.post(url, data=params, headers=header) 
        resp.raise_for_status() 
    except requests.exceptions.HTTPError as e: 
        if 500 <= e.response.status_code < 600 and retries > 0: 
            print(retries) 
            resp = download(url, params, retries-1)             
        else: 
            print(e.response.status_code) 
            print(e.response.reason) 
            print(e.request.headers)             
    return resp 

2) 결과 
[input] 
url = "http://pythonscraping.com/pages/files/processing.php" 
params = {"firstname":"sion", "lastname":"test"} 
result_form = download2(url, params) 
print(result_form.text) 
print(result_form.request.body)

[output] 

Hello there, sion test!

firstname=sion&lastname=test


3) 교보문고 데이터 가져오기
import requests

url = "http://www.kyobobook.co.kr/search/SearchCommonMain.jsp"
param = {
    "vPstrCategory": "TOT",
    "vPstrKeyWord": "abc",
    "vPplace": "top"
}
result = requests.post(url,data=param)
result.text

4) 네트워크에서 post 데이터 찾기

- Query String Parameters 또는 Form Data를 파라미터로 보내주면 된다