Realtor.com - element changes

[crushingbear]

Hi there,

Can you give me some input here? I am trying to parse the data-url elements within the pages.

    soup = bs(requests.get(requestUrl + "/pg-" + str(page), headers=userAgent).text, "html.parser")
    #print(soup)
    pagedList = soup.findAll("div", {"class": "data-wrap"})
    #pagedList = soup.findAll('div')
    numberrec = len(pagedList)
    print(numberrec)
    for propertyTag in pagedList:
    	try:
    		print(urls.append(a.attrs['data-url']))

    		#print(propertyTag.find("data-url")['href'].text)
	#		print(propertyTag.find(attrs={'data-url':'href'}).text)
    #		#print("Start tag:", tag)
		except:
    		print("error")

[pangrr]

Hi, I did expect anyone gets interested in this ancient repo. So what a surprise to me.

I haven't used python for years, so I don't remember the implementations.

Will it be helpful to solve your problem with javascript?

[crushingbear]

mainly looking for a solution in python, but I should be able to translate from javascript back to python.

[pangrr]

What exactly do you want as output?

[crushingbear]

just wanting to parse out the data-url URLs elements.

…

On Wed, May 15, 2019 at 10:15 AM pangrr ***@***.***> wrote: What exactly do you want as output? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=ADL6Z43AYWY3XZESQVQ37JLPVQSINA5CNFSM4HND35O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVO7VZY#issuecomment-492698343>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADL6Z43OFNJSQSMCMUWRYZTPVQSINANCNFSM4HND35OQ> .

[pangrr]

data-url is an attribute which appears in multiple elements in a page.
Do you want the output to be those elements? Or the data-url values?
What are the pages you are interested in?

[crushingbear]

yes.
Pages in: https://www.realtor.com/realestateandhomes-search/Houston_TX/sby-10

[pangrr]

Do you want the output to be those elements? Or the data-url values?

[crushingbear]

yes. the data-url values.

…

On Wed, May 15, 2019 at 10:46 AM pangrr ***@***.***> wrote: Do you want the output to be those elements? the data-url values? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=ADL6Z426M4IIO5JU54FU2IDPVQV3TA5CNFSM4HND35O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVPCWOI#issuecomment-492710713>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADL6Z4ZUXYRPKYG5FWVIUG3PVQV3TANCNFSM4HND35OQ> .

[crushingbear]

i solved my problem. i guess taking a break from it helps. i had to reference as unicode to obtain data.

    soup = bs(requests.get(requestUrl + "/pg-" + str(page), headers=userAgent).text, "html.parser")
    #print(soup)
    pagedList = soup.findAll("div", {"class": "data-wrap"})
    #pagedList = soup.findAll('div')
    numberrec = len(pagedList)
    #print(numberrec)
    for propertyTag in pagedList:
    	try:
    		dataurl = propertyTag[u'data-url']
    		print(dataurl)
    		#print(propertyTag.attrs)
    		#print(u'data-url'.text)
			#print(propertyTag.find(u"data-url").text)
    		#print(propertyTag.find("data-url")['href'].text)
	#		print(propertyTag.find(attrs={'data-url':'href'}).text)
    #		#print("Start tag:", tag)
		except:
    		print("error")

[pangrr]

Nice to hear that. Let me know if you need anything.

…

On Wed, May 15, 2019 at 1:40 PM crushingbear ***@***.***> wrote: i solved my problem. i guess taking a break from it helps. i had to reference as unicode to obtain data. soup = bs(requests.get(requestUrl + "/pg-" + str(page), headers=userAgent).text, "html.parser") #print(soup) pagedList = soup.findAll("div", {"class": "data-wrap"}) #pagedList = soup.findAll('div') numberrec = len(pagedList) #print(numberrec) for propertyTag in pagedList: try: dataurl = propertyTag[u'data-url'] print(dataurl) #print(propertyTag.attrs) #print(u'data-url'.text) #print(propertyTag.find(u"data-url").text) #print(propertyTag.find("data-url")['href'].text) # print(propertyTag.find(attrs={'data-url':'href'}).text) # #print("Start tag:", tag) except: print("error") — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=ACHFQ7IFSQXN2XOD4NBDBYDPVRDHLA5CNFSM4HND35O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVPM3YA#issuecomment-492752352>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACHFQ7KT7D6TO76RFU4MAHDPVRDHLANCNFSM4HND35OQ> .