Why is python showing empty brackets for my web scraping project?

Question

I thought it might be because the site asks that you login, so I went to cURL converter and got the cookies and header information. I also thought it could've been an outdated version of python, I installed the latest one and started a new project. I pip install lxml, bs4, and requests to be sure.

Here's the code:

from bs4 import BeautifulSoup
import requests

html_text = requests.get('https://newsfilter.io/latest/news', cookies=cookies, headers=headers).text
soup = BeautifulSoup(html_text, 'lxml')
jobs = soup.find_all('div', class_='sc-dnqmqq bxsfdc')

print(jobs)

Answer 1

Using the dev tool of your browser, look at the request made to the server.

Look at the html, and check that it is mostly empty andd then notice that it makes a json request to https://static.newsfilter.io/landing-page/articles-latest-news.json

It is just a matter to get that and parse it.

import requests
    
resp = requests.get('https://static.newsfilter.io/landing-page/articles-latest-news.json')
if resp.status_code == 200:
    for headlines in resp.json():
        print(f'{headlines["publishedAt"]}: {headlines["title"]}')