Twitter Scraping in 2023

Posted by Vlad Mishkin | March 8, 2023

Twitter is one of the most popular social media platforms, with millions of users tweeting and sharing their thoughts and opinions every day. As a result, Twitter has become a valuable source of data for businesses, researchers, and developers who want to analyze user behavior, sentiment, and trends.

The best way to get the data from Twitter is to use their official API. However, Twitter's API is becoming more and more restrictive: you can only get a limited amount of data from Twitter using their API. On top of that even the lowest tier of the API will require a monthly payment soon. That's why many people are looking for ways to scrape Twitter without these restrictions.

Here we've collected a few methods of scraping Twitter data without using the official API.

How to Scrape Twitter Tweets

For tweets, you can use the private search API. This API is not officially documented, but it's still working. You can use it to search for tweets by keyword, hashtag, or username. The API returns a JSON with the tweets, which you can then parse and save.

The usage of this API consists of 2 steps:

  • Getting the quest token
  • Making the search request

Getting the quest token

This search API is intended to be used by non-authenticated users, but it still requires a token to be sent with the request. This token is called the quest token. You can get the token by making a POST request to the following URL: This request itself requires a bearer token that is static and always the same for these endpoints.

import http.client
import urllib
import json

conn = http.client.HTTPSConnection("")

# 1. Getting the quest token

# this header is static and used for both requests
auth_header = 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA'

api_params = {
  'api_key': 'test-api-key',
  'js': 'false',
  'timeout': 25000,
  'url': f'',
  'headers[Authorization]': auth_header
conn.request("POST", f"/html?{urllib.parse.urlencode(api_params)}")

res = conn.getresponse()
json_raw ="utf-8")
json_object = json.loads(json_raw)
quest_token = json_object['guest_token']

print("Using quest token: " + quest_token)

Making the search request

Now we can use the quest token to make the search request. The search request is a GET request to the following URL: Also, we still need to add the same bearer headers as in the previous request.

twitter_params = {
  'q': 'test',
  'count': 100,
  # 'cursor': '',
  'include_want_retweets': 1,
  'include_quote_count': 'true',
  'include_reply_count': 1,
  'tweet_mode': 'extended',
  'include_entities': 'true',
  'include_user_entities': 'true',
  'simple_quoted_tweet': 'true',
  'tweet_search_mode': 'live',
  'query_source': 'typed_query'

api_params = {
  'api_key': 'test-api-key',
  'js': 'false',
  'timeout': 25000,
  'url': f'{urllib.parse.urlencode(twitter_params)}',
  'headers[Authorization]': auth_header,
  'headers[X-Guest-Token]': quest_token
conn.request("GET", f"/html?{urllib.parse.urlencode(api_params)}")

res = conn.getresponse()
json_raw ="utf-8")
json_object = json.loads(json_raw)

print(json.dumps(json_object, indent=1))

Advanced search

In this example we've just used the word "test", but you can search for any keyword, hashtag, or username, and use all the features available on Twitter search All you need to do, is to construct the search query using the advanced search builder and then use the query in the "q" parameter of the search request. This will allow you to search the tweets by:

  • Keywords
  • Phrases
  • Negative keywords
  • Hashtags
  • Languages
  • Usernames
  • Mentions
  • Replies
  • Dates
  • Minimum replies
  • Minimum retweets
  • Minimum replies


Here is an interactive example of this code:

Table of contents

You might also enjoy

Web Scraping with Python

Web Scraping with Python

A tutorial about web scraping in Python with examples. We will take a look at the most popular Python tools for web scraping: Requests, BeautifulSoup, lxml and others.

Posted by Vlad Mishkin | February 5, 2023
How to Scrape Instagram in 2023

How to Scrape Instagram in 2023

How to build an Instagram scraper without getting blocked

Posted by Vlad Mishkin | February 5, 2023