Search Returns 404

[Abolfazl]

I am using the following code to login to Twitter and perform searches

from twitter.search import Search

ct0 = 'XXXXXX'
auth_token = 'XXXXXX'
search = Search(cookies={"ct0": ct0, "auth_token": auth_token}, save=True, debug=1)

search.run(queries=[{'category': 'Latest','query': f'SEARCH QUERY'}], limit=150, latest=True)

I did some digging and it looks like the error is because https://twitter.com/i/api/graphql/nK1dw4oV3k4w5TdtcAdSww/SearchTimeline returns 404. However when I access directly from browser, it returns 403 (correct)

This script was working fine up until yesterday. The login is working fine but what can I do to resolve this issue?

[przemyslslaw]

I think the search endpoint doesn't work since April 24-25. Does anyone have any thoughts how to resolve this?
@trevorhobenshield, would you mind having a look?

[satinfive]

I think the search endpoint doesn't work since April 24-25. Does anyone have any thoughts how to resolve this? @trevorhobenshield, would you mind having a look?

I've been doing some research. What I've beeing noticing is that they require now the client-transaction-id header as mandatory, if not, they give a 404. Also if you send too many request after a 200 with the header, it also returns a 404; it seems to be some kind of time window to rate limit the requests.

This pkg seem a bit inactive... Others alternatives are that you go for scraping or that you compute by yourself this client-transaction-id header to "sign" your requests.

https://github.com/iSarabjitDhiman/XClientTransaction/tree/master

[Abolfazl]

Thanks for the link @satinfive . I used the code from the Quickstart and modifying the x-client-transaction-id fixed my issue. Just be sure to generate a new transaction ID for each transaction.

[TMDR]

could you please share a snippet on how could be done with this library? this library doesn't seem to support passing headers, so I assume I have to add it to the source code.

[SakiSakiSakiSakiSaki]

This pkg seem a bit inactive...

I pinged the author in my tickets and even sent an email, no response. I tried looking at other similar packages and I've yet to find any that has a similar users_by_ids() method that allows you to efficiently feed in a list of 200 IDs to save rate-limiting. If anyone has any leads, don't be afraid to ping me about it.

[Abolfazl]

@SakiSakiSakiSakiSaki Here is the full code to use the UsersByRestIds endpoint. You will need to install https://github.com/iSarabjitDhiman/XClientTransaction/

import requests

# Transaction ID Generator
from x_client_transaction.utils import handle_x_migration
from x_client_transaction import ClientTransaction
import urllib.parse
import json

from pprint import pprint



# ###################################
# ## START GENERATE TRANSACTION ID ##
# ###################################
headers = {"Authority": "x.com",
           "Accept-Language": "en-US,en;q=0.9",
           "Cache-Control": "no-cache",
           "Referer": "https://x.com",
           "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
           "X-Twitter-Active-User": "yes",
           "X-Twitter-Client-Language": "en"}

session = requests.Session()
session.headers = headers
response_twit = handle_x_migration(session)

endpoint_url = "https://x.com/i/api/graphql/GD4q8bBE2i6cqWw2iT74Gg/UsersByRestIds"
http_method = "GET"
endpoint_url_path = urllib.parse.urlparse(url=endpoint_url).path

ct = ClientTransaction(response_twit)
transaction_id = ct.generate_transaction_id(method=http_method, path=endpoint_url_path)
print(f"{transaction_id=}")
# #################################
# ## END GENERATE TRANSACTION ID ##
# #################################
ct0 = ''
auth_token = ''
cookies = {
    'auth_token': auth_token,
    'ct0': ct0,
}

headers = {
    'accept': '*/*',
    'accept-language': 'en-US,en;q=0.9',
    'authorization': 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA',
    'cache-control': 'no-cache',
    'content-type': 'application/json',
    'pragma': 'no-cache',
    'priority': 'u=1, i',
    'referer': 'https://x.com/',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36',
    'x-csrf-token': ct0,
    'x-twitter-active-user': 'yes',
    'x-twitter-auth-type': 'OAuth2Session',
    'x-twitter-client-language': 'en',
}


# Set Transaction ID
headers['x-client-transaction-id'] = transaction_id

_userIds = [1177149846509125632]
variables = {'userIds': [str(x) for x in _userIds]}
features = {
    'responsive_web_graphql_exclude_directive_enabled': True,
    'verified_phone_label_enabled': False,
    'responsive_web_graphql_skip_user_profile_image_extensions_enabled': False,
    'responsive_web_graphql_timeline_navigation_enabled': True,
}

params = {'variables': variables, 'features': features}
params = urllib.parse.urlencode({k: json.dumps(v, separators = (',', ':')) for k, v in params.items()}, quote_via = urllib.parse.quote)
response = requests.get(
    endpoint_url,
    params=params,
    cookies=cookies,
    headers=headers,
).json()['data']['users']

pprint(response)

Just set your ct0 and auth_token variables to your profile. I tested and this works for me. Just be sure to add your own checks that youre not over the ID limit

[SakiSakiSakiSakiSaki]

Just set your ct0 and auth_token variables to your profile. I tested and this works for me. Just be sure to add your own checks that youre not over the ID limit

Wow thanks for sharing this, can easily replace my code with a functionalized version of this. So seems like we're relying on requests for endpoint instead of twitter-api-client. I wonder what broke on the package-side of things...

[Abolfazl]

The only change that needs to be done to twitter-api-client is for the x-client-transaction-id header to be updated when the request is made.

[satinfive]

Just set your ct0 and auth_token variables to your profile. I tested and this works for me. Just be sure to add your own checks that youre not over the ID limit

Wow thanks for sharing this, can easily replace my code with a functionalized version of this. So seems like we're relying on requests for endpoint instead of twitter-api-client. I wonder what broke on the package-side of things...

Yes, as @Abolfazl mentioned, you need to modify the package to add this new header on each request. I think X required this header progressively as a mechanism to protect it from unauthorized clients.

You could even adapt the logic of the snippet using httpx instead of requests as this pkg does.

[SakiSakiSakiSakiSaki]

@Abolfazl You left your bearer token in your code block.

[Abolfazl]

@SakiSakiSakiSakiSaki It's hardcoded https://news.ycombinator.com/item?id=35549764

[SakiSakiSakiSakiSaki]

What a silly website. Thanks.