TSPD
TSPD is a WAF (Web Application Firewall) protection designed to prevent automated attacks and suspicious activity on a website. For regular users, its operation is almost unnoticeable, as its primary goal is to block scripts and bots. In the browser, TSPD’s presence is rare and usually only becomes noticeable during unusual or high-volume requests to the server.
-
Use your own proxies for this task.
-
Proxies used in TSPD tasks must support a static session. The same exit IP must be used throughout all stages (cookies retrieval → task solving → using the received cookies). Changing the session or IP between stages may lead to errors.
-
After solving, you will receive cookies that must be included in subsequent requests to the target website.
Request parameters
IMPORTANT: some parameter values are dynamic — they change with each render of the page using TSPD.
Extract them immediately before creating the task to avoid errors during solving.
See examples of automatic parameter extraction in the sections How to get cookies, How to get the page in Base64, and Examples of automatic TSPD solving.
type<string>requiredCustomTask
class<string>requiredtspd
websiteURL<string>requiredThe URL of the page where the TSPD challenge is located.
tspdCookie (inside metadata)<string>requiredCookies obtained on the TSPD challenge page:
"tspdCookie": "TS386a400d029=082670627aab2800722d179e73a60b575d00c96728a9f8dedd8be27a40f6a1aa5df467cebf7da7315a4e16675f010245; ....; ....;"
htmlPageBase64 (inside metadata)<string>requiredThe entire TSPD page encoded in base64, for example:
"htmlPageBase64": "PCFET0NUWVBFIGh0bWw+DQo8aHRtbD48aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IlByYWdtYSIgY29udGVudD0ibm8tY2FjaGUiLz4NCjxtZXRhIGh0dHAtZXF1aXY9IkV4cGlyZXMiIGNvbnRlbnQ9Ii0xIi8+DQo8bWV0YSBodHRwLWVxdWl2PSJDYWNoZUNvbnRyb2.....L2JvZHk+PC9odG1sPg=="
userAgent<string>optionalBrowser User-Agent.
Provide only the current Windows UA: userAgentPlaceholder
proxyType<string>requiredhttp - regular HTTP/HTTPS proxy;
https - use if http doesn’t work (required for some custom proxies);
socks4 - SOCKS4 proxy;
socks5 - SOCKS5 proxy.
proxyAddress<string>requiredProxy IP address (IPv4/IPv6). Not allowed:
- Transparent proxies
- Local machine proxies
proxyPort<integer>requiredProxy port.
proxyLogin<string>requiredProxy login.
Restrictions: must not contain the characters ;, |, )
proxyPassword<string>requiredProxy password.
Create task method
https://api.capmonster.cloud/createTask
Request
{
"type": "CustomTask",
"class": "tspd",
"websiteUrl": "https://example.com",
"metadata": {
"tspdCookie": "TS386a400d029=08...010245; TS386a400d029=08...01a06e; TS386a400d078=08...dbb3b0c; TSd2153684027=08...1944",
"htmlPageBase64": "PCFET0NU...k+PC9odG1sPg=="
},
"userAgent": "userAgentPlaceholder",
"proxyType": "http",
"proxyAddress": "8.8.8.8",
"proxyPort": 8080,
"proxyLogin": "proxyLoginHere",
"proxyPassword": "proxyPasswordHere"
}
Response
{
"errorId": 0,
"taskId": 407533072
}
Get task result method
Use the getTaskResult method to get the solution for the TSPD.
Important: upon successful completion, newer versions of TSPD may return the
TSPD_101_DIDcookie, whereas older versions may not include it.
https://api.capmonster.cloud/getTaskResult
Request
{
"clientKey":"API_KEY",
"taskId": 407533072
}
Response
{
"errorId":0,
"status":"ready",
"solution": {
"Domains": {
"example.com": {
"Cookies": {
"TS386a400d029": "08267...01a06e",
"TS386a400d078": "08267...bb3b0c",
"TSd2153684027": "08267...11944",
"TS00000000076": "08267...b70fc2",
"TSPD_101_DID": "08267...1d53f",
"TS386a400d075": "0402b...1000"
}
}
}
}
}
Working with a page containing TSPD
Example of an HTML page with a TSPD challenge
When sending a GET request to a page with a TSPD challenge, you will receive HTML that must be encoded in Base64 and passed in the htmlPageBase64 parameter.
During the first request, the server will also issue special cookies (for example, TS386a...400d029). These cookies must be included in the tspdCookie parameter, as they are required for proper TSPD solving and for receiving a valid response from the API.
GET request to a page with TSPD challenge
GET https://example.com/login?client_id=example.client.id&authorization_id=example_auth_id
sec-ch-ua: "Not:A-Brand";v="99", "Google Chrome";v="145", "Chromium";v="145"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
upgrade-insecure-requests: 1
user-agent: userAgentPlaceholder
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
sec-fetch-site: none
sec-fetch-mode: navigate
sec-fetch-user: ?1
sec-fetch-dest: document
accept-encoding: gzip, deflate, br
accept-language: en-US
priority: u=0,i
Upgrade-Insecure-Requests: 1
Page with TSPD-challenge (example)
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Pragma" content="no-cache"/>
<meta http-equiv="Expires" content="-1"/>
<meta http-equiv="CacheControl" content="no-cache"/>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<link rel="icon" href="data:;base64,iVBOw0KGgo="/>
<!-- TSPD Script / ThreatMetrix Fingerprinting -->
<script type="text/javascript">
(function() {
// TSPD indicator: bobcmn variable contains encrypted fingerprinting / session data
window["bobcmn"] = "101110111110102000000042000000052000000062000000012386a400d200000096200000000200000002300000000300000000300000006/TSPD/300000008TSPD_10130000000cTSPD_101_DID300000005https3000000b0082670627aab200065e4de40b46b945b49688c1d7f2da4f189123687c7515bd58ed223d60435730b086f24771b0a2800da70f3141fd5255393c12c8c87a64a83b8168f256387790c28dd3abb533560e42a9a0970df08ebd7300000002TS200000000200000000";
// TSPD indicator: failureConfig for detecting an "incorrect" attempt or error on the page
window["failureConfig"] = "524f6f70732e2e2e2e736f6d657468696e672077656e742077726f6e672e2e2e2e20796f757220737570706f72742069642069733a2025444f534c372e6368616c6c656e67652e737570706f72745f6964252e1338353931363039373735373932323232353437062f545350442f171801";
window.AtT = !!window.AtT;
// Main TSPD JS
try {
(function(){
var z = {
decrypt: function(z){
try {
return JSON.parse(
function(z){
z = z.split("l");
var s = "";
for (var _=0; _<z.length; ++_) s += String.fromCharCode(z[_]);
return s;
}(z)
);
} catch(_) {}
}
};
return z = { configuration: z.decrypt("123l34l97l99l116l105l118l101l34l58l34l110l111l34l44l34l100l101l98l117l103l103l105l110l34l58l34l110l111l34l44l34l109l111l100l117l108l11149l34l58l34l101l110l97l98l108l101l100l34l44l34l109l111l100l117l108l101l50l34l58l34l101l110l97l98l108l101l100l34l44l34l109l111l100l117l108l101l51l34l58l34l101l110l97l98l108l101l100l34l44l34l109l111l100l117l108l101l52l34l58l34l101l110l97l98l108l101l100l34l125")};
})();
var sz = 15;
// Browser environment check, IE9/IE9RGB indicators
zs(window[Zs[J(1086839, sz)]] === Zs);
zs(typeof ie9rgb4 !== J(1242178186184, sz));
})();
})();
</script>
<!-- TSPD / ThreatMetrix fingerprinting script, type 8 -->
<script type="text/javascript" src="/TSPD/082670627aab2000b8cb2cd11a623629ab3f79c29c36f891be5a445796e6258af0d27cef4a5db1d4?type=8"></script>
<script type="text/javascript">
// Another TSPD indicator: blobfp and slobfp - unique device fingerprints
(function(){
window["blobfp"] = "01010101b00400000100e803000000000d4200353665636365626565343132626436353030363938386138663833326530623934356538333435633861363438623036643666386238363263333064396466633465210068747470733a2f2f72652e73656375726974792e66356161732e636f6d2f72652f0700545350445f3734";
window["slobfp"] = "0827420c940b100087cabc9b010dfd3b94ac7988b1dcbba0";
})();
</script>
<!-- TSPD / ThreatMetrix fingerprinting script, type 12 -->
<script type="text/javascript" src="/TSPD/082670627aab2000b8cb2cd11a623629ab3f79c29c36f891be5a445796e6258af0d27cef4a5db1d4?type=12"></script>
<noscript>
Please enable JavaScript to view the page content.<br/>
Your support ID is: 8591609775792222547.
<link rel="stylesheet" href="/TSPD/?type=25" />
</noscript>
</head>
<body>
</body>
</html>
Example of successful cookie acceptance (TSPD does not appear on repeat request)
When making a repeat request to the target page with cookies obtained from CapMonster Cloud, the server will return a regular HTML response with status 200 and without any signs of TSPD. This means the cookies were accepted and you can continue working with your website.
GET request to the page
GET https://example.com/login?client_id=example.client.id&authorization_id=example_auth_id
sec-ch-ua: "Not:A-Brand";v="99", "Google Chrome";v="145", "Chromium";v="145"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
upgrade-insecure-requests: 1
user-agent: userAgentPlaceholder
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
sec-fetch-site: none
sec-fetch-mode: navigate
sec-fetch-user: ?1
sec-fetch-dest: document
accept-encoding: gzip, deflate, br
accept-language: en-US
# Use the cookies obtained as a result of solving the task via CapMonster Cloud
# Replace the example values below with the actual cookies from the API response
# Format: cookie: name1=value1; name2=value2; ...
cookie: TS386a400d029=08267...01a06e; TS386a400d078=0826...dbb3b0c; TSd2153684027=082670...811944; TS00000000076=082670...b70fc2; TSPD_101_DID=08267...1d53f; TS386a400d075=0402b1...701000
priority: u=0,i
Upgrade-Insecure-Requests: 1
HTML page (example)
<!DOCTYPE html>
<html lang="demo">
<head>
<meta charset="utf-8" />
<link rel="icon" type="image/png" href="/sandbox.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Sample Portal</title>
<!-- Demo Font -->
<link rel="stylesheet" href="https://demo-cdn.com/fonts/demo-font.css" />
<!-- Icons -->
<link rel="stylesheet" href="https://assets-sandbox.com/icons/demo-icons.css"/>
<link rel="stylesheet" href="/static/sandbox.css">
<script type="module" crossorigin src="/scripts/demo-app.js"></script>
<link rel="stylesheet" crossorigin href="/styles/sample-theme.css">
</head>
<body>
<noscript>Please enable scripts to continue.</noscript>
<div id="sandbox-root"></div>
</body>
</html>
How to get cookies
The example code below sends a GET request to the authorization page example.com/login with browser headers, prints the response status, the full HTML of the page, and the received cookies (including those that start with TS).
Node.js example
const url =
"https://example.com/login?client_id=example.client&authorization_id=example123";
// === Headers ===
const headers = {
"sec-ch-ua":
'"Not(A:Brand";v="8", "Chromium";v="145", "Google Chrome";v="145"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
"Upgrade-Insecure-Requests": "1",
"user-agent":
"userAgentPlaceholder",
accept:
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"sec-fetch-site": "same-site",
"sec-fetch-mode": "navigate",
"sec-fetch-user": "?1",
"sec-fetch-dest": "document",
"accept-language": "en-US",
priority: "u=0,i",
referer: "https://example.com/"
};
async function main() {
const response = await fetch(url, {
method: "GET",
headers,
redirect: "follow",
});
console.log("Status:", response.status);
console.log("Final URL:", response.url);
// === Get HTML ===
const html = await response.text();
console.log("\n===== FULL HTML RESPONSE =====\n");
console.log(html);
// === Get cookies ===
const rawHeaders = response.headers;
let setCookies = [];
if (typeof rawHeaders.getSetCookie === "function") {
setCookies = rawHeaders.getSetCookie();
}
// fallback (if available)
else if (rawHeaders.raw) {
setCookies = rawHeaders.raw()["set-cookie"] || [];
}
console.log("\n===== COOKIES FROM RESPONSE =====\n");
setCookies.forEach((c) => {
const cookiePair = c.split(";")[0];
console.log(cookiePair);
});
// === Only TS cookies ===
const tspdCookies = setCookies
.map((c) => c.split(";")[0])
.filter((c) => c.startsWith("TS"));
console.log("\n===== TSPD COOKIES =====\n");
console.log(tspdCookies);
}
main().catch(console.error);
Important:
If the server returns cookies in the following format (three or four lines):
TS386a400d029=082670...87599c;
TS386a400d029=082670...40a8ea3;
TS386a400d078=082670...b4cbe2c;
TSd2153684027=082670...4415a6
TS386a400d029=08777...83ff9,
TS386a400d029=08777...fb459e,
TSd2153684027=08777...f0ad368
or:
TS014d0691=01fef...1244b,
TS01fe94e8=01fef...9ed38,
TSafd868f7027=082670...a7ea7c
and if the TSPD_101_DID cookie is absent, this indicates that the request was processed by the protection mechanism and a blocking page (TSPD challenge) was returned. In this case, it is necessary to:
- Pass the obtained set of cookies to CapMonster Cloud.
- Wait for the solution result.
- Use the updated cookies for subsequent requests (see examples of automatic TSPD solving below).
How to get the page in Base64
After you have obtained the HTML of the page with the TSPD challenge, it must be encoded in Base64. This can be done using various tools or programming libraries. Below is a Node.js example that performs this task:
Node.js example
const url =
"https://example.com/login";
// === Set the required headers ===
const headers = {
"sec-ch-ua":
'"Not(A:Brand";v="8", "Chromium";v="145", "Google Chrome";v="145"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
"Upgrade-Insecure-Requests": "1",
"user-agent":
"userAgentPlaceholder",
accept:
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"sec-fetch-site": "same-site",
"sec-fetch-mode": "navigate",
"sec-fetch-user": "?1",
"sec-fetch-dest": "document",
"accept-language": "en-US",
priority: "u=0,i",
referer: "https://example.com/"
};
async function main() {
const response = await fetch(url, {
method: "GET",
headers,
redirect: "follow",
});
if (!response.ok) {
throw new Error(`HTTP error: ${response.status}`);
}
// Get full HTML
const html = await response.text();
// Encode to Base64
const base64 = Buffer.from(html, "utf-8").toString("base64");
console.log("===== BASE64 RESULT =====\n");
console.log(base64);
}
main().catch(console.error);
Examples of automatic TSPD solving
The examples are for demonstration purposes and show the general logic of working with websites that use TSPD protection. In real projects, the code may require adaptation to a specific website, its requests, and headers.
Important data (API keys, proxy settings, etc.) should be stored in .env files or environment variables.
How the algorithm works:
-
First request. The script sends an HTTP request to the page through a proxy with browser headers.
-
Retrieving cookies. The required protection cookies are extracted from the response. If the
TSPD_101_DIDcookie is returned (indicating that the protection has already been passed) or the format is invalid, execution is stopped. -
Creating a task. The script sends a task of type CustomTask (class: tspd) to CapMonster Cloud with the following data:
-
Page URL
-
Protection cookies
-
Page HTML (Base64)
-
User-Agent
-
Proxy
-
-
Getting the solution. The script waits for the result via the API (getTaskResult) and receives valid cookies.
-
Final request. The request is repeated with the obtained cookies, after which a regular HTML page is returned.
- JavaScript
- Python
Show code (Node.js)
import { gotScraping } from "got-scraping"; // For making HTTP requests with a TLS fingerprint
import fs from "fs";
const API_KEY = "YOUR_API_KEY"; // Replace with your CapMonster Cloud API key
const CAPMONSTER_URL = "https://api.capmonster.cloud";
// Replace with the target URL
const url = "https://example.com";
// Replace with your proxy
const proxy = {
host: "proxyAddress",
port: 8080,
login: "proxyLogin",
password: "proxyPassword",
type: "http",
};
const proxyUrl = `${proxy.type}://${proxy.login}:${proxy.password}@${proxy.host}:${proxy.port}`;
// Use a valid User-Agent
const userAgent =
"userAgentPlaceholder";
function delay(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function extractValidTspdCookies(setCookies) {
const cookies = setCookies.map((c) => c.split(";")[0].trim());
console.log("\nRaw cookies:");
console.log(cookies);
// Extract only TS cookies
const tsCookies = cookies.filter((c) => c.startsWith("TS"));
// Check that TSPD_101_DID is absent
const hasDid = cookies.some((c) => c.startsWith("TSPD_101_DID="));
if (hasDid) {
console.log("TSPD_101_DID detected. Request was blocked.");
return null;
}
const validFormat = tsCookies.every((c) => /^[^=]+=[^=]+$/.test(c));
if (!validFormat) {
console.log("Invalid TS cookie format.");
return null;
}
console.log("Valid TSPD cookie set detected.");
return tsCookies.join("; ");
}
function solutionToCookieHeader(solution) {
const domain = Object.keys(solution.domains)[0];
const cookies = solution.domains[domain].cookies || {};
const result = [];
for (const [name, value] of Object.entries(cookies)) {
result.push(`${name}=${value}`);
}
return result.join("; ");
}
async function createTask(task) {
const response = await gotScraping
.post(`${CAPMONSTER_URL}/createTask`, {
json: {
clientKey: API_KEY,
task,
},
})
.json();
if (response.errorId !== 0) {
throw new Error(response.errorDescription);
}
return response.taskId;
}
async function getTaskResult(taskId) {
while (true) {
await delay(3000);
const response = await gotScraping
.post(`${CAPMONSTER_URL}/getTaskResult`, {
json: {
clientKey: API_KEY,
taskId,
},
})
.json();
if (response.errorId !== 0) {
throw new Error(response.errorDescription);
}
if (response.status === "ready") {
return response.solution;
}
console.log("Waiting for solution...");
}
}
async function main() {
try {
console.log("Sending initial request...");
const response = await gotScraping(url, {
proxyUrl,
headers: {
"sec-ch-ua":
'"Not:A-Brand";v="99", "Google Chrome";v="145", "Chromium";v="145"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
"upgrade-insecure-requests": "1",
"user-agent": userAgent,
accept:
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"sec-fetch-site": "same-site",
"sec-fetch-mode": "navigate",
"sec-fetch-user": "?1",
"sec-fetch-dest": "document",
"accept-language": "en-US",
referer: "https://example.com/", // Replace with a valid referer
},
});
let html = response.body;
const htmlBase64 = Buffer.from(html).toString("base64");
const setCookies = response.headers["set-cookie"] || [];
const tspdCookies = extractValidTspdCookies(setCookies);
if (!tspdCookies) {
console.log(
"Stopping script. No valid TSPD cookies detected. Continuing normal site interaction.",
);
return;
}
const task = {
type: "CustomTask",
class: "tspd",
websiteUrl: url,
metadata: {
tspdCookie: tspdCookies,
htmlPageBase64: htmlBase64,
},
userAgent,
proxyType: proxy.type,
proxyAddress: proxy.host,
proxyPort: proxy.port,
proxyLogin: proxy.login,
proxyPassword: proxy.password,
};
console.log("\nCreating CapMonster Cloud task...");
const taskId = await createTask(task);
console.log("Task ID:", taskId);
const solution = await getTaskResult(taskId);
console.log("\nSolution received:");
console.log(JSON.stringify(solution, null, 2));
const cookieHeader = solutionToCookieHeader(solution);
console.log("\nFinal Cookie header:");
console.log(cookieHeader);
const finalHeaders = {
"sec-ch-ua":
'"Not:A-Brand";v="99", "Google Chrome";v="145", "Chromium";v="145"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
"upgrade-insecure-requests": "1",
"user-agent": userAgent,
accept:
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"sec-fetch-site": "same-site",
"sec-fetch-mode": "navigate",
"sec-fetch-user": "?1",
"sec-fetch-dest": "document",
"accept-encoding": "gzip, deflate, br, zstd",
"accept-language": "en-US",
cookie: cookieHeader,
priority: "u=0,i",
"Upgrade-Insecure-Requests": "1",
referer: "https://example.com/", // Replace with a valid referer
};
console.log("\nFinal request headers:");
console.log(JSON.stringify(finalHeaders, null, 2));
const finalResponse = await gotScraping(url, {
proxyUrl,
headers: finalHeaders,
followRedirect: true,
});
console.log("\nFinal page length:", finalResponse.body.length);
fs.writeFileSync("final_page.html", finalResponse.body);
console.log("Page saved to final_page.html");
} catch (err) {
console.error("Error:", err.message);
}
}
main();
Show code
import asyncio
import base64
import re
import json
from curl_cffi.requests import AsyncSession # For making HTTP requests with a TLS fingerprint
API_KEY = "YOUR_API_KEY" # Replace with your CapMonster Cloud API key
CAPMONSTER_URL = "https://api.capmonster.cloud"
url = "https://example.com" # Replace with the target URL
# Replace with your proxy
proxy = {
"host": "proxyAddress",
"port": 8080,
"login": "proxyLogin",
"password": "proxyPassword",
"type": "http"
}
proxy_url = f'{proxy["type"]}://{proxy["login"]}:{proxy["password"]}@{proxy["host"]}:{proxy["port"]}'
# Use a valid User-Agent
user_agent = "userAgentPlaceholder"
def extract_ts_cookies(set_cookies):
cookies = [c.split(";")[0].strip() for c in set_cookies]
print("\nRaw cookies:")
print(cookies)
# Check that TSPD_101_DID is absent
has_did = any(c.startswith("TSPD_101_DID=") for c in cookies)
if has_did:
print("TSPD_101_DID detected. Request was blocked.")
return None
# Extract only TS cookies
ts = [c for c in cookies if c.startswith("TS")]
if not ts:
return None
if not all(re.match(r"^[^=]+=[^=]+$", c) for c in ts):
print("Invalid TS cookie format.")
return None
print("Valid TSPD cookie set detected.")
return "; ".join(ts)
def solution_to_cookie_dict(solution):
domain = list(solution["domains"].keys())[0]
return solution["domains"][domain].get("cookies", {})
def solution_to_cookie_header(solution):
domain = list(solution["domains"].keys())[0]
cookies = solution["domains"][domain].get("cookies", {})
return "; ".join([f"{k}={v}" for k, v in cookies.items()])
async def create_task(session, task):
r = await session.post(
f"{CAPMONSTER_URL}/createTask",
json={"clientKey": API_KEY, "task": task},
)
data = r.json()
if data["errorId"] != 0:
raise Exception(data["errorDescription"])
return data["taskId"]
async def get_result(session, task_id):
while True:
await asyncio.sleep(3)
r = await session.post(
f"{CAPMONSTER_URL}/getTaskResult",
json={"clientKey": API_KEY, "taskId": task_id},
)
data = r.json()
if data["errorId"] != 0:
raise Exception(data["errorDescription"])
if data["status"] == "ready":
return data["solution"]
print("Waiting...")
async def main():
async with AsyncSession(
impersonate="chrome124",
proxies={
"http": proxy_url,
"https": proxy_url,
},
headers={
"user-agent": user_agent,
"accept-language": "en-US,en;q=0.9",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"upgrade-insecure-requests": "1",
},
) as session:
print("Step 1: Initial request")
r = await session.get(url)
html = r.text
html_b64 = base64.b64encode(html.encode()).decode()
set_cookies = r.headers.get_list("set-cookie")
tspd = extract_ts_cookies(set_cookies)
if not tspd:
print("No TS cookies")
return
print("\nTS cookies:")
print(tspd)
task = {
"type": "CustomTask",
"class": "tspd",
"websiteUrl": url,
"metadata": {
"tspdCookie": tspd,
"htmlPageBase64": html_b64,
},
"userAgent": user_agent,
"proxyType": proxy["type"],
"proxyAddress": proxy["host"],
"proxyPort": proxy["port"],
"proxyLogin": proxy["login"],
"proxyPassword": proxy["password"],
}
print("\nStep 2: Create task")
task_id = await create_task(session, task)
print("Task ID:", task_id)
solution = await get_result(session, task_id)
print("\nSolution received:")
print(json.dumps(solution, indent=2))
cookies = solution_to_cookie_dict(solution)
print("\nApplying cookies to session:")
for k, v in cookies.items():
print(f"{k}={v}")
session.cookies.set(k, v)
cookie_header = solution_to_cookie_header(solution)
print("\nFinal Cookie header:")
print(cookie_header)
print("\nStep 3: Final request")
final = await session.get(url)
print("\nFinal status:", final.status_code)
print("Final page length:", len(final.text))
with open("final.html", "w", encoding="utf-8") as f:
f.write(final.text)
print("Page saved to final.html")
if __name__ == "__main__":
asyncio.run(main())
