# API Documentation # YetAnotherAPI Overview ### Introduction Welcome to the **YetAnotherAPI** documentation! Our platform offers a robust set of APIs designed to empower developers with a wide range of functionalities, from data extraction to workflow automation. Whether you're looking to enhance your app's capabilities, streamline business processes, or build complex integrations, YetAnotherAPI provides the tools to help you succeed. ### API Endpoint The base URL for all API requests is: ``` https://api.yetanotherapi.com/ ``` This is the primary endpoint for accessing the various services provided by YetAnotherAPI. Each API has its own endpoint path that you can utilize based on the functionality you're looking to access. ### Limitations To ensure stable and consistent performance for all users, the API has certain limitations: | Limitation Type | Details | | --------------- | --------------------------------------- | | Request Payload | Maximum size of 100KB per request | | Data Requests | Specific limits vary by API endpoint | | Execution Time | Time limits apply to complex operations | These limitations are subject to change based on updates or upgrades to the API infrastructure. ### Rate Limits We enforce rate limits to maintain fair usage for all users: * 60 requests per minute * Exceeding limits results in `429 Too Many Requests` response * Higher limits available with subscription upgrades For questions or support, please contact our team through the support portal. # Authentication ### Overview All requests to YetAnotherAPI services require authentication. This document outlines the authentication process and best practices for securing your API access. ### API Key Authentication #### Obtaining an API Key 1. Sign up for an account at YetAnotherAPI platform 2. Navigate to the API Keys section in your dashboard 3. Generate a new API key for your application #### Using Your API Key Include your API key in the request header for all API calls: ``` x-api-key: YOUR_API_KEY ``` #### Example Request ```bash curl --location 'https://api.yetanotherapi.com/v1/endpoint' \ --header 'x-api-key: YOUR_API_KEY_HERE' \ --header 'Content-Type: application/json' ``` ### Security Best Practices 1. **Key Protection** * Never expose your API key in client-side code * Don't commit API keys to version control * Rotate keys periodically for enhanced security 2. **Environment Management** * Use different API keys for development and production * Store keys in secure environment variables * Implement key rotation procedures 3. **Access Control** * Monitor API key usage regularly * Revoke compromised keys immediately * Use the minimum required permissions for each key ### Error Responses | Status Code | Description | Solution | | ----------- | ------------------------ | ---------------------------------------- | | 401 | Invalid API key | Check if key is correct and active | | 403 | Insufficient permissions | Verify key has required access levels | | 429 | Rate limit exceeded | Reduce request frequency or upgrade plan | ### Support If you encounter authentication issues or need assistance, contact our support team through the support portal. # Integrations # Pabbly-Connect ## Using YetAnotherAPI with Pabbly Connect For seamless integration with **YetAnotherAPI** through **Pabbly Connect**, there’s no need to configure APIs manually. You can quickly set up automation by adding the **YetAnotherAPI** app in Pabbly Connect using this link: [YetAnotherAPI Pabbly App](https://connect.pabbly.com/share-app/WhIAMlAHAmEHTQRuDklTdAwYAQUFXFY1U0oEFlBfUysHSlMoUBEIYlsXACJfEAdmB04IYlQIBD1YS1BnVwtVbwERCAFUMVwaVAsFdAJhCn9aEgAyUAcCYQdNBG4OTVN0DBgBBQVcVjZTSgQWUF9TKwdJUzhQEQhiWxcAIl8-Bz4HXAh4VDE#). #### How to Use YetAnotherAPI with Pabbly 1. **Add the App**: Use the link above to add **YetAnotherAPI** to your Pabbly Connect account. 2. **Available Triggers and Actions**: * **Trigger: Webhook** * To set up a webhook trigger, search for **YetAnotherAPI** in the triggers list and select the **Webhook** option. This feature is helpful if you want to send data from the **LLM Web Scrapper** to a specific webhook URL. * **Action: LLM Web Scrapper** * Choose the **LLM Web Scrapper** action to make an API request directly through Pabbly. This allows for real-time data extraction and response from the **LLM Web Scrapper** API without requiring any manual API setup. With Pabbly’s integration, automation with **YetAnotherAPI** becomes straightforward, enabling you to connect, trigger, and automate actions effectively. # Document parser ## Document Parser API Documentation Base URL: `https://api.yetanotherapi.com` ### Overview The Document Parser API allows you to extract text content from various document formats including PDF, Word documents, and images. The service provides both synchronous and asynchronous processing with optional webhook notifications for completion. ### Authentication All API requests require an API key sent in the header: ``` x-api-key: YOUR_API_KEY ``` ### API Endpoints ## Submit Document for Processing Submit a document for text extraction. **Endpoint:** `POST /documents/parse` **Headers:** ``` Content-Type: application/json x-api-key: YOUR_API_KEY ``` **Request Body:** ```json { "url": "string", // Required: URL of the document "type": "string", // Optional: Document type (default: "pdf") "output": "string", // Optional: Output format (default: "plain") "webhook": "string" // Optional: Webhook URL for completion notification } ``` **Supported File Types:** * `pdf`: PDF documents * `doc`: Microsoft Word documents (.doc) * `docx`: Microsoft Word documents (.docx) * `jpg`/`jpeg`: JPEG images * `png`: PNG images * `txt`: Plain text files **Output Formats:** * `plain`: Plain text (default) * `markdown`: Formatted markdown text **Response:** * Quick Processing (< 20 seconds): ```json { "requestId": "string", "status": "COMPLETED", "data": "string" // Extracted text content } ``` * Async Processing (> 20 seconds): ```json { "requestId": "string", "status": "PROCESSING", "message": "Processing in progress. Please check status endpoint." } ``` * Error Response: ```json { "error": "string", "details": ["string"] // Array of error details } ``` **Status Codes:** * 200: Success (processing completed) * 202: Accepted (processing continues asynchronously) * 400: Bad Request (invalid input) * 401: Unauthorized (invalid API key) * 500: Internal Server Error ### Example Usage #### cURL Example ```bash curl --location 'https://api.yetanotherapi.com/documents/parse' \ --header 'x-api-key: YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://example.com/document.pdf", "type": "pdf", "output": "markdown", "webhook": "https://your-webhook-url.com/callback" }' ``` #### ### Error Codes and Descriptions | Error Code | Description | | ---------- | ----------------------- | | 400-001 | Invalid file type | | 400-002 | Invalid URL format | | 400-003 | Invalid webhook URL | | 400-004 | Missing required field | | 400-005 | Invalid output format | | 401-001 | Invalid API key | | 429-001 | Rate limit exceeded | | 500-001 | Processing error | | 500-002 | Storage error | | 500-003 | Webhook delivery failed | ### Notes 1. Processing time varies based on document size and complexity 2. Files are stored temporarily and deleted after 7 days 3. Webhook endpoints should respond within 30 seconds 4. All timestamps are in Unix epoch format # PDF Parser ## PDF Document Processing Guide Document Parser API supports extraction of text content from PDF files. ### Endpoint `POST /documents/parse` ### PDF-Specific Configuration ```json { "url": "string", // URL of the PDF document "type": "pdf", // Specify "pdf" for PDF processing "output": "plain|markdown", "webhook": "string" // Optional webhook URL } ``` ### Supported PDF Features * Single and multi-page PDFs * Text-based PDFs * Scanned PDFs (using OCR) * Password-protected PDFs (not supported) * Maximum file size: 50MB ### Example Request ```bash curl --location 'https://api.yetanotherapi.com/documents/parse' \ --header 'x-api-key: YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://example.com/document.pdf", "type": "pdf", "output": "markdown" }' ``` ### Response Format Each page's content is separated by a page break marker: ```json { "requestId": "string", "status": "COMPLETED", "data": "Page 1 content\n\n=== Page Break ===\n\nPage 2 content" } ``` ### PDF-Specific Limitations 1. Forms and fillable fields are processed as static text 2. Complex layouts may affect text ordering 3. Headers and footers are included in the extracted text 4. Images within PDFs are not processed 5. PDF versions supported: 1.0 to 2.0 # Doc Parser ## Word Document Processing Guide Document Parser API supports extraction of text from Microsoft Word documents (.doc and .docx). ### Endpoint `POST /documents/parse` ### Word-Specific Configuration ```json { "url": "string", // URL of the Word document "type": "doc", // Use "doc" for .doc or .docx files "output": "plain|markdown", "webhook": "string" // Optional webhook URL } ``` ### Supported Word Features * DOC format (.doc) * DOCX format (.docx) * Text content * Tables (converted to text) * Headers and footers * Maximum file size: 50MB ### Example Request ```bash curl --location 'https://api.yetanotherapi.com/documents/parse' \ --header 'x-api-key: YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://example.com/document.doc", "type": "doc", "output": "markdown" }' ``` ### Response Format Each page's content is separated by a page break marker: ```json { "requestId": "string", "status": "COMPLETED", "data": "Page 1 content\n\n=== Page Break ===\n\nPage 2 content" } ``` ### Word-Specific Limitations 1. Macros are ignored 2. Comments are not included 3. Track changes are processed in their current state 4. Complex formatting may be simplified 5. Images are not processed # PNG & JPG Parser ## Image Processing Guide Document Parser API supports text extraction from various image formats using OCR technology. ### Endpoint `POST /documents/parse` ### Supported Image Types 1. JPEG/JPG ```json { "url": "string", "type": "jpg", // or "jpeg" "output": "plain|markdown" } ``` 2. PNG ```json { "url": "string", "type": "png", "output": "plain|markdown" } ``` 3. TIFF/TIF ```json { "url": "string", "type": "tiff", // or "tif" "output": "plain|markdown" } ``` 4. HEIC ```json { "url": "string", "type": "heic", "output": "plain|markdown" } ``` ### Image Processing Features * Text extraction using OCR * Multi-page support for TIFF * Various image resolutions supported * Color, grayscale, and black/white images * Maximum file size: 50MB ### Example Request ```bash curl --location 'https://api.yetanotherapi.com/documents/parse' \ --header 'x-api-key: YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://example.com/scan.jpg", "type": "jpg", "output": "markdown" }' ``` ### Response Format For single-page images: ```json { "requestId": "string", "status": "COMPLETED", "data": "Extracted text content" } ``` For multi-page TIFF: ```json { "requestId": "string", "status": "COMPLETED", "data": "Page 1 content\n\n=== Page Break ===\n\nPage 2 content" } ``` ### Image-Specific Limitations 1. Image quality affects OCR accuracy 2. Minimum resolution required: 300 DPI 3. Handwritten text may not be accurately recognized 4. Complex backgrounds can affect accuracy 5. Text must be properly oriented # TXT Parser ## Text File Processing Guide Document Parser API supports processing of plain text files. ### Endpoint `POST /documents/parse` ### Text File Configuration ```json { "url": "string", // URL of the text file "type": "txt", // Specify "txt" for text files "output": "plain|markdown", "webhook": "string" // Optional webhook URL } ``` ### Text Processing Features * UTF-8 encoding (default) * Fallback to Latin-1 encoding * Form feed character (\f) recognition for page breaks * Maximum file size: 50MB ### Example Request ```bash curl --location 'https://api.yetanotherapi.com/documents/parse' \ --header 'x-api-key: YOUR_API_KEY' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://example.com/document.txt", "type": "txt", "output": "markdown" }' ``` ### Markdown Output Features When output is set to "markdown": 1. Lines ending with ':' are converted to H3 headers 2. Short lines (<50 chars) at paragraph starts become H2 headers 3. Empty lines create paragraph breaks 4. Form feeds create page breaks ### Response Format ```json { "requestId": "string", "status": "COMPLETED", "data": "Processed text content with optional markdown formatting" } ``` ### Text-Specific Limitations 1. Binary text files not supported 2. Maximum line length: 1MB 3. Maximum number of lines: 1,000,000 4. Non-standard encodings may cause issues 5. Control characters (except \f) are stripped # Parser Processing Status ## Document Processing Status Checking Guide ### Overview The status checking endpoint allows you to monitor the progress of your document processing requests and retrieve results. ### Endpoint `GET /documents/status/{requestId}` Base URL: `https://api.yetanotherapi.com` ### Authentication ```http x-api-key: YOUR_API_KEY ``` ### Response States #### 1. Processing State ```json { "requestId": "string", "status": "PROCESSING", "type": "string", // Document type (pdf, doc, etc.) "createdAt": "number" // Unix timestamp } ``` #### 2. Completed State ```json { "requestId": "string", "status": "COMPLETED", "type": "string", "createdAt": "number", "data": "string" // Extracted text content with page separators } ``` #### 3. Failed State ```json { "requestId": "string", "status": "FAILED", "type": "string", "createdAt": "number", "error": "string" // Error description } ``` ### Status Codes * 200: Status retrieved successfully * 404: Request ID not found * 401: Invalid API key * 500: Internal server error ### Example Usage #### cURL Example ```bash curl --location 'https://api.yetanotherapi.com/documents/status/7a509d7c-f61b-4755-a09a-d5782ca27489' \ --header 'x-api-key: YOUR_API_KEY' ``` #### Python Example ```python import requests api_key = "YOUR_API_KEY" request_id = "7a509d7c-f61b-4755-a09a-d5782ca27489" response = requests.get( f"https://api.yetanotherapi.com/documents/status/{request_id}", headers={"x-api-key": api_key} ) print(response.json()) ``` ### Polling Recommendations * Initial check: Wait 5 seconds after submission * Subsequent checks: Every 10 seconds * Maximum polling duration: 10 minutes * Implement exponential backoff for long-running processes ### Error Handling | Error Code | Description | | ---------- | ----------------------------------- | | 404-001 | Request ID not found | | 404-002 | Request expired (older than 7 days) | | 401-001 | Invalid API key | | 500-001 | Internal server error | ### Best Practices 1. Implement exponential backoff in polling 2. Handle all possible status codes 3. Use webhook notifications for long-running processes 4. Store request IDs for future reference 5. Check expiration (7 days) for old requests # Web Scrapper \[deprecated] ## **Web Scraper API Documentation** **Overview**\ The Web Scraper API allows you to extract specific information from web pages using natural language prompts. It combines web scraping with optional Large Language Model (LLM) processing to provide structured data based on your requirements. **Cost** * Cost per API call: 1 credit **Base URL**\ `https://api.yetanotherapi.com/v1/llm-web-scrapper` **Authentication**\ Authentication is required for all API requests. Use your API key in the `x-api-key` header.\ `x-api-key: YOUR_API_KEY_HERE` **Endpoints**\ **Scrape Web Page**\ Scrapes a specified URL and extracts information based on a given prompt. * **HTTP Method**: POST * **Endpoint**: / **Request Headers** * `x-api-key`: YOUR\_API\_KEY\_HERE * `Content-Type`: application/json **Request Body** * `url`: The URL of the web page to scrape. * `prompt`: The natural language prompt specifying the information to extract. * `use_llm`: A boolean value indicating whether to use LLM for processing. * `webhook`: (Optional) A webhook URL to send the response to. **Example Request** ```bash curl --location 'https://api.yetanotherapi.com/v1/llm-web-scrapper' \ --header 'x-api-key: $API_KEY_HERE' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://www.amazon.in/AMVR-Controller-Compatible-Accessories-Adjustable/dp/B0CJRK7B8J/ref=pd_rhf_gw_s_pd_crcd_d_sccl_1_3/261-2157292-0625645", "prompt": "product name and 5 of its features", "use_llm": true, "webhook": "https://connect.pabbly.com/workflow/sendwebhookdata/IjU3NjYwNTZkMDYzNTA0M2M1MjZiNTUzNjUxMzYi_pc" }' ``` **Response**\ The API will attempt to process the scraped content and provide JSON before 20 seconds. If it is not processed within that time, a request ID will be returned, which can be used to call the following endpoint to get the output. The API also supports sending the response to a webhook URL if provided. **Example Response** ```bash curl --location 'https://api.yetanotherapi.com/v1/llm-web-scrapper/c07ab203-1b4a-42f2-99dd-6628268668d2' ``` **Response Structure**\ The API returns a JSON object with the following structure: **Example Response** ```json { "request_id": "c03995eb-e117-4eca-85c8-e6d398a968d9", "llm_json_structure": { "dowJonesIndexValue": 42313.0 } } ``` **Error Handling**\ In case of errors, the API will return a JSON object with an error message and HTTP status code. **Example Error Response**: ```json { "error": "Invalid URL format", "status_code": 400 } ``` **Rate Limiting**\ Currently, there are no rate limits enforced during the beta phase. However, users should design their applications to handle potential rate limiting in the future. **Beta Version Notice**\ This is a beta version of the API. Users may encounter occasional issues or bugs. * The beta release has limited scope and features. * Subscription plans will be introduced once the API is out of beta. * Not all websites are supported for scraping. **Support**\ For additional questions or to report issues, please contact support at . *** # Basic Web Scraper ## Basic Web Scraper API Documentation ### Introduction The Basic Web Scraper API enables extraction of raw HTML content and specified elements from web pages. This version focuses on direct web scraping without LLM processing. ### Authentication All API requests require authentication using an API key. Include your key in the `x-api-key` header: ``` x-api-key: YOUR_API_KEY_HERE ``` ### Base URL ``` https://api.yetanotherapi.com/v1/llm-web-scrapper ``` ### Pricing Each API call costs 1 credit. ### Endpoint Details #### Scrape Web Page Extract raw content from a specified URL using CSS selectors or XPath. **HTTP Method**: POST\ **Endpoint**: `/` **Request Headers** | Header | Value | Description | | ------------ | -------------------- | ---------------------------------- | | x-api-key | YOUR\_API\_KEY\_HERE | Your unique API authentication key | | Content-Type | application/json | Specify JSON request body | **Request Body Parameters** | Parameter | Type | Required | Description | | --------- | ------- | -------- | ------------------------------------------------- | | url | string | Yes | URL of the web page to scrape | | selector | string | No | CSS selector or XPath to target specific elements | | use\_llm | boolean | Yes | Must be set to `false` for basic scraping | | webhook | string | No | Optional webhook URL for receiving response | **Example Request** ```bash curl --location 'https://api.yetanotherapi.com/v1/llm-web-scrapper' \ --header 'x-api-key: $API_KEY_HERE' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://www.amazon.in/AMVR-Controller-Compatible-Accessories-Adjustable/dp/B0CJRK7B8J", "selector": "#productTitle", "use_llm": false, "webhook": "https://your-webhook.com/endpoint" }' ``` **Response Structure** ```json { "request_id": "c03995eb-e117-4eca-85c8-e6d398a968d9", "scraped_content": { "html": "
Product Name Here
", "text": "Product Name Here" } } ``` ### Error Handling ```json { "error": "Invalid URL format", "status_code": 400 } ``` Common Error Codes: * 400: Bad Request (invalid parameters) * 401: Unauthorized (invalid API key) * 403: Forbidden (blocked by target website) * 404: Page Not Found * 429: Too Many Requests ### Limitations * JavaScript rendering is not supported * Some websites may block automated access * Maximum page size: 5MB * Timeout: 20 seconds ### Support For technical support or to report issues, contact: # LLM Web Scraper ## LLM Web Scraper API Documentation ### Introduction The LLM Web Scraper API combines web scraping with Large Language Model processing to extract specific information from web pages using natural language prompts. This version provides structured, intelligent data extraction. ### Authentication All API requests require authentication using an API key. Include your key in the `x-api-key` header: ``` x-api-key: YOUR_API_KEY_HERE ``` ### Base URL ``` https://api.yetanotherapi.com/v1/llm-web-scrapper ``` ### Pricing Each API call costs 1 credit. ### Endpoint Details #### LLM-Enhanced Web Scraping Extract structured information from a webpage using natural language prompts. **HTTP Method**: POST\ **Endpoint**: `/` **Request Headers** | Header | Value | Description | | ------------ | -------------------- | ---------------------------------- | | x-api-key | YOUR\_API\_KEY\_HERE | Your unique API authentication key | | Content-Type | application/json | Specify JSON request body | **Request Body Parameters** | Parameter | Type | Required | Description | | --------- | ------- | -------- | -------------------------------------------------- | | url | string | Yes | URL of the web page to scrape | | prompt | string | Yes | Natural language prompt describing what to extract | | use\_llm | boolean | Yes | Must be set to `true` for LLM processing | | webhook | string | No | Optional webhook URL for receiving response | **Example Request** ```bash curl --location 'https://api.yetanotherapi.com/v1/llm-web-scrapper' \ --header 'x-api-key: $API_KEY_HERE' \ --header 'Content-Type: application/json' \ --data '{ "url": "https://www.amazon.in/AMVR-Controller-Compatible-Accessories-Adjustable/dp/B0CJRK7B8J", "prompt": "Extract the product name, price, rating, and top 3 features", "use_llm": true, "webhook": "https://your-webhook.com/endpoint" }' ``` **Response Structure** ```json { "request_id": "c03995eb-e117-4eca-85c8-e6d398a968d9", "llm_json_structure": { "product_name": "AMVR VR Controller Grip Cover", "price": 1499.00, "rating": 4.5, "top_features": [ "Anti-slip surface", "Adjustable fit", "Compatible with Meta Quest 2" ] } } ``` ### Prompt Guidelines * Be specific about what information you want to extract * Specify the desired format for numerical data * Mention if you want specific sorting or filtering * Include any required unit conversions Example prompts: * "Extract product specifications in a structured format" * "Find all prices and convert them to USD" * "List main article headings and their first paragraphs" ### Error Handling ```json { "error": "Invalid prompt format", "status_code": 400 } ``` Common Error Codes: * 400: Bad Request (invalid parameters) * 401: Unauthorized (invalid API key) * 422: Unprocessable Content (LLM processing failed) * 429: Too Many Requests ### Limitations * Maximum processing time: 20 seconds * Complex prompts may require status checking * Some websites may block automated access * LLM processing may not be 100% accurate ### Support For technical support or to report issues, contact: # Scrapper Processing Status ## Status Check API Documentation ### Introduction The Status Check API endpoint allows you to retrieve results for long-running scraping operations, whether they use basic scraping or LLM processing. ### Authentication All status checks require authentication using an API key. Include your key in the `x-api-key` header: ``` x-api-key: YOUR_API_KEY_HERE ``` ### Base URL ``` https://api.yetanotherapi.com/v1/llm-web-scrapper ``` ### Endpoint Details #### Check Request Status Retrieve the status and results of a previous scraping request. **HTTP Method**: GET\ **Endpoint**: `/{request_id}` **Request Headers** | Header | Value | Description | | --------- | -------------------- | ---------------------------------- | | x-api-key | YOUR\_API\_KEY\_HERE | Your unique API authentication key | **URL Parameters** | Parameter | Type | Description | | ----------- | ------ | -------------------------------------------- | | request\_id | string | The request ID received from initial request | **Example Request** ```bash curl --location 'https://api.yetanotherapi.com/v1/llm-web-scrapper/c07ab203-1b4a-42f2-99dd-6628268668d2' \ --header 'x-api-key: YOUR_API_KEY_HERE' ``` **Response Structure** For Basic Web Scraping: ```json { "request_id": "c07ab203-1b4a-42f2-99dd-6628268668d2", "status": "completed", "scraped_content": { "html": "
Content here
", "text": "Content here" } } ``` For LLM Web Scraping: ```json { "request_id": "c07ab203-1b4a-42f2-99dd-6628268668d2", "status": "completed", "llm_json_structure": { // Structured data based on original prompt } } ``` **Status Values** | Status | Description | | ---------- | ------------------------------------ | | pending | Request is queued for processing | | processing | Content is being scraped or analyzed | | completed | Results are ready | | failed | Processing encountered an error | **Error Responses** ```json { "error": "Request ID not found", "status_code": 404 } ``` ```json { "error": "Processing failed", "status_code": 500, "details": "Target website blocked access" } ``` ### Important Notes * Status checks are free and don't consume API credits * Request IDs expire after 24 hours * Maximum retry attempts: 3 per request ID * Recommended polling interval: 5 seconds * Results are typically available within: * Basic scraping: 5-10 seconds * LLM processing: 15-20 seconds ### Support For technical support or to report issues, contact: # Web Scraper Welcome to the Web Scraper API documentation. Our API provides powerful web scraping capabilities with optional LLM (Language Model) processing, making it easy to extract and structure web content for your applications. ### Features * **Basic Web Scraping**: Extract text content, links, images, and metadata from any webpage * **Markdown Support**: Get content in either plaintext or markdown format * **LLM Processing**: Use AI to structure and analyze scraped content * **Caching**: Improve performance with optional result caching * **Webhook Notifications**: Receive results asynchronously via webhooks * **Rich Metadata**: Extract meta tags, schema.org data, and structured content ### Authentication All API requests require an API key, which should be included in the `x-api-key` header: ```bash --header 'x-api-key: your-api-key' ``` ### Base URL ``` https://api.yetanotherapi.com/web-scrapper/ ``` ### Available Endpoints | Method | Endpoint | Description | Documentation | | ------ | --------------- | ------------------------------------ | -------------- | | POST | `/` | Submit scraping request | Basic Scraping | | POST | `/` | Submit LLM scraping request | LLM Scraping | | GET | `/{request_id}` | Check request status and get results | Status Check | ### Quick Start #### Basic Scraping Request ```bash curl --location 'https://api.yetanotherapi.com/web-scrapper/' \ --header 'Content-Type: application/json' \ --header 'x-api-key: your-api-key' \ --data '{ "url": "https://example.com", "output_type": "plaintext" }' ``` #### LLM Processing Request ```bash curl --location 'https://api.yetanotherapi.com/web-scrapper/' \ --header 'Content-Type: application/json' \ --header 'x-api-key: your-api-key' \ --data '{ "url": "https://example.com", "use_llm": true, "prompt": "Extract main topics and summarize key points", "openai_key_id": "your-key-id" }' ``` ### Processing Modes #### Synchronous Processing * Results returned immediately if processing completes within 20 seconds * Best for simple pages and quick scraping tasks #### Asynchronous Processing * For longer running requests * Status check endpoint for polling results * Webhook notifications available ### Common Use Cases 1. **Content Aggregation** * Extract articles and blog posts * Monitor news and updates * Collect product information 2. **Data Analysis** * Extract structured data * Analyze web content * Generate insights using LLM 3. **Content Transformation** * Convert HTML to markdown * Extract clean text content * Generate structured JSON ### Best Practices 1. **Use Caching** * Enable `use_cache` for frequently accessed pages. * Cache results available for 15 days. * Reduces processing time and AI cost. 2. **Handle Asynchronous Processing** * Implement webhook endpoint for notifications * Use status check endpoint with reasonable polling intervals (5-15 minutes) * Handle timeout scenarios gracefully 3. **LLM Processing** * Write clear, specific prompts * Consider content length and complexity * Test with sample content first ### Error Handling All API endpoints use standard HTTP response codes: * 200: Success * 202: Accepted (processing) * 400: Bad request * 401: Unauthorized * 429: Rate limit exceeded * 500: Server error Error responses include detailed messages and codes: ```json { "error": "E001: Invalid request format" } ``` ### Support For support requests or questions: 1. Email: 2. API Support Portal or send an email: ### Changelog See our changelog for API updates and changes. # Basic Web Scraper This API endpoint allows you to extract content from websites in either plaintext or markdown format. ### Endpoint ``` POST https://api.yetanotherapi.com/web-scrapper/ ``` ### Headers | Header | Required | Description | | ------------ | -------- | --------------------------- | | Content-Type | Yes | Must be `application/json` | | x-api-key | Yes | Your API authentication key | ### Request Body ```json { "url": "https://example.com", "output_type": "plaintext", "use_cache": false, "webhook": "https://your-webhook-url.com" (optional) } ``` #### Parameters | Parameter | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------- | | url | string | Yes | The URL of the website to scrape | | output\_type | string | No | Either "plaintext" (default) or "markdown" | | use\_cache | boolean | No | If true, returns cached result if available. Default: false | | webhook | string | No | URL to receive webhook notification when processing is complete | ### Response #### Immediate Response (HTTP 200) If processing completes within 20 seconds, you'll receive the full result: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "completed", "timestamp": 1635545600, "text_content": "Extracted text content...", "meta": { "title": "Page Title", "description": "Meta description..." }, "links": [ { "text": "Link text", "url": "https://example.com/link", "type": "internal" } ], "images": [ { "url": "https://example.com/image.jpg", "alt": "Image description" } ] } ``` #### Processing Response (HTTP 202) If processing takes longer than 20 seconds: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "processing", "message": "Processing your request. Please check status later." } ``` #### Error Response (HTTP 4XX/5XX) ```json { "error": "ERROR_CODE: Error message" } ``` Common error codes: * E001: Invalid request format * E003: Invalid URL format * E006: Storage service error * E008: Content processing failed ### Response Fields | Field | Type | Description | | ------------- | ------ | ------------------------------------------------------ | | request\_id | string | Unique identifier for the request | | url | string | The URL that was scraped | | status | string | Status of the request ("completed" or "processing") | | timestamp | number | Unix timestamp of when the request was processed | | text\_content | string | The extracted text content (if output\_type=plaintext) | | meta | object | Metadata from the page | | links | array | Array of links found on the page | | images | array | Array of images found on the page | ### Example Curl Request ```bash curl --location 'https://api.yetanotherapi.com/web-scrapper/' \ --header 'Content-Type: application/json' \ --header 'x-api-key: your-api-key' \ --data '{ "url": "https://example.com", "output_type": "plaintext", "use_cache": false }' ``` ### Notes * The API supports both synchronous and asynchronous processing * For pages requiring longer processing time, use the status check endpoint to poll for results * Use webhooks for automatic notification when processing completes * Cache results are available for 15 days # Webpage links Scrapper This API endpoint allows you to extract content from websites in either plaintext or markdown format. ### Endpoint ``` POST https://api.yetanotherapi.com/web-scrapper/ ``` ### Headers | Header | Required | Description | | ------------ | -------- | --------------------------- | | Content-Type | Yes | Must be `application/json` | | x-api-key | Yes | Your API authentication key | ### Request Body ***Though you don't have to explicitly mention about links. we will scrape it by default and just make API call using below payload.*** ```json { "url": "https://example.com", "output_type": "markdown", "use_cache": false, "webhook": "https://your-webhook-url.com" (optional) } ``` #### Parameters | Parameter | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------- | | url | string | Yes | The URL of the website to scrape | | output\_type | string | No | Either "plaintext" (default) or "markdown" | | use\_cache | boolean | No | If true, returns cached result if available. Default: false | | webhook | string | No | URL to receive webhook notification when processing is complete | ### Response #### Immediate Response (HTTP 200) If processing completes within 20 seconds, you'll receive the full result: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "completed", "timestamp": 1635545600, "text_content": "Extracted text content...", "meta": { "title": "Page Title", "description": "Meta description..." }, "links": [ { "text": "Link text", "url": "https://example.com/link", "type": "internal" } ], "images": [ { "url": "https://example.com/image.jpg", "alt": "Image description" } ] } ``` #### Processing Response (HTTP 202) If processing takes longer than 20 seconds: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "processing", "message": "Processing your request. Please check status later." } ``` #### Error Response (HTTP 4XX/5XX) ```json { "error": "ERROR_CODE: Error message" } ``` Common error codes: * E001: Invalid request format * E003: Invalid URL format * E006: Storage service error * E008: Content processing failed ### Response Fields | Field | Type | Description | | ------------- | ------ | ------------------------------------------------------ | | request\_id | string | Unique identifier for the request | | url | string | The URL that was scraped | | status | string | Status of the request ("completed" or "processing") | | timestamp | number | Unix timestamp of when the request was processed | | text\_content | string | The extracted text content (if output\_type=plaintext) | | meta | object | Metadata from the page | | links | array | Array of links found on the page | | images | array | Array of images found on the page | ### Example Curl Request ```bash curl --location 'https://api.yetanotherapi.com/web-scrapper/' \ --header 'Content-Type: application/json' \ --header 'x-api-key: your-api-key' \ --data '{ "url": "https://example.com", "output_type": "plaintext", "use_cache": false }' ``` ### Notes * The API supports both synchronous and asynchronous processing * For pages requiring longer processing time, use the status check endpoint to poll for results * Use webhooks for automatic notification when processing completes * Cache results are available for 15 days # Metadata Scrapper This API endpoint allows you to extract content from websites in either plaintext or markdown format. ### Endpoint ``` POST https://api.yetanotherapi.com/web-scrapper/ ``` ### Headers | Header | Required | Description | | ------------ | -------- | --------------------------- | | Content-Type | Yes | Must be `application/json` | | x-api-key | Yes | Your API authentication key | ### Request Body ***Though you don't have to explicitly mention about metadata. we will scrape it by default and just make API call using below payload.*** ```json { "url": "https://example.com", "output_type": "plaintext", "use_cache": false, "webhook": "https://your-webhook-url.com" (optional) } ``` #### Parameters | Parameter | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------- | | url | string | Yes | The URL of the website to scrape | | output\_type | string | No | Either "plaintext" (default) or "markdown" | | use\_cache | boolean | No | If true, returns cached result if available. Default: false | | webhook | string | No | URL to receive webhook notification when processing is complete | ### Response #### Immediate Response (HTTP 200) If processing completes within 20 seconds, you'll receive the full result: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "completed", "timestamp": 1635545600, "text_content": "Extracted text content...", "meta": { "title": "Page Title", "description": "Meta description..." }, "links": [ { "text": "Link text", "url": "https://example.com/link", "type": "internal" } ], "images": [ { "url": "https://example.com/image.jpg", "alt": "Image description" } ] } ``` #### Processing Response (HTTP 202) If processing takes longer than 20 seconds: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "processing", "message": "Processing your request. Please check status later." } ``` #### Error Response (HTTP 4XX/5XX) ```json { "error": "ERROR_CODE: Error message" } ``` Common error codes: * E001: Invalid request format * E003: Invalid URL format * E006: Storage service error * E008: Content processing failed ### Response Fields | Field | Type | Description | | ------------- | ------ | ------------------------------------------------------ | | request\_id | string | Unique identifier for the request | | url | string | The URL that was scraped | | status | string | Status of the request ("completed" or "processing") | | timestamp | number | Unix timestamp of when the request was processed | | text\_content | string | The extracted text content (if output\_type=plaintext) | | meta | object | Metadata from the page | | links | array | Array of links found on the page | | images | array | Array of images found on the page | ### Example Curl Request ```bash curl --location 'https://api.yetanotherapi.com/web-scrapper/' \ --header 'Content-Type: application/json' \ --header 'x-api-key: your-api-key' \ --data '{ "url": "https://example.com", "output_type": "plaintext", "use_cache": false }' ``` ### Notes * The API supports both synchronous and asynchronous processing * For pages requiring longer processing time, use the status check endpoint to poll for results * Use webhooks for automatic notification when processing completes * Cache results are available for 15 days # Webhook Notification When you provide a webhook URL in your scraping request, our system will automatically send the results to your specified endpoint once processing is complete. ### Webhook Configuration Add the webhook URL to your scraping request: ```json { "url": "https://example.com", "webhook": "https://your-webhook-url.com/endpoint" } ``` ### Webhook Request Details #### Headers | Header | Value | | ------------ | ----------------------- | | Content-Type | application/json | | User-Agent | Web-Scraper-Webhook/1.0 | #### Payload Structure The webhook will send a POST request with the following JSON structure: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "status": "completed", "url": "https://example.com", "timestamp": 1635545600, "content": { "txt": "Extracted text content...", "markdown": "# Markdown content...", // If markdown was requested "meta": { "title": "Page Title", "description": "Meta description...", "og:image": "https://example.com/image.jpg", // ... other meta tags }, "schema": [ // Array of schema.org structured data { "type": "Article", "properties": { "headline": "Article Title", "datePublished": "2023-01-01" } } ], "images": [ { "url": "https://example.com/image.jpg", "alt": "Image description", "title": "Image title" } ], "links": [ { "text": "Link text", "url": "https://example.com/link", "type": "internal", "location": 0 } ] }, "llm_output": { // Only present if LLM processing was requested // Structured JSON based on the provided prompt } } ``` ### Webhook Behavior #### Retry Policy * Maximum retries: 3 attempts * Retry interval: Exponential backoff starting at 5 seconds * Timeout: 10 seconds per attempt #### Success Criteria * HTTP 2XX response is considered successful * Any other response code will trigger a retry * After all retry attempts are exhausted, the webhook status will be marked as failed #### Error Handling If the webhook delivery fails, the status can be checked via the status endpoint: ```json { "webhook_status": "failed", "webhook_error": "Failed to deliver webhook notification", "webhook_timestamp": 1635545600 } ``` ### Testing Webhooks For development and testing, we recommend: 1. Using tools like [webhook.site](https://webhook.site) for initial testing 2. Setting up a local tunnel using [ngrok](https://ngrok.com) for development 3. Implementing a test endpoint that logs webhook payloads # Status Check This endpoint allows you to check the status and retrieve results of a previously submitted scraping request. ### Endpoint ``` GET https://api.yetanotherapi.com/web-scrapper/{request_id} ``` ### Headers | Header | Required | Description | | --------- | -------- | --------------------------- | | x-api-key | Yes | Your API authentication key | ### Path Parameters | Parameter | Type | Description | | ----------- | ------ | -------------------------------------------- | | request\_id | string | The request ID returned from the scraper API | ### Response #### Success Response (HTTP 200) For completed requests: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "completed", "timestamp": 1635545600, "content": { "text": "Extracted text content...", "markdown": "# Extracted markdown content...", // If markdown was requested "meta": { "title": "Page Title", "description": "Meta description..." }, "links": [ { "text": "Link text", "url": "https://example.com/link", "type": "internal" } ], "images": [ { "url": "https://example.com/image.jpg", "alt": "Image description" } ] }, "llm_output": { // Only present if LLM processing was requested // Structured JSON output based on the prompt } } ``` #### Processing Response (HTTP 200) For requests still processing: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "processing", "timestamp": 1635545600 } ``` #### Error Response (HTTP 4XX/5XX) ```json { "error": "ERROR_CODE: Error message" } ``` Common error codes: * E001: Invalid request format * E002: Request ID not found * E006: Storage service error ### Response Fields | Field | Type | Description | | ----------- | ------ | --------------------------------------------------- | | request\_id | string | Unique identifier for the request | | url | string | The URL that was scraped | | status | string | Current status of the request | | timestamp | number | Unix timestamp of last status update | | content | object | Contains extracted content if status is "completed" | | llm\_output | object | Present only if LLM processing was requested | #### Possible Status Values | Status | Description | | ---------- | ----------------------------------------------- | | received | Request has been received but not yet processed | | processing | Request is currently being processed | | completed | Processing has completed successfully | | failed | Processing failed with an error | ### Example Curl Request ```bash curl --location 'https://api.yetanotherapi.com/web-scrapper/550e8400-e29b-41d4-a716-446655440000' \ --header 'x-api-key: your-api-key' ``` ### Error Handling If processing failed, the response will include error details: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "failed", "timestamp": 1635545600, "error": "E008: Content processing failed", "error_trace": "Detailed error information" // Only in development environment } ``` ### Notes * Polling interval should be at least 5 minutes * Results are available for 15 hours after completion # LLM Web Scraper ### Overview The LLM Web Scraper API combines powerful web scraping capabilities with Language Model processing to extract and structure web content intelligently. It can analyze web pages and return structured data based on your specific requirements. ### Base URL 
POST https://api.yetanotherapi.com/web-scrapper/
### Authentication All requests require an API key passed in the `x-api-key` header. ### Request Headers | Header | Required | Description | | ------------ | -------- | --------------------------- | | Content-Type | Yes | Must be `application/json` | | x-api-key | Yes | Your API authentication key | ### Request Body ```json { "url": "https://example.com", "output_type": "plaintext", //optional "use_llm": true, "prompt": "Extract product details including name, price, and specifications", "openai_key_id": "752724", //optional but recommended "use_cache": false, //optional "webhook": "https://your-webhook-url.com" //optional } ``` #### Request Parameters | Parameter | Type | Required | Default | Description | | --------------- | ------- | -------- | --------- | ------------------------------------------------------------ | | url | string | Yes | - | The URL of the website to scrape | | output\_type | string | No | plaintext | Either "plaintext" or "markdown" | | use\_llm | boolean | Yes | - | Must be set to true for LLM processing | | prompt | string | Yes | - | Instructions for the LLM about what to extract | | openai\_key\_id | string | No | null | Optional ID of your registered OpenAI key | | use\_cache | boolean | No | false | If true, returns cached result if available | | webhook | string | No | null | URL to receive webhook notification when processing complete | ## Important Parameter Notes 1. **Cache Behavior** * When `use_cache: true`, all other parameters except `url` are ignored * Returns most recent cached result for the URL * 404 error if no cache exists 2. **OpenAI Key ID** * Optional parameter * If provided, uses the specified OpenAI key from your account * If not provided, uses your most recently added OpenAI key * Manage multiple keys through your [yetanotherapi dashboard](https://app.yetanotherapi.com/integration) 3. **Webhook** * Optional callback URL for asynchronous processing * Receives full results when processing completes * Must be publicly accessible HTTPS endpoint ### Responses #### Immediate Success Response (HTTP 200) When processing completes within 20 seconds: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "completed", "timestamp": 1635545600, "content": { "text": "Extracted text content...", "meta": { "title": "Page Title", "description": "Meta description..." }, "links": [ { "text": "Link text", "url": "https://example.com/link", "type": "internal" } ], "images": [ { "url": "https://example.com/image.jpg", "alt": "Image description" } ] }, "llm_output": { // Structured JSON based on prompt } } ``` #### Processing Response (HTTP 202) When processing takes longer than 20 seconds: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "processing", "message": "Processing your request. Please check status later." } ``` #### Error Response (HTTP 4XX/5XX) ```json { "error": "ERROR_CODE: Error message" } ``` ### Error Codes | Code | Description | HTTP Status | | ---- | ------------------------- | ----------- | | E001 | Invalid request format | 400 | | E003 | Invalid URL format | 400 | | E004 | Authentication error | 401 | | E008 | Content processing failed | 500 | | E009 | Validation error | 400 | ### Webhook Integration When providing a webhook URL, you'll receive a POST request with the complete results: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "status": "completed", "url": "https://example.com", "timestamp": 1635545600, "content": { // Scraped content }, "llm_output": { // LLM processed data } } ``` # Basic Text This guide provides examples of basic text prompts for the LLM Web Scraper API. These prompts are ideal when you want more natural, flexible output without strict structure requirements. ### Article Analysis #### News Article Analysis ```json { "url": "https://example.com/news-article", "prompt": "Analyze this news article and provide: the main story topic, key events described, important people mentioned, locations involved, and any significant statistics or numbers cited. Also note any bias or perspective in the reporting." } ``` Example Response: ```json { "mainTopic": "Global Climate Summit 2024", "keyEvents": [ "Agreement signed by 195 countries", "New emissions targets set for 2030", "$100B climate fund established" ], "keyPeople": [ { "name": "Dr. Sarah Chen", "role": "UN Climate Chief", "significance": "Led negotiations" }, { "name": "John Smith", "role": "EU Climate Commissioner", "significance": "Proposed funding mechanism" } ], "locations": [ { "place": "Geneva, Switzerland", "context": "Summit location" }, { "place": "Pacific Islands", "context": "Most affected region discussed" } ], "statistics": [ { "value": "45% reduction", "context": "Proposed emissions cut by 2030" }, { "value": "$100 billion", "context": "Annual climate fund size" } ], "reportingPerspective": "Generally neutral with emphasis on urgency of climate action" } ``` #### Blog Post Analysis ```json { "url": "https://example.com/blog-post", "prompt": "Analyze this blog post and extract: the main topic, key arguments or points made, any examples or case studies used, the author's conclusions, and the overall tone. Also identify any calls to action or reader recommendations." } ``` Example Response: ```json { "mainTopic": "Remote Work Productivity", "keyPoints": [ "Remote work increases productivity by reducing distractions", "Digital tools enable effective collaboration", "Work-life balance improves employee retention" ], "caseStudies": [ { "company": "TechCorp Inc", "outcome": "37% productivity increase after remote transition", "keyLearnings": [ "Clear communication protocols needed", "Regular virtual team meetings essential" ] } ], "conclusions": [ "Remote work is here to stay", "Success requires proper tools and policies", "Hybrid models likely to become standard" ], "tone": "Optimistic and practical", "callsToAction": [ "Assess your team's remote work readiness", "Invest in collaboration tools", "Develop clear remote work policies" ] } ``` ### Product Review Analysis #### Single Product Review ```json { "url": "https://example.com/product-review", "prompt": "Analyze this product review and extract: the overall verdict, main pros and cons, key features discussed, any performance metrics mentioned, value for money assessment, and comparison with competitors. Also note any specific use cases or scenarios mentioned." } ``` Example Response: ```json { "overallVerdict": "Excellent premium smartphone with outstanding camera", "rating": 4.5, "pros": [ "Exceptional camera quality", "All-day battery life", "Premium build quality" ], "cons": [ "High price point", "No expandable storage", "Average charging speed" ], "keyFeatures": [ { "feature": "Camera System", "details": "48MP main, 12MP ultra-wide", "performance": "Outstanding in low light" }, { "feature": "Battery", "details": "4500mAh", "performance": "15-18 hours typical use" } ], "valueAssessment": "Premium priced but justified for photography enthusiasts", "competitorComparison": [ { "competitor": "Phone Y", "advantage": "Better camera", "disadvantage": "Higher price" } ], "useCases": [ "Professional photography", "Heavy multitasking", "Gaming" ] } ``` ### E-commerce Product Page #### Product Information Extraction ```json { "url": "https://example.com/product", "prompt": "Analyze this product page and extract all relevant information including: product name, description, pricing details, available variants, technical specifications, customer ratings, key features, shipping options, and any promotional offers. Also identify any unique selling points or special features highlighted." } ``` Example Response: ```json { "productInfo": { "name": "Ultra HD Smart TV X1000", "description": "Next-generation smart TV with AI-powered features", "pricing": { "current": 899.99, "original": 1299.99, "discount": "30% off", "savingsAmount": 400 }, "variants": [ { "size": "55 inch", "price": 899.99, "availability": "In Stock" }, { "size": "65 inch", "price": 1299.99, "availability": "Pre-order" } ] }, "specifications": { "display": "4K Ultra HD", "hdrSupport": "HDR10+", "refreshRate": "120Hz", "smartFeatures": [ "Voice control", "AI upscaling", "Gaming mode" ] }, "customerFeedback": { "averageRating": 4.7, "totalReviews": 256, "highlightedFeatures": [ "Picture quality", "Smart features", "Easy setup" ] }, "shipping": { "options": [ { "method": "Standard", "cost": "Free", "time": "3-5 days" }, { "method": "Express", "cost": 29.99, "time": "1-2 days" } ] }, "promotions": [ { "type": "Bundle Deal", "description": "Free soundbar with purchase", "value": "199.99" } ], "uniqueFeatures": [ "AI-powered upscaling", "Gaming-optimized HDMI 2.1", "Room temperature adaptive brightness" ] } ``` ### Best Practices for Basic Prompts 1. **Be Specific** * Clearly state what information you want * List out all desired elements * Indicate if you need specific details 2. **Consider Scope** * Break down complex requests into categories * Specify any particular areas of focus * Indicate if certain aspects can be skipped 3. **Request Context** * Ask for related information when relevant * Request comparisons or references * Include temporal context if important 4. **Include Qualitative Elements** * Request tone analysis when relevant * Ask for subjective assessments * Include sentiment analysis requirements 5. **Handle Missing Information** * Specify how to handle missing data * Request confidence levels * Ask for explanations of gaps ### Common Use Cases 1. **Content Analysis** * Blog posts * News articles * Research papers * Technical documentation 2. **Product Information** * Product reviews * Specifications * Comparisons * Pricing analysis 3. **User Generated Content** * Reviews * Comments * Forum discussions * Social media posts 4. **Documentation** * Technical specs * User guides * API documentation * Release notes # Structured JSON This guide demonstrates how to create prompts that return precisely structured JSON responses from the LLM Web Scraper API. These prompts are ideal when you need consistent, predictable data structures for integration with other systems. ### Key Principles for Structured Prompts 1. **Explicitly Define Keys** * Specify exact key names * Define expected data types * Indicate required vs optional fields 2. **Specify Nesting Structure** * Define object hierarchies * Specify array structures * Indicate relationships between objects 3. **Include Validation Rules** * Specify allowed values * Define number ranges * Indicate format requirements ### E-commerce Examples #### Product Details Extraction ```json { "url": "https://example.com/product", "prompt": "Extract product information using these exact keys and types: 'productName' (string), 'manufacturer' (string), 'price' (object with keys: 'current' (number), 'original' (number), 'currency' (string)), 'availability' (string: either 'In Stock', 'Out of Stock', or 'Pre-order'), 'specifications' (array of objects with keys: 'name' (string), 'value' (string)), 'features' (array of strings), 'shipping' (object with keys: 'free' (boolean), 'methods' (array of objects with keys: 'name' (string), 'price' (number), 'duration' (string)))" } ``` Example Response: ```json { "productName": "Professional DSLR Camera X100", "manufacturer": "PhotoTech", "price": { "current": 1299.99, "original": 1499.99, "currency": "USD" }, "availability": "In Stock", "specifications": [ { "name": "Sensor Type", "value": "Full Frame CMOS" }, { "name": "Resolution", "value": "24.2 MP" }, { "name": "Shutter Speed", "value": "1/8000 to 30 sec" } ], "features": [ "4K video recording", "Built-in WiFi", "Weather-sealed body", "Dual card slots" ], "shipping": { "free": true, "methods": [ { "name": "Standard", "price": 0, "duration": "3-5 business days" }, { "name": "Express", "price": 29.99, "duration": "1-2 business days" } ] } } ``` #### Multiple Product Comparison ```json { "url": "https://example.com/category", "prompt": "Extract information for all products on the page using this structure: 'products' (array of objects), each product must have: 'id' (string), 'name' (string), 'brand' (string), 'category' (string), 'price' (number), 'rating' (object with keys: 'score' (number 0-5), 'count' (number)), 'specs' (object with keys matching exactly what's found in the product specs table), 'comparisonPoints' (array of objects with keys: 'feature' (string), 'value' (string), 'relativeMerit' (string: 'better', 'worse', or 'same'))" } ``` Example Response: ```json { "products": [ { "id": "CAM-X100", "name": "DSLR X100", "brand": "PhotoTech", "category": "Professional Cameras", "price": 1299.99, "rating": { "score": 4.8, "count": 245 }, "specs": { "sensorType": "Full Frame", "resolution": "24.2MP", "weight": "780g" }, "comparisonPoints": [ { "feature": "Image Quality", "value": "Exceptional", "relativeMerit": "better" }, { "feature": "Battery Life", "value": "1200 shots", "relativeMerit": "same" } ] } ] } ``` ### Article Examples #### Structured Article Analysis ```json { "url": "https://example.com/article", "prompt": "Analyze the article using this exact structure: 'metadata' (object with keys: 'title' (string), 'author' (string), 'publishDate' (string in YYYY-MM-DD format), 'category' (string), 'readingTime' (number in minutes)), 'content' (object with keys: 'summary' (string), 'mainPoints' (array of strings), 'conclusions' (array of strings)), 'analysis' (object with keys: 'tone' (string: 'positive', 'negative', 'neutral'), 'audience' (string), 'expertise_level' (string: 'beginner', 'intermediate', 'advanced')), 'references' (array of objects with keys: 'text' (string), 'source' (string))" } ``` Example Response: ```json { "metadata": { "title": "The Future of AI in Healthcare", "author": "Dr. Sarah Johnson", "publishDate": "2024-01-15", "category": "Technology", "readingTime": 12 }, "content": { "summary": "Comprehensive analysis of AI's growing role in healthcare delivery and diagnosis", "mainPoints": [ "AI improving diagnostic accuracy by 40%", "Reduction in administrative workload", "Enhanced patient monitoring capabilities", "Cost savings through automation" ], "conclusions": [ "AI will be integral to future healthcare", "Human oversight remains crucial", "Cost barriers decreasing rapidly" ] }, "analysis": { "tone": "positive", "audience": "Healthcare professionals", "expertise_level": "intermediate" }, "references": [ { "text": "WHO AI in Healthcare Report 2023", "source": "World Health Organization" }, { "text": "AI Diagnostic Accuracy Study", "source": "Medical AI Journal" } ] } ``` ### FAQ Page Structured Extraction ```json { "url": "https://example.com/faq", "prompt": "Extract FAQ content using this structure: 'categories' (array of objects with keys: 'name' (string), 'description' (string), 'questions' (array of objects with keys: 'question' (string), 'answer' (string), 'tags' (array of strings), 'related_questions' (array of numbers referencing question indices))). Each answer should be concise and formatted as a single paragraph." } ``` Example Response: ``` ``` # Best Practices ### Best Practices #### 1. Prompt Engineering **Clear and Specific Instructions** ```json // ❌ Bad { "prompt": "Get product information" } // ✅ Good { "prompt": "Extract the product's name, current price, original price (if on sale), available sizes, and color options. For prices, include the currency symbol." } ``` **Define Expected Format** ```json // ❌ Bad { "prompt": "What are the key features of this product?" } // ✅ Good { "prompt": "List the product's key features as an array of strings, with each feature being a concise single sentence." } ``` **Include Validation Rules** ```json // ❌ Bad { "prompt": "Get the product price and rating" } // ✅ Good { "prompt": "Extract the product price (as a number without currency symbol) and rating (must be between 0 and 5, with one decimal place)" } ``` #### 2. Data Structuring **Clear Hierarchy** ```json // ❌ Bad { "prompt": "Get all prices from the page" } // ✅ Good { "prompt": "Extract pricing information in this structure: base_price (number), additional_options (array of objects with name and price), discounts (array of objects with description and amount)" } ``` **Handle Missing Data** ```json // ❌ Bad { "prompt": "Get the author's name and bio" } // ✅ Good { "prompt": "Extract the author's details with these rules: name (string, use 'Anonymous' if not found), bio (string, use null if not present), role (string, use 'Contributor' if not specified)" } ``` #### 3. Performance Optimization **Focused Extraction** ```json // ❌ Bad { "prompt": "Get everything from the page" } // ✅ Good { "prompt": "Extract only the technical specifications table, converting it to a JSON object with spec_name as keys and spec_value as values" } ``` **Batch Processing** ```json // ❌ Bad: Multiple separate requests { "prompt": "Get product prices" } { "prompt": "Get product features" } // ✅ Good: Single comprehensive request { "prompt": "Extract all product information in a single structured response: prices (object), features (array), specifications (object)" } ``` # Use Cases #### 1. E-commerce **Product Detail Extraction** ```json { "prompt": "Extract product information using these exact keys: productName, brand, currentPrice, originalPrice, discount (calculate if both prices present), availability (in stock/out of stock), variants (array of size/color combinations), specifications (key-value pairs), features (array), shipping (object with methods and prices)" } ``` **Product Comparison** ```json { "prompt": "Compare all products on the page using this structure: products (array), each with: name, price, key_features (array), pros (array), cons (array), best_for (string describing ideal use case). Add a comparison_summary highlighting key differences." } ``` **Review Analysis** ```json { "prompt": "Analyze product reviews and provide: averageRating, totalReviews, sentimentBreakdown (positive/negative/neutral counts), commonPros (array), commonCons (array), featureSentiment (how different features are rated), recommendationRate (percentage who recommend)" } ``` #### 2. Content Analysis **Article Summarization** ```json { "prompt": "Analyze this article and provide: title, author, publishDate, summary (150 words max), mainPoints (array), conclusions (array), tone (formal/informal), targetAudience, expertiseLevel (beginner/intermediate/advanced), keyTerms (array with definitions)" } ``` **Technical Documentation** ```json { "prompt": "Extract from this technical documentation: apiEndpoints (array with method, path, parameters, responses), authenticationMethods (array), errorCodes (array with code, message, solution), examples (array of code snippets with language and description)" } ``` **News Analysis** ```json { "prompt": "Analyze this news article for: mainTopic, eventDate, location, keyPeople (array with roles), organizations (array), quotes (array with speaker and context), statistics (array), sources (array), bias (object analyzing potential biases)" } ``` #### 3. Research and Analysis **Academic Paper Analysis** ```json { "prompt": "Extract from this research paper: title, authors (array), abstract, methodology, keyFindings (array), conclusions (array), references (array), statisticalSignificance (any mentioned p-values or confidence intervals), limitations (array)" } ``` **Market Research** ```json { "prompt": "Analyze this market report for: marketSize, growthRate, keyPlayers (array with market share), trends (array), opportunities (array), threats (array), segmentAnalysis (breakdown by market segments), forecast (future projections)" } ``` #### 4. Real Estate **Property Listing Analysis** ```json { "prompt": "Extract property details: price, location (object with address components), propertyType, size (square footage), bedrooms, bathrooms, features (array), amenities (array), nearbyServices (array), taxes, utilities, photos (array with descriptions), contactInfo" } ``` **Market Comparison** ```json { "prompt": "Compare properties on this page: create array of properties, each with standardized metrics (price per sqft, total rooms, condition, etc.). Add marketAnalysis object with priceRange, averagePrice, commonFeatures, uniqueFeatures" } ``` #### 5. Job Listings **Job Description Analysis** ```json { "prompt": "Extract job details: title, company, location (remote/hybrid/onsite), salary (if listed), requirements (array), responsibilities (array), benefits (array), requiredSkills (array), preferredSkills (array), experienceLevel, educationRequirements, applicationDeadline" } ``` # Status Check This endpoint allows you to check the status and retrieve results of a previously submitted scraping request. ### Endpoint ``` GET https://api.yetanotherapi.com/web-scrapper/{request_id} ``` ### Headers | Header | Required | Description | | --------- | -------- | --------------------------- | | x-api-key | Yes | Your API authentication key | ### Path Parameters | Parameter | Type | Description | | ----------- | ------ | -------------------------------------------- | | request\_id | string | The request ID returned from the scraper API | ### Response #### Success Response (HTTP 200) For completed requests: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "completed", "timestamp": 1635545600, "content": { "text": "Extracted text content...", "markdown": "# Extracted markdown content...", // If markdown was requested "meta": { "title": "Page Title", "description": "Meta description..." }, "links": [ { "text": "Link text", "url": "https://example.com/link", "type": "internal" } ], "images": [ { "url": "https://example.com/image.jpg", "alt": "Image description" } ] }, "llm_output": { // Only present if LLM processing was requested // Structured JSON output based on the prompt } } ``` #### Processing Response (HTTP 200) For requests still processing: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "processing", "timestamp": 1635545600 } ``` #### Error Response (HTTP 4XX/5XX) ```json { "error": "ERROR_CODE: Error message" } ``` Common error codes: * E001: Invalid request format * E002: Request ID not found * E006: Storage service error ### Response Fields | Field | Type | Description | | ----------- | ------ | --------------------------------------------------- | | request\_id | string | Unique identifier for the request | | url | string | The URL that was scraped | | status | string | Current status of the request | | timestamp | number | Unix timestamp of last status update | | content | object | Contains extracted content if status is "completed" | | llm\_output | object | Present only if LLM processing was requested | #### Possible Status Values | Status | Description | | ---------- | ----------------------------------------------- | | received | Request has been received but not yet processed | | processing | Request is currently being processed | | completed | Processing has completed successfully | | failed | Processing failed with an error | ### Example Curl Request ```bash curl --location 'https://api.yetanotherapi.com/web-scrapper/550e8400-e29b-41d4-a716-446655440000' \ --header 'x-api-key: your-api-key' ``` ### Error Handling If processing failed, the response will include error details: ```json { "request_id": "550e8400-e29b-41d4-a716-446655440000", "url": "https://example.com", "status": "failed", "timestamp": 1635545600, "error": "E008: Content processing failed", "error_trace": "Detailed error information" // Only in development environment } ``` ### Notes * Polling interval should be at least 5 minutes * Results are available for 15 hours after completion # UChat Webhook System ### Overview The Bot Webhook API provides functionality to manage user interactions in a messaging system, including user creation, field updates, and flow management. This API integrates with uChat's backend API services. **Important Notes:** * This API is FREE and will continue to remain free. * **IMPORTANT**: Authentication process will be updated in future releases. * Currently works best for whatsapp bots. We plan to expend this to support multiple channels. * For support contact: Endpoint URL: `https://api.yetanotherapi.com/bot-webhook` ### Authentication All API requests require Bearer token authentication. ```http Authorization: Bearer ``` Your API token can be obtained from the UChat account settings. ### Endpoints #### Send Bot Webhook `POST /bot-webhook` Handles user management and flow distribution in the messaging system. **Request** **Headers:** * `Content-Type: application/json` (Required) * `Authorization: Bearer ` (Required) **Body Parameters:** ```json { "phone_number": string, // Required: User's phone number "flow_ns": string, // Required: Flow namespace identifier "user_fields": { // Optional: Custom user fields "field_name": "value" } } ``` **Response** **Success Response (200 OK)** ```json { "user_ns_retrieval": { "success": true, "message": string, "user_ns": string }, "custom_fields_update": { "success": true, "message": string, "details": [ { "field": string, "status": string } ] }, "flow_sent": { "success": true, "message": string }, "request_id": string } ``` **Partial Success Response (206 Partial Content)** Returned when some operations succeed but others fail. **Error Responses:** * `400 Bad Request`: Missing required fields or invalid JSON * `404 Not Found`: User not found and creation failed * `500 Internal Server Error`: Server-side error Each response includes a unique `request_id` in both the response body and header (`X-Request-ID`). **Example Request** ```bash curl --location 'https://api.yetanotherapi.com/bot-webhook' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "phone_number": "917406796004", "flow_ns": "f105713s755621", "user_fields": { "scheme_name": "test", "transaction_code": "test" } }' ``` ### Error Handling The API implements a robust error handling system with: * Input validation * Retry mechanism (max 3 retries) for user retrieval * Detailed error messages in responses * Comprehensive error logging ### Logging and Monitoring All API interactions are logged with: * Request details * API call traces * Execution timestamps * Response data * Error information Each request can be tracked using its unique `request_id`. ### Support For any queries or support, please contact: