> ## Documentation Index > Fetch the complete documentation index at: https://docs.promptloop.com/llms.txt > Use this file to discover all available pages before exploring further. # API Overview > Run AI tasks on large datasets with PromptLoop ## Overview PromptLoop AI's Batch API v0 allows you to process large datasets, such as CSVs of websites to scrape, using AI tasks. This guide will help you get started with sending inputs, monitoring jobs, and retrieving results. ## API Authentication Before using the Batch API, ensure you have an API key. You can obtain this from your PromptLoop dashboard. ### Creating an API Key On your PromptLoop dashboard, navigate to the [API settings page](https://promptloop.com/account/settings) and create a key. You will only be able to view the key once, so make sure to store it securely. PromptLoop does not store your key. ### Setting Up Authentication Include your API key in the `x-api-key` header of all requests. Here's an example using cURL: ``` curl --request GET \ --url https://api.promptloop.com/v0/batches/{id}/results \ --header 'x-api-key: YOUR_API_KEY' ``` ## Data Format The Batch API expects input data in JSONL (JSON Lines) format. Each line represents a single input object. ## Input Structure Each input object should have: `data_uuid:` A unique identifier for the input `inputs:` An array containing the input data. Tasks specify whether the input should be a link or a string. This will be checked in the validation step. Example: ```json theme={null} {"data_uuid": "1", "inputs": ["https://example.com"]} {"data_uuid": "2", "inputs": ["https://another-example.com"]} ``` ### Validating Input Data Ensure your input data is properly formatted and includes all required fields. Invalid data will result in a `400 Bad Request` error. You can use the `v0/batches/validate-data` endpoint to validate your input data before submitting a job. The response will indicate whether the data is valid or not. **Too many inputs:** ```json theme={null} { "status": "success", "data": { "job_count": 4, "job_improper_format_count": 1, "job_improper_format": [ { "error_index": 4, "error_message": "JSONL object inputs length does not match task inputs length: 1 vs 2. Expecting inputs schema: [{\"name\":\"websiteUrl\",\"schema\":\"LinkItemSchema\",\"description\":\"Website URL\"}]" } ] }, "message": "", "error": null } ``` **Incorrect Input Type:** ```json theme={null} { "status": "success", "data": { "job_count": 4, "job_improper_format_count": 1, "job_improper_format": [ { "error_index": 4, "error_message": "JSONL object inputs do not match task inputs: Expected a valid URL for input: websiteUrl, received: Not a Link" } ] }, "message": "", "error": null } ``` ## API Workflow Follow these steps to use the Batch API effectively: Send your JSONL data to the `/v0/launch-job` endpoint to start processing. Ensure your data is properly formatted and includes the required fields. ```http theme={null} POST /v0/batches Content-Type: application/json {"data_uuid": "1", "inputs": ["https://example.com"]} {"data_uuid": "2", "inputs": ["https://another-example.com"]} ``` You'll receive a job ID in response, which you'll use in the next steps. Check the progress of your job using the `/v0/job-status/{id}` endpoint. Replace `{id}` with the job ID you received in step 1. ```http theme={null} GET /v0/batches/{id} ``` The response will include the current status of your job (e.g., "in\_progress", "completed", "failed"). Once the job status is "completed", fetch the processed data using the `/v0/batch/{id}/results` endpoint. ```http theme={null} GET /v0/batch/{id}/results ``` The response will contain the results of your batch processing job. The results will be returned in json format, with results presented in key-value pairs within the object. One object for each input row. Each object will also include the `data_uuid` passed in with that row. Order is not guaranteed to be maintained when processing, so use this to identify rows. ## API Rate Limits To ensure fair usage and maintain the performance of our services, we enforce rate limits on our APIs. The rate limits are defined based on the API endpoints and user plans. ### Default Rate Limits Each API has a specific rate limit, measured in Transactions Per Minute (TPM). The following table summarizes the default rate limits: #### Tasks | API Method | Endpoint | TPM Restriction | | ---------- | ---------------- | --------------- | | `GET` | `/v0/tasks` | 20 TPM | | `GET` | `/v0/tasks/{id}` | 10 TPM | | `POST` | `/v0/tasks/{id}` | 5 TPM | #### Batches | API Method | Endpoint | TPM Restriction | | ---------- | --------------------------- | --------------- | | `GET` | `/v0/batches` | 10 TPM | | `POST` | `/v0/batches` | 4 TPM | | `POST` | `/v0/batches/validate-data` | 20 TPM | | `GET` | `/v0/batches/{id}` | 20 TPM | | `PUT` | `/v0/batches/{id}` | 4 TPM | | `GET` | `/v0/batches/{id}/results` | 10 TPM | ### Custom For custom use-cases requiring higher TPM allocations, please reach out to support. Rate limit increases will be granted on a case-by-case basis for users on Company plans. ### Handling Rate Limit Errors If you exceed the rate limit, the API will return a `429 Too Many Requests` response. It's important to implement proper error handling in your application to manage these scenarios effectively. ### Example Rate Limit Error Handling Here’s an example of how to handle rate limit errors in your application: ```javascript theme={null} // Example: Handling rate limit errors in a JavaScript application fetch('https://api.yourdomain.com/[ENDPOINT]') .then(response => { if (response.status === 429) { console.error( 'Rate limit exceeded. Please wait before making more requests.' ); // Optionally, implement a retry mechanism here } else { return response.json(); } }) .then(data => console.log(data)) .catch(error => console.error('Error:', error)); ``` Remember to handle errors and implement retries as necessary throughout this process. ## Error Handling The API uses standard HTTP status codes. Common errors: * 400: Bad Request (e.g., invalid input format) * 401: Unauthorized (invalid API key) * 429: Too Many Requests (rate limit exceeded) Always check the response body for detailed error messages. ## Batch Size Limits Batches are limited to 100mb in size per JSONL upload. Your account will also be limited to existing usage limits which can be viewed on your [dashboard](https://www.promptloop.com/account/settings/usage) For larger batches, we recommend retrieving the results using `output_type=stream` for the `/v0/batch/{id}/results` endpoint. This allows for more efficient transfers. ## Webhooks PromptLoop's Batch API supports webhooks, allowing you to receive real-time updates about your batch processing jobs. ### Setting Up a Webhook When creating a new batch job, you can specify a webhook URL to receive notifications: ``` POST /v0/batches Content-Type: application/json { "webhook_url": "https://your-webhook-endpoint.com", "data": [ {"data_uuid": "1", "inputs": ["https://example.com"]}, {"data_uuid": "2", "inputs": ["https://another-example.com"]} ] } ``` ### Webhook Payload When the batch job completes, PromptLoop will send a POST request to your specified webhook URL with the following payload: ``` { "status": "completed", "message": "Your data processing is complete", "data": "...", // JSON Array of the processed results "timestamp": "2024-07-23T03:57:20.643Z" "batch_id": "12345" } ``` The `data` field contains a JSON array with the processed results for each input, similar to the response from the `/v0/batch/{id}/results` endpoint. The Payload will be sent with a `Content-Type: application/json` header. ### Handling Webhook Requests Ensure your webhook endpoint can: 1. Receive POST requests 2. Process the JSON payload 3. Respond with a 2xx status code to acknowledge receipt Implement proper security measures, such as verifying the webhook source and using HTTPS. ## Next Steps With this overview, you're ready to explore the specific endpoints. Refer to the [API reference](/v0) for detailed information on each endpoint's parameters and responses.