Rate Limits

Understanding API rate limits and how to handle them gracefully in your applications.

Overview

API rate limits prevent abuse and ensure fair usage for all users. Different endpoints have different rate limits based on their resource intensity.

Rate Limit Tiers

Each endpoint has three rate limit tiers:

• Per Minute: Short-term burst protection
• Per Hour: Medium-term usage limit
• Per Day: Daily quota for sustained usage

Limits by Endpoint

Rate Limit Headers

Every API response includes headers that tell you your current rate limit status:

Example Response Headers

HTTP/1.1 200 OK
X-RateLimit-Limit: 30
X-RateLimit-Remaining: 25
X-RateLimit-Reset: 1705324200
Content-Type: application/json

{ "success": true, ... }

When You Exceed Limits

When you exceed a rate limit, you'll receive a 429 Too Many Requests response:

{
  "success": false,
  "error": {
    "code": "RATE_LIMITED",
    "message": "Rate limit exceeded for this endpoint",
    "details": {
      "limit": 30,
      "window": "minute",
      "retry_after": 45
    }
  },
  "meta": {
    "request_id": "req_...",
    "timestamp": "2025-01-15T10:30:00Z",
    "version": "v1"
  }
}

The retry_after field tells you how many seconds to wait before retrying.

Handling Rate Limits in Code

JavaScript Example

async function makeRequest(url, options, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await fetch(url, options);
    
    // Log rate limit status
    const limit = response.headers.get('X-RateLimit-Limit');
    const remaining = response.headers.get('X-RateLimit-Remaining');
    console.log(`Rate limit: ${remaining}/${limit} remaining`);
    
    if (response.status === 429) {
      // Get retry delay from response or use exponential backoff
      const retryAfter = response.headers.get('Retry-After') || Math.pow(2, attempt);
      const waitMs = parseInt(retryAfter) * 1000;
      
      console.log(`Rate limited. Waiting ${waitMs}ms before retry...`);
      await new Promise(resolve => setTimeout(resolve, waitMs));
      continue;
    }
    
    return response;
  }
  
  throw new Error('Max retries exceeded');
}

// Usage
try {
  const response = await makeRequest(
    'https://agentsfordata.com/api/public/v1/convert',
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiKey}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({ ... })
    }
  );
  
  const data = await response.json();
  console.log('Success:', data);
} catch (error) {
  console.error('Failed after retries:', error);
}

Python Example

import requests
import time

def make_request_with_retry(url, headers, json_data, max_retries=3):
    """Make API request with automatic retry on rate limit."""
    for attempt in range(max_retries):
        response = requests.post(url, headers=headers, json=json_data)
        
        # Log rate limit status
        limit = response.headers.get('X-RateLimit-Limit')
        remaining = response.headers.get('X-RateLimit-Remaining')
        print(f'Rate limit: {remaining}/{limit} remaining')
        
        if response.status_code == 429:
            # Get retry delay from response or use exponential backoff
            retry_after = int(response.headers.get('Retry-After', 2 ** attempt))
            
            print(f'Rate limited. Waiting {retry_after} seconds...')
            time.sleep(retry_after)
            continue
        
        response.raise_for_status()
        return response.json()
    
    raise Exception('Max retries exceeded')

# Usage
try:
    result = make_request_with_retry(
        'https://agentsfordata.com/api/public/v1/convert',
        headers={
            'Authorization': f'Bearer {api_key}',
            'Content-Type': 'application/json'
        },
        json_data={'source_format': 'csv', ...}
    )
    print('Success:', result)
except Exception as e:
    print(f'Failed: {e}')

Best Practices

1. Monitor Rate Limit Headers

Always check the rate limit headers in responses to know when you're approaching limits:

const remaining = parseInt(response.headers.get('X-RateLimit-Remaining'));
const limit = parseInt(response.headers.get('X-RateLimit-Limit'));
const percentUsed = ((limit - remaining) / limit) * 100;

if (percentUsed > 80) {
  console.warn(`Warning: ${percentUsed.toFixed(0)}% of rate limit used`);
}

2. Implement Exponential Backoff

When retrying after rate limits, use exponential backoff instead of fixed delays:

const delay = Math.min(1000 * Math.pow(2, attempt), 32000); // Cap at 32s
await new Promise(resolve => setTimeout(resolve, delay));

3. Batch Operations Intelligently

For batch conversions, respect the per-minute limits:

async function batchConvert(files, concurrency = 5) {
  const results = [];
  
  for (let i = 0; i < files.length; i += concurrency) {
    const batch = files.slice(i, i + concurrency);
    const batchResults = await Promise.all(
      batch.map(file => convertFile(file))
    );
    results.push(...batchResults);
    
    // Small delay between batches to avoid hitting limits
    if (i + concurrency < files.length) {
      await new Promise(resolve => setTimeout(resolve, 6000)); // 6s delay
    }
  }
  
  return results;
}

4. Cache Format Listings

The formats endpoint has high limits because it's meant to be cached:

// Cache formats for 1 hour
let formatsCacheTime = 0;
let formatsCache = null;

async function getFormats() {
  const now = Date.now();
  if (formatsCache && (now - formatsCacheTime) < 3600000) {
    return formatsCache;
  }
  
  const response = await fetch(
    'https://agentsfordata.com/api/public/v1/convert/formats',
    { headers: { 'Authorization': `Bearer ${apiKey}` } }
  );
  
  const data = await response.json();
  formatsCache = data.data.formats;
  formatsCacheTime = now;
  
  return formatsCache;
}

5. Use Queues for High Volume

For high-volume operations, use a queue system to manage rate limits:

• Queue conversion requests
• Process queue respecting rate limits
• Implement priority for urgent conversions
• Track progress and failures

Enterprise Options

Need higher rate limits for your application? We offer custom rate limits for enterprise customers:

• Custom per-minute, per-hour, and per-day limits
• Dedicated infrastructure for high-volume usage
• Priority support
• SLA guarantees

Contact us at enterprise@agentsfordata.com to discuss your needs.