Fetch files with the API

On this page

The fetch API makes it easy to copy files from an HTTP address into your Sirv account.

About the fetch API

The fetch API is one of 40 commands available in the Sirv REST API.

Useful links if you're thinking of using the fetch API:

Fetch API documentation - along with all other API commands.
Postman API collection - quick way to play with Sirv API commands.
Upload file API - to upload a local file instead of a remote URL, use the upload API.

Fetch a file

For each fetch request, you should submit two parameters:

url - the URL of the file to be fetched.
filename - the folder and file name of the file to be saved in your Sirv account.

Example:

{
 "url": "https://www.un.org/sites/un2.un.org/files/aan_660.png",
 "filename": "/folder/path-to/my-new-image.jpg",
}

You can specify any folder path and filename - if the folders don't already exist in your Sirv account, they will be created.

Multiple URLs in one request

Each fetch request can contain up to 20 file URLs. This request will fetch 3 images:

[
  {
    "url": "https://www.w3schools.com/html/img_chania.jpg",
    "filename": "/Example-folder/Italian-backstreet.jpg"
  },
    "url": "https://www.w3schools.com/html/img_girl.jpg",
    "filename": "/Example-folder/jacket-hat.jpg"
  },
    "url": "https://www.w3schools.com/html/pic_trulli.jpg",
    "filename": "/Example-folder/Greek-villas.jpg"
  }
]

Got a CSV? If you have a CSV of URLs, an even easier solution is to use Sirv's bulk upload tool. Every file listed in your CSV will be automatically uploaded to your Sirv account. You can choose the folder structure, file names and you can enrich each file with meta data. See the CSV upload documentation.

HTTP authentication

If the files are protected by HTTP authentication, your request should provide the username and password for authentication:

{
  "url": "https://demo.sirv.com/Example-Folder/lv-M46197_PM2_Front.jpg",
  "filename": "/folder/path-to/my-new-image.jpg",
  "auth": {
    "username": "Example",
    "password": "StrongPassword%50",
  }
}

Fetching limits

There are limits on: number of files; size of files; time to fetch each file.

Hourly fetching limit

You can fetch 1000-2000 files per hour, depending on your account type. This is burstable, allowing triple the limit for 24 hours, once per month.

	Enterprise	Business	Trial	Free
Fetch requests / hour	2000	1000	1000	None

To increase your hourly limit, please describe your requirements to the Sirv support team.

File size limit

The size of the remote file is limited:

	Enterprise	Business	Trial	Free
File size limit	10 MB	5 MB	3 MB	-
Large files	64 MB 20 per hour	32 MB 10 per hour	32 MB 10 per hour	-

If you need to fetch larger files, please tell the Sirv support team how big your largest files are.

Time limit

Files should be fully returned within a time limit. An extra-long limit is provided for slow files.

	Enterprise	Business	Trial	Free
Time limit	10 seconds	10 seconds	10 seconds	-
Slow files	30 seconds 20 per hour	30 seconds 10 per hour	30 seconds 10 per hour	-

How to handle fetch errors

A successful request will return a 200 response. If you receive an error response, follow the advice below:

401 - your API access token may have reached its 20 minute expiry. If so, get a new token and try again.
403 - the remote server refused access to the file. There are many possible reasons, such as forbidden access, rejected IP address or invalid configuration. This probably won't be resolved by retrying - you should contact the person responsible for the file.
404 - the file is missing from the server. Check that the URL is correct. If so, contact the person responsible for the file.
429 - too many requests, when your hourly fetch limit is reached. Wait until the limit refreshes, then continue submitting requests. Your limit, current usage and reset time is shown in each response and also on your Usage page.
503/504 (header) - if Sirv's response header returns a 5xx error, Sirv's upload service may be overloaded. This rare occurence will resolve itself after some time. Please wait 30 minutes and try again.
503 (body) - if a URL in the body has a status code 503, it means an error on the remote server could not return the file. It might work if you try again later.
504 (body) - the remote server terminated the request as a timeout. Contact the person responsible for the server - ask them to whitelist Sirv's IP addresses or user agent.
Timeout was reached - Sirv timed out the request because the server didn't return the file within the allowed time (10 or 30 seconds). Try again after 1 hour and the file might be successfully received.
SSL peer certificate or SSH remote key was not OK - the domain of the remote server does not have a valid SSL certificate. This won't be resolved by retrying - you should contact the person responsible for the server.
Number of redirects hit maximum amount - the remote server did not return the image - it redirected Sirv's request to another location. One redirect is permitted; a second redirect will fail. Do not retry this file as it will fail again.

It will help you to troubleshoot any issue if save each API response in a log.

Wait before retrying

If a file fails to be fetched, you must wait before retrying it. The first retry will be permitted after 1 minute; the second attempt after an additional 2 minutes, then 4 minutes, 9 minutes, 16, 25, 36, 49 minutes etc. If a retry is attempted too soon, it will be counted as an attempt and you'll have to wait longer, so please observe the retry time.

Look for the retryAfter value in the body response, to know what time the next attempt will be permitted.

Failure notifications

To help make you aware of a failed fetch, a notification will be displayed on your Events page. Notifications contain the source URL, referrer, status code and error code. Example:

Example response

Each response contains useful information in the header or body.

Response body

The body of the response contains the most important information.

Of particular value in the body response are:

filename - the filename specified in your request.
success - whether the file was successfully fetched (true/false).
attempted - whether Sirv attempted to fetch the file (true/false). If a file with the same name already exists on Sirv, it will not be fetched.
statusCode - if the fetch failed, the status code will help you decide what to do next.
retryAfter - if the fetch failed, this states when a retry will be permitted.

Below is an example response body for 3 requests (1 succeeded, 1 was not attempted, 1 failed):

[
  {
    "filename":"/folder/path-to/my-new-image.png",
    "success":true,
    "attempted":true,
    "attempts":[
      {
        "url":"https://images.ctfassets.net/8cd2csgvqd3m/4PZg3P4irAYjVwAPcFIw9v/5d52752cda2a7304b748f0e5a5ee9733/Beosound_A5_-_PDP_-_Image_1.png",
        "initiator":{
          "type":"api",
          "remoteAddr":"81.166.37.144",
          "userAgent":"PostmanRuntime/7.32.2",
          "date":"2023-06-10T10:31:12.585Z"
        },
        "statusCode":200,
        "headers":{
          "result":{
            "version":"HTTP/2",
            "code":200,
            "reason":""
          },
          "date":"Sat, 10 Jun 2023 10:31:15 GMT",
          "content-type":"image/png",
          "content-length":"104952",
          "last-modified":"Sat, 10 Jun 2023 10:31:15 GMT",
          "etag":""648450f3-199f8"",
          "access-control-allow-origin":"*",
          "access-control-allow-headers":"*",
          "expires":"Mon, 10 Jul 2023 10:31:14 GMT",
          "accept-ranges":"bytes"
        },
        "contentLength":104952,
        "timing":{
          "ns":0.007042,
          "connect":0.007407,
          "start":0.054624,
          "total":0.055398
        },
        "date":"2023-06-10T10:31:15.965Z"
      }
    ],
    "retryAfter":"2023-07-10T10:31:14.000Z"
  },
  {
    "filename":"/Products/Dell/XPS-13.png",
    "success":true,
    "attempted":false,
    "attempts":[
      {
        "url":"https://i.dell.com/sites/csimages/Video_Imagery/all/xps-13-9320-thumb.png",
        "initiator":{
          "type":"api",
          "remoteAddr":"81.109.65.144",
          "userAgent":"PostmanRuntime/7.32.2",
          "date":"2023-06-09T20:16:14.292Z"
        },
        "statusCode":200,
        "headers":{
          "result":{
            "version":"HTTP/2",
            "code":200,
            "reason":""
          },
          "date":"Fri, 09 Jun 2023 20:16:14 GMT",
          "content-type":"image/png",
          "content-length":"2018456",
          "last-modified":"Fri, 09 Jun 2023 20:16:14 GMT",
          "cache-control":"max-age=2591999",
          "etag":""6483888e-31476"",
          "access-control-allow-origin":"*",
          "access-control-allow-headers":"*",
          "expires":"Sun, 09 Jul 2023 20:16:13 GMT",
          "accept-ranges":"bytes"
        },
        "contentLength":2018456,
        "timing":{
          "ns":0.006758,
          "connect":0.007059,
          "start":0.05164,
          "total":0.053263
        },
        "date":"2023-06-09T20:16:14.836Z"
      }
    ],
    "retryAfter":"2023-07-09T20:16:13.000Z"
  },
  {
    "filename":"/Example/my-fetched-image.jpg",
    "success":false,
    "attempted":true,
    "attempts":[
      {
        "url":"https://example.com/somewhere/something.jpg",
        "initiator":{
          "type":"api",
          "remoteAddr":"81.109.65.144",
          "userAgent":"PostmanRuntime/7.32.2",
          "date":"2023-06-09T20:16:14.292Z"
        },
        "statusCode":404,
        "headers":{
          "result":{
            "version":"HTTP/2",
            "code":404,
            "reason":""
          },
          "date":"Fri, 09 Jun 2023 20:16:14 GMT",
          "content-type":"text/html; charset=utf-8",
          "content-length":"4140",
          "vary":"Accept-Encoding",
          "etag":"W/"102c-wKpRt+Cg9Cnw/NpTSkF5+w"",
          "access-control-allow-origin":"*",
          "access-control-allow-headers":"*",
        },
        "contentLength":4140,
        "timing":{
          "ns":0.007738,
          "connect":0.008061,
          "start":0.027649,
          "total":0.027799
        },
        "error":{
          "name":"Error",
          "message":"Request not successful: expected status code 200 but got 404"
        },
        "date":"2023-06-09T20:16:14.489Z"
      },
      {
        "url":"https://example.com/somewhere/something.jpg",
        "initiator":{
          "type":"api",
          "remoteAddr":"81.109.65.144",
          "userAgent":"PostmanRuntime/7.32.2",
          "date":"2023-06-10T10:31:12.585Z"
        },
        "statusCode":404,
        "headers":{
          "result":{
            "version":"HTTP/2",
            "code":404,
            "reason":""
          },
          "date":"Sat, 10 Jun 2023 10:31:15 GMT",
          "content-type":"text/html; charset=utf-8",
          "content-length":"4140",
          "vary":"Accept-Encoding",
          "access-control-allow-origin":"*",
          "access-control-allow-headers":"*",
        },
        "contentLength":4140,
        "timing":{
          "ns":0.006635,
          "connect":0.007103,
          "start":0.021783,
          "total":0.021908
        },
        "error":{
          "name":"Error",
          "message":"Request not successful: expected status code 200 but got 404"
        },
        "date":"2023-06-10T10:31:15.897Z"
      }
    ],
    "retryAfter":"2023-06-10T10:35:15.897Z"
  }
]

Response header

The header will return a 200 response to show that the request was received. It does not mean that the file(s) were successfully fetched - you should check the body for the status of each fetch.

These are the most useful headers:

X-Global-RateLimit-Reset - states when your hourly rate limit will refresh in 10-digit epoch time.
X-Global-RateLimit-Limit - your global total API requests per hour.
X-Global-RateLimit-Remaining - states how many global API requests are remaining - every type of API request counts 1 towards this limit.

Header example:

< HTTP/1.1 200
< date: Sat, 10 Jun 2023 10:31:16 GMT
< content-type: application/json; charset=utf-8
< transfer-encoding: chunked
< connection: keep-alive
< x-global-ratelimit-limit: 7000
< x-global-ratelimit-remaining: 1996
< x-global-ratelimit-reset: 1686396672
< access-control-allow-origin: *
< access-control-expose-headers: *
< cache-control: no-cache
< vary: accept-encoding
< content-encoding: gzip
< server: Sirv.API
< strict-transport-security: max-age=31536000

Example scripts

The REST API fetch documentation contains an example payload, JSON schema, response and scripts for 10 popular programming languages:

Service information

If you require a list of Sirv servers (IP addresses) used for fetching, send us a request via your account.

The HTTP Referrer header field used by the Sirv agent is:

Sirv Image Service

Create your Sirv account

If you don't have a Sirv account yet, create your account today.

Get expert help

For help with the image and file fetch API, please contact the Sirv customer success team.

Fetch files with the API

About the fetch API

Fetch a file

Multiple URLs in one request

HTTP authentication

Fetching limits

Hourly fetching limit

File size limit

Time limit

How to handle fetch errors

Wait before retrying

Failure notifications

Example response

Response body

Response header

Example scripts

Service information

Create your Sirv account

Get expert help

Was this article helpful? Click here to cancel reply.

Get help from a Sirv expert

Fetch files with the API

About the fetch API

Fetch a file

Multiple URLs in one request

HTTP authentication

Fetching limits

Hourly fetching limit

File size limit

Time limit

How to handle fetch errors

Wait before retrying

Failure notifications

Example response

Response body

Response header

Example scripts

Service information

Create your Sirv account

Get expert help

Was this article helpful? Click here to cancel reply.

Related articles

Get help from a Sirv expert