Fetch files with the API
The fetch API makes it easy to copy files from an HTTP address into your Sirv account.
About the fetch API
The fetch API is one of 40 commands available in the Sirv REST API.
Useful links if you're thinking of using the fetch API:
- Fetch API documentation - along with all other API commands.
- Postman API collection - quick way to play with Sirv API commands.
- Upload file API - to upload a local file instead of a remote URL, use the upload API.
Fetch a file from a URL
For each fetch request, you should submit two parameters:
- url - the URL of the file to be fetched.
- filename - the folder and file name of the file to be saved in your Sirv account.
Example:
{
"url": "https://www.un.org/sites/un2.un.org/files/aan_660.png",
"filename": "/folder/path-to/my-new-image.jpg",
}
You can specify any folder path and filename - if the folders don't already exist in your Sirv account, they will be created.
Multiple URLs in one request
Each fetch request can contain up to 20 file URLs. This request will fetch 3 images:
[
{
"url": "https://www.w3schools.com/html/img_chania.jpg",
"filename": "/Example-folder/Italian-backstreet.jpg"
},
"url": "https://www.w3schools.com/html/img_girl.jpg",
"filename": "/Example-folder/jacket-hat.jpg"
},
"url": "https://www.w3schools.com/html/pic_trulli.jpg",
"filename": "/Example-folder/Greek-villas.jpg"
}
]
Got a CSV? If you have a CSV of URLs, an even easier solution is to use Sirv's bulk upload tool. Every file listed in your CSV will be automatically uploaded to your Sirv account. You can choose the folder structure, file names and you can enrich each file with meta data. See the CSV upload documentation.
HTTP authentication
If the files are protected by HTTP authentication, your request should provide the username and password for authentication:
{
"url": "https://demo.sirv.com/Example-Folder/lv-M46197_PM2_Front.jpg",
"filename": "/folder/path-to/my-new-image.jpg",
"auth": {
"username": "Example",
"password": "StrongPassword%50",
}
}
Example response
Each response will contain useful information in the header or body.
Response body
The body of the response will contain the most important information.
Of particular interest are:
- filename - the filename specified in your request.
- success - whether the file was successfully fetched (true/false).
- attempted - whether Sirv attempted to fetch the file (true/false). If a file with the same name already exists on Sirv, it will not be fetched.
- statusCode - if the fetch failed, the status code will help you decide what to do next.
- retryAfter - if the fetch failed, this states when a retry will be permitted.
Here is an example response body for 3 requests (1 succeeded, 1 was not attempted, 1 failed):
[
{
"filename":"/folder/path-to/my-new-image.png",
"success":true,
"attempted":true,
"attempts":[
{
"url":"https://images.ctfassets.net/8cd2csgvqd3m/4PZg3P4irAYjVwAPcFIw9v/5d52752cda2a7304b748f0e5a5ee9733/Beosound_A5_-_PDP_-_Image_1.png",
"initiator":{
"type":"api",
"remoteAddr":"81.166.37.144",
"userAgent":"PostmanRuntime/7.32.2",
"date":"2023-06-10T10:31:12.585Z"
},
"statusCode":200,
"headers":{
"result":{
"version":"HTTP/2",
"code":200,
"reason":""
},
"date":"Sat, 10 Jun 2023 10:31:15 GMT",
"content-type":"image/png",
"content-length":"104952",
"last-modified":"Sat, 10 Jun 2023 10:31:15 GMT",
"etag":""648450f3-199f8"",
"access-control-allow-origin":"*",
"access-control-allow-headers":"*",
"expires":"Mon, 10 Jul 2023 10:31:14 GMT",
"accept-ranges":"bytes"
},
"contentLength":104952,
"timing":{
"ns":0.007042,
"connect":0.007407,
"start":0.054624,
"total":0.055398
},
"date":"2023-06-10T10:31:15.965Z"
}
],
"retryAfter":"2023-07-10T10:31:14.000Z"
},
{
"filename":"/Products/Dell/XPS-13.png",
"success":true,
"attempted":false,
"attempts":[
{
"url":"https://i.dell.com/sites/csimages/Video_Imagery/all/xps-13-9320-thumb.png",
"initiator":{
"type":"api",
"remoteAddr":"81.109.65.144",
"userAgent":"PostmanRuntime/7.32.2",
"date":"2023-06-09T20:16:14.292Z"
},
"statusCode":200,
"headers":{
"result":{
"version":"HTTP/2",
"code":200,
"reason":""
},
"date":"Fri, 09 Jun 2023 20:16:14 GMT",
"content-type":"image/png",
"content-length":"2018456",
"last-modified":"Fri, 09 Jun 2023 20:16:14 GMT",
"cache-control":"max-age=2591999",
"etag":""6483888e-31476"",
"access-control-allow-origin":"*",
"access-control-allow-headers":"*",
"expires":"Sun, 09 Jul 2023 20:16:13 GMT",
"accept-ranges":"bytes"
},
"contentLength":2018456,
"timing":{
"ns":0.006758,
"connect":0.007059,
"start":0.05164,
"total":0.053263
},
"date":"2023-06-09T20:16:14.836Z"
}
],
"retryAfter":"2023-07-09T20:16:13.000Z"
},
{
"filename":"/Example/my-fetched-image.jpg",
"success":false,
"attempted":true,
"attempts":[
{
"url":"https://example.com/somewhere/something.jpg",
"initiator":{
"type":"api",
"remoteAddr":"81.109.65.144",
"userAgent":"PostmanRuntime/7.32.2",
"date":"2023-06-09T20:16:14.292Z"
},
"statusCode":404,
"headers":{
"result":{
"version":"HTTP/2",
"code":404,
"reason":""
},
"date":"Fri, 09 Jun 2023 20:16:14 GMT",
"content-type":"text/html; charset=utf-8",
"content-length":"4140",
"vary":"Accept-Encoding",
"etag":"W/"102c-wKpRt+Cg9Cnw/NpTSkF5+w"",
"access-control-allow-origin":"*",
"access-control-allow-headers":"*",
},
"contentLength":4140,
"timing":{
"ns":0.007738,
"connect":0.008061,
"start":0.027649,
"total":0.027799
},
"error":{
"name":"Error",
"message":"Request not successful: expected status code 200 but got 404"
},
"date":"2023-06-09T20:16:14.489Z"
},
{
"url":"https://example.com/somewhere/something.jpg",
"initiator":{
"type":"api",
"remoteAddr":"81.109.65.144",
"userAgent":"PostmanRuntime/7.32.2",
"date":"2023-06-10T10:31:12.585Z"
},
"statusCode":404,
"headers":{
"result":{
"version":"HTTP/2",
"code":404,
"reason":""
},
"date":"Sat, 10 Jun 2023 10:31:15 GMT",
"content-type":"text/html; charset=utf-8",
"content-length":"4140",
"vary":"Accept-Encoding",
"access-control-allow-origin":"*",
"access-control-allow-headers":"*",
},
"contentLength":4140,
"timing":{
"ns":0.006635,
"connect":0.007103,
"start":0.021783,
"total":0.021908
},
"error":{
"name":"Error",
"message":"Request not successful: expected status code 200 but got 404"
},
"date":"2023-06-10T10:31:15.897Z"
}
],
"retryAfter":"2023-06-10T10:35:15.897Z"
}
]
Response header
The header will return a 200 response to show that the request was received. It does not mean that the file(s) were successfully fetched - you should check the body for the status of each fetch.
These are the most useful headers:
- X-Global-RateLimit-Reset - states when your hourly rate limit will refresh in 10-digit epoch time.
- X-Global-RateLimit-Limit - your global total API requests per hour.
- X-Global-RateLimit-Remaining - states how many global API requests are remaining - every type of API request counts 1 towards this limit.
Header example:
< HTTP/1.1 200 < date: Sat, 10 Jun 2023 10:31:16 GMT < content-type: application/json; charset=utf-8 < transfer-encoding: chunked < connection: keep-alive < x-global-ratelimit-limit: 7000 < x-global-ratelimit-remaining: 1996 < x-global-ratelimit-reset: 1686396672 < access-control-allow-origin: * < access-control-expose-headers: * < cache-control: no-cache < vary: accept-encoding < content-encoding: gzip < server: Sirv.API < strict-transport-security: max-age=31536000
Example scripts
The REST API fetch documentation contains an example payload, JSON schema, response and scripts for 10 popular programming languages:
- cURL fetch example
- Python fetch example
- Node.js fetch example
- JavaScript fetch example
- Java fetch example | more Java examples
- PHP fetch example | more PHP examples
- Ruby fetch example
- Swift fetch example
- Go fetch example
- C# fetch example
Fetching limits
There are various limits to observe:
- Number of files per hour
- Size of files
- Time to fetch each file
Hourly fetching limit
You can fetch 1000-2000 files per hour, depending on your account type. This is burstable, allowing triple the limit for 24 hours, once per month.
| Enterprise | Business | Trial | Free | |
|---|---|---|---|---|
| Fetch requests / hour | 2000 | 1000 | 1000 | None |
To increase your hourly limit, please describe your requirements to the Sirv support team.
File size limit
The size of the remote file is limited:
| Enterprise | Business | Trial | Free | |
|---|---|---|---|---|
| File size limit | 10 MB | 5 MB | 3 MB | - |
| Large files | 64 MB 20 per hour |
32 MB 10 per hour |
32 MB 10 per hour |
- |
If you need to fetch larger files, please tell the Sirv support team how big your largest files are.
Time limit
Files should be fully returned within a time limit. An extra-long limit is provided for slow files.
| Enterprise | Business | Trial | Free | |
|---|---|---|---|---|
| Time limit | 10 seconds | 10 seconds | 10 seconds | - |
| Slow files | 30 seconds 20 per hour |
30 seconds 10 per hour |
30 seconds 10 per hour |
- |
How to handle fetch errors
A successful request will return a 200 response. If you receive an error response, follow the advice below:
- 401 - your API access token has probably reached its expiry. You can request a new bearer token and you can consider setting a longer bearer token expiry. The default expiry is 20 minutes - you can increase it to as much as 7 days.
- 403 - the remote server refused access to the file. There are many possible reasons, such as forbidden access, rejected IP address or invalid configuration. This probably won't be resolved by retrying - you should contact the person responsible for the file.
- 404 - the file is missing from the server. Check that the URL is correct. If so, contact the person responsible for the file.
- 429 - too many requests, when your hourly fetch limit is reached. Wait until the limit refreshes, then continue submitting requests. Your limit, current usage and reset time is shown in each response and also on your Usage page.
- 503/504 (header) - if Sirv's response header returns a 5xx error, Sirv's upload service may be overloaded. This rare occurence will resolve itself after some time. Please wait 30 minutes and try again.
- 503 (body) - if a URL in the body has a status code 503, it means an error on the remote server could not return the file. It might work if you try again later.
- 504 (body) - the remote server terminated the request as a timeout. Contact the person responsible for the server - ask them to whitelist Sirv's IP addresses or user agent.
- Timeout was reached - Sirv timed out the request because the server didn't return the file within the allowed time (10 or 30 seconds). Try again after 1 hour and the file might be successfully received.
- SSL peer certificate or SSH remote key was not OK - the domain of the remote server does not have a valid SSL certificate. This won't be resolved by retrying - you should contact the person responsible for the server.
- Number of redirects hit maximum amount - the remote server did not return the image - it redirected Sirv's request to another location. One redirect is permitted; a second redirect will fail. Do not retry this file as it will fail again.
Log the responses
We recommend saving a copy of each response as it will help you to troubleshoot any issues. Each response contains highly detailed information that will reveal exactly what happened and why an error occurred (if it occurred).
Wait before retrying
If a file fails to be fetched, you must wait before retrying it. This is a protection mechanism to protect against flooding the API with excessive requests. You can retry the file after 1 minute. If the second attempt also failes, you must wait 2 minutes, then 4 minutes, 9 minutes, 16, 25, 36, 49 minutes etc. If a retry is attempted too soon, it will be counted as an attempt and you'll have to wait longer, so please observe the retry time.
The retry time is shown as the retryAfter value in the body response, so you'll know when the next attempt will be permitted.
Failure notifications
To help make you aware of a failed fetch, a notification will be displayed on your Events page. Notifications contain the source URL, referrer, status code and error code. Example:
Service information
If you require a list of Sirv servers (IP addresses) used for fetching, send us a request via your account.
The HTTP Referrer header field used by the Sirv agent is:
Sirv Image Service
Create your Sirv account
If you don't have a Sirv account yet, create your account today.
Get expert help
For help with the image and file fetch API, please contact the Sirv customer success team.
