Fetch images automatically

On this page

Sirv can automatically fetch images from your server by HTTP, HTTPS or S3.

Whenever an image is requested from your Sirv account, or it is missing, it will attempt to fetch it from your server or S3 bucket. Once fetched, Sirv stores the original image in your Sirv account.

HTTP fetching

Sirv can fetch images from a particular folder on your server by HTTP / HTTPS.

Let's assume your source images are located in this folder of your website:

https://www.your-website.com/my-images/

Now let's say you want to serve this image from Sirv:

https://www.your-website.com/my-images/example-folder/an-image.jpg

You can request that file from your Sirv account, like so:

https://your-account.sirv.com/example-folder/an-image.jpg

If it doesn't already exist in your Sirv account, Sirv will fetch it from your server, then optimize and serve it. Fetching typically happens in less than 1 second, depending how quickly your server responds.

S3 fetching

Sirv can also fetch images from your Amazon S3 bucket or any non-Amazon S3 bucket.

Either the whole bucket or a specific folder can be fetched from.

You can use either virtual hosted-style or path-style URLs to access your S3 bucket:

Virtual hosted-style: https://<bucket>.<s3_server>/object
Path-style: https://<s3_server>/<bucket>/object

Enable fetching

To enable fetching, go to your Settings page in your Sirv account and click "Set up Auto-Fetch":

Either configure HTTP fetching by entering your remote server URL, folder (optional) and username/password (if your domain uses authentication):

Or configure S3 fetching by entering the Amazon AWS S3 or other S3 bucket. Also enter your access key ID, secret access key and folder path (optional):

File fetching is available to all paid Sirv plans. If you have a Trial or Free plan, switch to a paid plan to start using fetching immediately.

Image processing

All the usual image processing options can be applied to the image. For example, if this URL is requested:

https://your-account.sirv.com/example-folder/image.jpg?w=500&profile=xyz

First, Sirv will fetch this image from your server:

https://www.your-website.com/my-images/example-folder/image.jpg

... then serve the image with these options applied ?w=500&profile=xyz.

Updating a changed image

If an image changes on your server, the old image will continue to be served by Sirv until you delete it from your Sirv account. Once deleted, Sirv will refetch the image the next time it is requested.

Limits

Rate limit

Up to 1000 fetch requests can be sent per hour (2000 per hour for Enterprise accounts). If your account exceeds that, a 429 error (rate exceeded) will be served until the limit is reset (within 1 hour). If you need a larger limit, please request this from our customer support team.

File size limit

The maximum file size Sirv will fetch via S3 is 300MB.

The maximum file size Sirv will fetch via HTTP is:

	Typical file	Large file
Enterprise plan	7 MB, 2000 files per hour	32 MB, 10 files per hour
Business plan	5 MB, 1000 files per hour	32 MB, 10 files per hour
Trial plan	3 MB, 1000 files per hour	32 MB, 10 files per hour
Free plan	None	None

If you would like a higher limit, please submit a request explaining your requirements.

Response time limit

Sirv will wait up to 10 seconds to receive the file from the remote server. In case you have any very slow files, Sirv will wait up to 30 seconds (up to 10 times per hour).

If a request does not complete within the time limit, the connection will time-out and an error response will be returned. A notification will be displayed on the Events page of your account.

Time-outs are the most resource-hungry of fetching requests, so they will count as 5 requests towards your hourly limit.

Troubleshooting

Failed fetch attempts are shown in your Sirv account. Each error contains information to help you investigate whether there is a problem that needs your attention.

Possible causes of errors

Common causes of why files are not fetched:

403 error - your server refused Sirv access to the file. There are many possible reasons, such as forbidden access, rejected IP address or invalid configuration.
404 error - the file is missing from your server. Resolve this by checking the Referrer field, to see which web page is referencing the file, then either fix the broken URL or upload the missing image to your server.
504 error - your server terminated the request as a timeout, after some time. Check if your server is using rate limiting. If so, either increase or remove the rate limit for Sirv IP addresses.
Timeout was reached - Sirv timed out the request after 10 seconds and returned a 504 response because your server either didn't respond or didn't finish responding. Your server may have a firewall or proxy that is rate-limiting or blocking requests from Sirv IPs, particularly if many requests were sent in a short space of time. If your firewall/proxy is delaying/blocking the Sirv user agent Sirv Image Service, add a firewall rule to allow requests from Sirv's user agent.
SSL peer certificate or SSH remote key was not OK - the domain of the remote server does not have a valid SSL certificate. Configure an SSL certificate on your domain or fetch over HTTP instead of HTTPS. Sirv will return a 526 error response.
Number of redirects hit maximum amount - the remote domain did not return the image - it redirected Sirv's request to another location. One redirect is permitted; a second redirect will fail. Sirv will return a 500 error response.

Automatic retry

If a file fails to be fetched from your server, Sirv will allow that same URL to be retried after a 1 minute delay, followed by further attempts at increasing intervals. The second attempt is permitted after 2 minutes, then 4 minutes, 9 minutes, 16, 25, 36, 49 minutes etc. This technique is known as exponential backoff and it helps protect your account from hitting its hourly autofetch request limit due to a large volume of fetch requests for a file that cannot be fetched.

Look for the retryAfter value in the response, to know what time the next attempt will be permitted. The retryAfter time can also be seen by visiting the file URL directly - see below.

Additional troubleshooting information

To help you investigate deeper, each fetch attempt can be seen by directly visiting the file URL in your browser, for example:

https://test-account.sirv.com/no-image-here.jpg

You will see a list of fetching attempts with date, time, error name and message e.g.

The date and time is shown in ISO 8601 date format e.g. 2021-07-08T00:54:57.814Z, where Z denotes UTC time.

Failure notifications

Failed fetches are displayed as notifications on your Events page. Notifications contain the source URL, referrer, status code and error code:

Service information

If you require a list of Sirv servers (IP addresses) used for fetching, send us a request via your account.

The HTTP Referrer header field used by the Sirv agent is:

Sirv Image Service

FAQ

Can Sirv fetch all my images from S3?

There's no tool to copy all your S3 images to Sirv but we can probably write a script to do this for you. Or you can use DragonDisk to copy files from Amazon S3 to Sirv.

Create your Sirv account

If you don't have a Sirv account yet, create your account today.