An upload callback is a process in which the client includes callback parameters in its upload request. Once the upload is complete, the TOS server sends a synchronous POST request with the parameters to the specified third-party application server. The client only receives the results once the application server acknowledges the request and sends a response. Due to the additional steps of sending the callback request and waiting for a response, the upload process takes longer than a standard upload.

How it works

APIs that support upload callbacks

PutObject

PostObject

CompleteMultipartUpload

Workflow for upload callbacks

Step 1: Construct callback parameters

To trigger a callback, you must specify callback parameters and variables in the upload request.

Callback parameters

Example

Callback parameters are provided as a Base64-encoded JSON string.

// Example 1: When request body type is "application/json"
{
"callbackUrl" : "http://domainname.com/callback",
"callbackHost" : "alternative-domainname.com",
"callbackBody" : "{\"bucket\" : ${bucket}, \"object\" : ${object}, \"key1\" : ${x:key1}, \"key2\" : ${x:key2}}",
"callbackBodyType" : "application/json"
}

// Example 2: When request body type is "application/x-www-form-urlencoded"
{
"callbackUrl" : "http://domain-name/callback",
"callbackHost" : "alternative-domain-name",
"callbackBody" : "bucket=${bucket}&object=${object}&key1=${x:key1}&key2=${x:key2}",
"callbackBodyType" : "application/x-www-form-urlencoded"
}

Parameters

Parameter	Description	Constraints and limitations
callbackUrl	Indicates the URL of the third‑party application server to which TOS will send an HTTP or HTTPS request after a successful upload.	callbackUrl can be left empty. This indicates that no upload callback parameter has been set for this request. The URL must use either the HTTP or HTTPS protocol. If no protocol is specified, the default will be https://. The path and query parameters of the URL must be URL-encoded. You can add up to five URLs to this field. Separate them with semicolons (;). The TOS server will sequentially try each URL until it receives a successful response. Callback variables are not supported. callbackUrl cannot contain special addresses such as 127.0.0.1, 0.0.0.0, 0:0:0:0:0:0:0:1, 0:0:0:0:0:0:0:0, or localhost.
callbackHost	Represents the HOST header of the callback request. The value of this field overrides the host derived from callbackUrl.	callbackHost can be left empty. If provided, it must comply with the rules for domain names or IP addresses. Special addresses such as 127.0.0.1, 0.0.0.0, 0:0:0:0:0:0:0:1, 0:0:0:0:0:0:0:0, and localhost are not permitted. Callback variables are not supported.
callbackBody	Represents the body sent with the callback request. Its format must match the content type specified by callbackBodyType.	callbackBody cannot be empty and must be in a valid format. The TOS server strictly validates this field. Callback variables are supported.
callbackBodyType	Indicates the content type of the callback request. This field must be used with callbackBody.	callbackBodyType can be left empty and will default to "application/x-www-form-urlencoded" if left empty. Only "application/x-www-form-urlencoded" and "application/json" are supported. Other values are considered invalid. Callback variables are not supported.

Callback variables

The callbackBody parameter supports two types of variables:

System variables, which can be entered directly in the callbackBody field.

Custom variables (callback-var), which must be included separately by the client with the request.

The TOS server parses JSON-formatted custom variables and merges them with system variables to create a set of available key-value pairs. These pairs are then used to replace placeholders in callbackBody in the form of ${variable_name} or ${x:variable_name}.

System variables

The following table lists the system variables that can be configured within callbackBody.

System variables must be used with the ${} operator in callbackBody. For example, ${bucket} retrieves the bucket to which the object was uploaded.

When callbackBodyType is "application/json", the TOS server populates the JSON field values according to the data types of the variables.

For example, if "bucket" is the string "bucket-test" and "size" is the integer 1024, then the expression {"bucket": ${bucket}, "size" : ${size}} will be resolved to {"bucket" : "bucket-test", "size" : 1024}.

Variable	Type	Description
bucket	string	The name of the bucket to which the object was uploaded.
key	string	The name of the uploaded object.
object	string	The name of the uploaded object.
size	int64	The size of the uploaded object.
etag	string	The ETag of the uploaded object.
crc64ecma	string	The CRC-64 value of the uploaded object.
versionId	string	The version ID of the uploaded object, available only when versioning is enabled for the bucket.
filename	string	The name of the file that was uploaded to TOS via the PostObject API
fname	string	The name of the file that was uploaded to TOS via the PostObject API
mimeType	string	The content type that the client specified when uploading an object. If not specified, the default is "binary/octet-stream".
requestId	string	The ID of this request.

Custom variables

During an upload callback, the client can include custom key-value pairs in JSON format within the request. The TOS server parses these custom variables and uses them to replace placeholders in callbackBody. Example:

Custom variables must start with x:. Otherwise, they are considered invalid.

Custom variables must be used with the ${} operator in callbackBody. For example, ${x:key1} retrieves the variable x:key1.

When callbackBodyType is "application/json", the TOS server populates custom variables the same way as with system variables.

When callbackBodyType is "application/x-www-form-urlencoded", system variables (key, object, fname, and filename) in callbackBody are URL-encoded during population.

Step 2: Construct a callback request

How to construct

The APIs that support upload callbacks are PutObject, PostObject, and CompleteMultipartUpload. PutObject and CompleteMultipartUpload carry callback parameters and custom variables via URL or header, while PostObject carries them through form fields. The following explains how the client constructs a callback request.

Supported APIs	How callback parameters are passed	Where to include	Signing method	Constraints and limitations
PutObject and CompleteMultipartUpload	Through request headers	Request headers: x-tos-callback x-tos-callback-var	Includes signatures in CanonicalQueryString and SignedHeaders according to the signature mechanism.	Both x-tos-callback and x-tos-callback-var must be JSON-formatted and Base64-encoded. Callback parameters and custom variables cannot appear in both headers and query strings simultaneously. However, they can be combined by including one in the headers and the other in the query string.
PutObject and CompleteMultipartUpload		Through the URL	Query strings: x-tos-callback x-tos-callback-var		Includes signatures in CanonicalQueryString according to the signature mechanism.
PostObject	Using POST form fields	x-tos-callback x-tos-callback-var or custom variables	Includes x-tos-callback, x-tos-callback-var, and custom variables as conditions in the policy for form-based uploads, according to the signature mechanism for form uploads.	Both x-tos-callback and x-tos-callback-var must be JSON-formatted and Base64-encoded. x-tos-callback-var takes precedence over custom variables. If x-tos-callback-var is not provided, each custom variable is passed as an individual form field.

Remarks

Passing parameters in the URL: Callback parameters are carried via x-tos-callback, and custom variables are carried via x-tos-callback-var.

Passing parameters in request headers: Callback parameters are carried via x-tos-callback, and custom variables are carried via x-tos-callback-var.

Passing parameters via POST form fields: Callback parameters are carried via x-tos-callbackand, and custom variables are carried via x-tos-callback-var. If x-tos-callback-var is not provided, each custom variable is passed as an individual form field. Example:

--9431149156168
Content-Disposition: form-data; name="x:key1"
value1
--9431149156168
Content-Disposition: form-data; name="x:key2"
123

Walkthrough

This section demonstrates how to create a callback request to a third-party application server. The response will be {'Status':'OK'}.

Construct callback parameters.

// Callback parameters:
{
"callbackUrl" : "http://domainname.com/callback",
"callbackHost" : "alternative-domainname.com",
"callbackBody" : "{\"bucket\" : ${bucket}, \"object\" : ${object}, \"key1\" : ${x:key1}, \"key2\" : ${x:key2}}",
"callbackBodyType" : "application/json"
}

// Callback variables:
{
"x:key1" : "value1",
"x:key2" : 123
}

// Result returned by the third-party application server:
{"Status":"OK"}

Encode the callback parameters and variables using Base64.

// x-tos-callback: Base64-encoded callback parameters
Cgl7CgkJImNhbGxiYWNrVXJsIiA6ICJodHRwOi8vZG9tYWlubmFtZS5jb20vY2FsbGJhY2siLCAKCQkiY2FsbGJhY2tIb3N0IiA6ICJhbHRlcm5hdGl2ZS1kb21haW5uYW1lLmNvbSIsICAgICAgICAgICAgICAgCgkJImNhbGxiYWNrQm9keSIgOiAie1wiYnVja2V0XCIgOiAke2J1Y2tldH0sIFwib2JqZWN0XCIgOiAke29iamVjdH0sIFwia2V5MVwiIDogJHt4OmtleTF9LCBcImtleTJcIiA6ICR7eDprZXkyfX0iLCAKCQkiY2FsbGJhY2tCb2R5VHlwZSIgOiAiYXBwbGljYXRpb24vanNvbiIgICAgICAgICAgICAgICAgCgl9
// x-tos-callback-var: Base64-encoded custom variables
ewogICAgIng6a2V5MSIgOiAidmFsdWUxIiwKICAgICJ4OmtleTIiIDogMTIzLAp9

Construct a callback request.

PutObject
CompleteMultipartUpload
PostObject

Create a PutObject request that includes callback parameters and custom variables in headers. Example:

PUT /objectName HTTP/1.1
Host: bucketname.tos-cn-beijing.volces.com
Date: Fri, 30 Jul 2021 08:05:36 +0000
x-tos-callback-var: ewogICAgIng6a2V5MSIgOiAidmFsdWUxIiwKICAgICJ4OmtleTIiIDogMTIzLAp9
x-tos-callback: Cgl7CgkJImNhbGxiYWNrVXJsIiA6ICJodHRwOi8vZG9tYWlubmFtZS5jb20vY2FsbGJhY2siLCAKCQkiY2FsbGJhY2tIb3N0IiA6ICJhbHRlcm5hdGl2ZS1kb21haW5uYW1lLmNvbSIsICAgICAgICAgICAgICAgCgkJImNhbGxiYWNrQm9keSIgOiAie1wiYnVja2V0XCIgOiAke2J1Y2tldH0sIFwib2JqZWN0XCIgOiAke29iamVjdH0sIFwia2V5MVwiIDogJHt4OmtleTF9LCBcImtleTJcIiA6ICR7eDprZXkyfX0iLCAKCQkiY2FsbGJhY2tCb2R5VHlwZSIgOiAiYXBwbGljYXRpb24vanNvbiIgICAgICAgICAgICAgICAgCgl9
Authorization: authorization_string
Content-Type: text/plain
Content-Length: 0

Step 3: Send the callback request

Once the upload is complete, the TOS server sends a POST request to the third-party application server based on the callback parameters. To process the request, the application server must meet the following requirements:

Category	Requirement
Interface protocol	HTTP or HTTPS. When using HTTPS, the third-party application must have a valid certificate.
Request method	POST
Request body format	application/x-www-form-urlencoded or application/json
Response status code	200
Request body format	application/json
Response body size	No larger than 3 MB.

Based on the callback parameters and variables in the upload request, TOS sends a POST request to the third-party application server. Example:

(Optional) Step 4: Verify the signature of the callback request

After receiving the callback request, the third-party application server can verify its origin using the signature to confirm that the request comes from TOS.

TOS server generates a signature

Third-party application server verifies the signature

The TOS server automatically generates a signature using the RSA asymmetric encryption method and sends it to the third-party application server. The following describes how the TOS server generates signatures.

Create a string-to-sign (StringToSign).

url_decode(path) + '?' + url_decode(sorted(query)) + '\\n' + body

Calculate the hash of the string-to-sign using the MD5 algorithm.

md5(StringToSign)

Generate the signature by encrypting the hash value with a private key.

rsa_sign(private_key, url_decode(path) + '?' + url_decode(sorted(query)) + '\\n' + body, md5)

Encode the signature using Base64. Then, add the encoded signature to the Authorization header of the callback request and send the request to the third-party application server.

Authorization = base64_encode(rsa_sign(private_key, url_decode(path) + '?' + url_decode(sorted(query)) + '\\n' + body, md5))

The following explains how third-party application servers verify signatures.

The x-tos-pub-key-url callback request header contains the Base64-encoded URL of the signature public key. First, decode x-tos-pub-key-url using Base64. Then, download the public key locally.

tip

To verify that the public key was issued by TOS, check the download address. The download address must start with http://tos-public.volccdn.com/ or https://tos-public.volccdn.com/.

The public key remains unchanged for an extended period. For future use, we recommend storing it in a local cache.

Decode the callback request header Authorization using Base64 to obtain the signature.

Create a string-to-sign (StringToSign), using the request path, request parameters, and request body obtained from the request.

url_decode(path) + '?' + url_decode(sorted(query)) + '\\n' + body

warning

When creating a string-to-sign, sort the query parameters lexicographically.

When creating a string-to-sign, the query parameter must be formatted as “key=” if it contains only a key.

Calculate the hash of the string-to-sign using the MD5 algorithm.

md5(StringToSign)

Verify the signature using the public key, the signature itself, and the hash of the string-to-sign.

result = rsa_verify(public_key, md5(StringToSign), signature)

The following example in Go demonstrates how third-party application servers verify signatures.

package callback
import (
"bytes"
"crypto"
"crypto/md5"
"crypto/rsa"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"errors"
log "github.com/sirupsen/logrus"
"io/ioutil"
"net/http"
"sort"
jsoniter "github.com/json-iterator/go"
)
const XTosPublicKeyUrl = "x-tos-pub-key-url"
const Authorization = "Authorization"
type CallbackAPI struct {
}
type ErrResp struct {
Err string `json:"Err"`
RetMsg string `json:"RetMsg"`
}
func (api *CallbackAPI) CallbackHandler(w http.ResponseWriter, r *http.Request) {
sign, err := getSignature(r)
if err != nil {
log.Warn("authorization: %s is not base64 encoding", r.Header.Get(Authorization))
JsonResponse(w, 401, ErrResp{Err: "[Callback] invalid Authorization, err:" + err.Error()})
return
}
publicKey, err := getPublicKey(r)
if err != nil {
JsonResponse(w, 401, ErrResp{Err: "[Callback] invalid public key, err:" + err.Error()})
return
}
signMd5, body, err := getContentAndSignMD5(r)
if err != nil {
JsonResponse(w, 401, ErrResp{Err: "[Callback] invalid public key, err:" + err.Error()})
return
}
if err = verifySignature(publicKey, signMd5, sign); err != nil {
JsonResponse(w, 401, ErrResp{Err: "[Callback] verify signature failed, err:" + err.Error()})
return
}
log.Info("[Callback] handle callback request")
bodyType := r.Header.Get("Content-Type")
switch bodyType {
case "application/json":
if !jsoniter.Valid(body) {
JsonResponse(w, 500, ErrResp{Err: "[Callback] the body is not a valid json"})
return
}
JsonResponse(w, 200, body)
default: // application/x-www-form-urlencoded
JsonResponse(w, 200, struct {
EncodedBody string `json:"encoded_body"`
}{
EncodedBody: base64.StdEncoding.EncodeToString(body),
})
}
}
func verifySignature(publicKey []byte, signMd5 []byte, sign []byte) error {
pubBlock, _ := pem.Decode(publicKey)
if pubBlock == nil {
log.Warn("Failed to parse PEM block containing the public key")
return errors.New("invalid public key pem")
}
pubInterface, err := x509.ParsePKIXPublicKey(pubBlock.Bytes)
if err != nil || pubInterface == nil {
log.Warn("x509.ParsePKIXPublicKey(publicKey) failed, err:%v", err)
return errors.New("invalid public key pem")
}
pub := pubInterface.(*rsa.PublicKey)
err = rsa.VerifyPKCS1v15(pub, crypto.MD5, signMd5, sign)
if err != nil {
log.Warn("verify signature failed, err:%v", err)
return err
}
return nil
}
func getContentAndSignMD5(r *http.Request) ([]byte, []byte, error) {
bodyContent, err := ioutil.ReadAll(r.Body)
_ = r.Body.Close()
if err != nil {
log.Warn("read request body failed, err:%v", err)
return nil, nil, err
}
stringToSign := calcStringToSign(r, bodyContent)
log.Info("stringToSign: %s", stringToSign)
authMd5 := md5.New()
authMd5.Write(stringToSign)
signMd5 := authMd5.Sum(nil)
return signMd5, bodyContent, nil
}
func getPublicKey(r *http.Request) ([]byte, error) {
var bytePublicKey []byte
publicKeyURLBase64 := r.Header.Get(XTosPublicKeyUrl)
if publicKeyURLBase64 == "" {
log.Warn("GetPublicKey from Request header failed : No x-tos-pub-key-url field. ")
return bytePublicKey, errors.New("no x-tos-pub-key-url field in Request header ")
}
publicKeyURL, _ := base64.StdEncoding.DecodeString(publicKeyURLBase64)
resp, err := http.Get(string(publicKeyURL)) // ignore_security_alert
if err != nil {
log.Warn("Get PublicKey Content from URL failed : %s", err.Error())
return bytePublicKey, err
}
bytePublicKey, err = ioutil.ReadAll(resp.Body)
defer resp.Body.Close()
if err != nil {
log.Warn("Read PublicKey Content from URL failed : %s", err.Error())
return bytePublicKey, err
}
if resp.StatusCode >= http.StatusBadRequest {
log.Warn("get public key failed, res:%s", string(bytePublicKey))
return nil, errors.New("get public key failed")
}
return bytePublicKey, nil
}
func getSignature(r *http.Request) ([]byte, error) {
authorization := r.Header.Get(Authorization)
if authorization == "" {
return nil, nil
}
return base64.StdEncoding.DecodeString(authorization)
}
// url_decode(path) + '?' + url_decode(sorted(query)) + '\\n' + body
func calcStringToSign(req *http.Request, body []byte) []byte {
buf := bytes.NewBuffer(nil)
buf.WriteString(req.URL.Path)
query := req.URL.Query()
if len(query) > 0 {
buf.WriteByte('?')
keys := make([]string, 0, len(query))
for key := range query {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
values := query[key]
for _, value := range values {
buf.WriteString(key)
buf.WriteByte('=')
buf.WriteString(value)
buf.WriteByte('&')
}
}
buf.Truncate(buf.Len() - 1)
}
buf.WriteByte('\\n')
buf.Write(body)
return buf.Bytes()
}
func JsonResponse(w http.ResponseWriter, status int, entity interface{}) {
// Set the content type before calling WriteHeader. Otherwise, it will not take effect.
if entity != nil {
w.Header().Set("Content-Type", "application/json")
}
w.WriteHeader(status)
if entity != nil {
if body, ok := entity.([]byte); ok {
_, _ = w.Write(body)
return
}
body, _ := json.Marshal(entity)
_, _ = w.Write(body)
}
}

Step 5: Return the callback result

The third-party application server sends a response to TOS. Example:

Step 6: Return the upload result

TOS forwards the content returned by the third-party application server to the client.

PostObject request response

CompleteMultipartUpload request response

PostObject request response

Error codes

HTTP status code	Error codes	Error message
203	CallbackFailed	The upload was successful, but the callback request to the third-party application server failed. Possible reasons for this failure include a DNS error, a request timeout, or invalid response content.
400	InvalidCallbackArgument	The fields in the callback parameters (callback) are invalid. The format for custom variables (callback-var) is invalid. The headers or query string contain duplicate custom parameters or variables.
403	AccessDenied	Verification of the callback parameters and custom variables against the PostObject request policy document has failed.

Torch Object Storage