Simulating Post Object Form Upload to OSS in Java

Introduction

Alibaba Cloud Object Storage Service (OSS) is an easy-to-use service that enables you to store, backup and archive large amounts of data on the cloud. It acts as an encrypted central repository from where one can securely access files from around the globe. Further, it guarantees up to 99.9% availability and is a perfect fit for global teams and international project management.

Alibaba Cloud Object Storage Service (OSS) stores objects securely within resources called 'buckets'. OSS provides you full access to the buckets and allows you to view logs and objects within each bucket. You can read, write, delete, and store unlimited objects in your bucket. The high performance of OSS supports multiple reads/writes simultaneously. Data transfers to your buckets are via SSL and are encrypted.

Post Object uses an HTML form to upload a file to a specified bucket as a replacement for Put. This makes it possible to upload files to the bucket through a browser. The demand for implementing Post Object in Java derives from a short description from various support personnel. As per them, when a user that needs this feature meets various challenging problems during his attempts to implement the feature by following the official documentation. This happens because no official code reference exists.

The Procedure

Let us look at the steps that a user needs to follow.

● The official website first provides the HTTP request syntax. It is the HTTP request header and the form of the multipart/form-data-encoded form field in the message body for meeting the parameters.
● Next, it introduces the "required" form fields such as "file" and "key" one-by-one in a form field table. The user requires the form fields such as "OSSAccessKeyId", "policy" and "Signature" when any one of them appears. Also, he/she needs the REST request header, x-oss -meta- * user meta and other optional form fields.
● Post this, it introduces several special usage and precautions for some form fields. It also shares a stack of information on Post Policy and Signature features and their usage.
● The documentation gives some clarity, however, it adds to the challenge of investigating problems owing to various hard-to-comprehend concepts. Further, owing to the absence of highlights on error-prone points in its implementation. A user could come across two major challenges, further involving two aspects:

○ Unfamiliarity with MIME-type encoding such as multipart/form-data;
○ Unfamiliarity with OSS implementation rules for parsing Post Object requests.

Next, we move on to explaining the two aspects mentioned above.

For detailed multipart/form-data introductions, you can see RFC 2388. There are several points to note here. Let us discuss them one-by-one.

1.The first point says that a "multipart/form-data" request contains a series of fields. Each field has a "form-data"-type content-disposition header. This header also contains the parameter "name" to describe the form field content. Therefore, every field will have a format similar to the example shown in the documentation, also mentioned below.

Content-Disposition: form-data; name="your_key"

Note: The ":" and ";" are both followed by a space.

2.The second point to note is that in case there is a requirement to upload a user file in the form, you may need a file name or other file attributes in the content-disposition header, such as the parameter "filename". Additionally, for any MIME-type values in the form field, an optional Content-Type attribute also exists to identify the file content type. Therefore, the documentation lists an example of the "file" form field as follows:

Content-Disposition: form-data; name="file"; filename="MyFilename.jpg"
Content-Type: image/jpeg

Note: The ";" before "filename" still has a trailing space. Similarly, the ":" after "Content-Type" also has a trailing space.

3.The third point would be separating the data with a boundary. You should try to use a complicated boundary to distinguish it from the main content. You can achieve this in a manner similar to the content in the HTTP header, as depicted in the documentation.

Content-Type: multipart/form-data; boundary=9431149156168

4.The fourth point to note says that the structure of each form field is fixed. The specification is that each form field begins with "--"boundary+ followed by a carriage return (/r/n). Then comes the description of the form field (see point 1), and /r/n in order. If the content you want to transfer is a file, the file name information will also include the file content type following the carriage return (/r/n) (see point 2). Further, there is another carriage return (/r/n) to start the actual content which you should end with /r/n.

5.You should also note that the last form field ends with "--"+boundary+"--", indicating the end of the request body.

6.Additionally, you also need the /r/n mark to distinguish the HTTP request header and the body information (at the junction of the header and the first form field). This is essentially an extra blank line, such as in the documentation and is as depicted below.

Content-Type: multipart/form-data; boundary=9431149156168

--9431149156168
Content-Disposition: form-data; name="key"

Discussed above is the general description of the request syntax provided in the OSS official documentation and related analysis in comparison with the RFC 2388 standard.

Now, we will delve into the explanation of a small part of the process for the OSS system to parse the Post Object request and its related notes.

The general procedure for the OSS to parse a POST request is as shown in the figure below for your reference.

Summarizing the request processing flow into three core steps namely:

1. Parse the boundary in the HTTP request header to distinguish field boundaries;
2. Parse the content of various fields until the flow reaches the 'file' form field;
3. Parse the 'file' form field.

Hence, the documentation emphasizes the placement of the 'file' form field in the "last field". Otherwise, form fields after "file" may not take effect. If you place the required form field "key" after "file", the result will certainly be InvalidArgument.

Next, we will briefly describe some work flows as illustrated in the figure:

1) Check POLICY, OSSACCESSKEYID, SIGNATURE existence:
This check is essential. In case one of the three fields of POLICY, OSSACCESSKEYID or SIGNATURE appears, the other two fields become necessary.
2) Authorization:
Verify the validity of the Post request based on the POLICY, OSSACCESSKEYID and SIGNATURE information.
3) Policy rule check:
Check whether settings in various form fields of the request comply with the policy configuration.
4) Check length Legality:
This aims to check the length of optional fields as there is a limit on the total length of the Post request body.
5) ParseContentType in ParseFile:
Parse the ContentType field in the "file" field. You do not require this field.

Now, we can conclude with the Java code (Maven project) implementing Post Object upload in the OSS for reference and use of those who are familiar with OSS.

import javax.activation.MimetypesFileTypeMap;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.HashMap;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;

/**
 * Created by yushuting on 16/4/17.
 */
public class OssPostObject {

    private String postFileName = "your_file";  
Make sure that the file exists at the path indicated in the run code. 
    private String ossEndpoint = "your_endpoint";  
For example: http://oss-cn-shanghai.aliyuncs.com
    private String ossAccessId = "your_accessid";  This is your access AK
    private String ossAccessKey = "your_accesskey";  This is your access AK
    private String objectName = "your_object_name";  This is the object name after you upload the file
    private String bucket = "your_bucket";  Make sure that the bucket you created previously has been created. 

    private void PostObject() throws Exception {

        String filepath=postFileName;
        String urlStr = ossEndpoint.replace("http://", "http://"+bucket+".");  This is the URL for the submitted form is the bucket domain name

        LinkedHashMap<String, String> textMap = new LinkedHashMap<String, String>();
        // key
        String objectName = this.objectName;
        textMap.put("key", objectName);
        // Content-Disposition
        textMap.put("Content-Disposition", "attachment;filename="+filepath);
        // OSSAccessKeyId
        textMap.put("OSSAccessKeyId", ossAccessId);
        // policy
        String policy = "{\"expiration\": \"2120-01-01T12:00:00.000Z\",\"conditions\": [[\"content-length-range\", 0, 104857600]]}";
        String encodePolicy = java.util.Base64.getEncoder().encodeToString(policy.getBytes());
        textMap.put("policy", encodePolicy);
        // Signature
        String signaturecom = com.aliyun.oss.common.auth.ServiceSignature.create().computeSignature(ossAccessKey, encodePolicy);
        textMap.put("Signature", signaturecom);

        Map<String, String> fileMap = new HashMap<String, String>();
        fileMap.put("file", filepath);

        String ret = formUpload(urlStr, textMap, fileMap);
        System.out.println("[" + bucket + "] post_object:" + objectName);
        System.out.println("post reponse:" + ret);
    }

    private static String formUpload(String urlStr, Map<String, String> textMap, Map<String, String> fileMap) throws Exception {
        String res = "";
        HttpURLConnection conn = null;
        String BOUNDARY = "9431149156168";
        try {
            URL url = new URL(urlStr);
            conn = (HttpURLConnection) url.openConnection();
            conn.setConnectTimeout(5000);
            conn.setReadTimeout(30000);
            conn.setDoOutput(true);
            conn.setDoInput(true);
            conn.setRequestMethod("POST");
            conn.setRequestProperty("User-Agent",
                    "Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.6)");
            conn.setRequestProperty("Content-Type",
                    "multipart/form-data; boundary=" + BOUNDARY);

            OutputStream out = new DataOutputStream(conn.getOutputStream());
            // text
            if (textMap != null) {
                StringBuffer strBuf = new StringBuffer();
                Iterator iter = textMap.entrySet().iterator();
                int i = 0;
                while (iter.hasNext()) {
                    Map.Entry entry = (Map.Entry) iter.next();
                    String inputName = (String) entry.getKey();
                    String inputValue = (String) entry.getValue();
                    if (inputValue == null) {
                        continue;
                    }
                    if (i == 0) {
                        strBuf.append("--").append(BOUNDARY).append(
                                "\r\n");
                        strBuf.append("Content-Disposition: form-data; name=\""
                                + inputName + "\"\r\n\r\n");
                        strBuf.append(inputValue);
                    } else {
                        strBuf.append("\r\n").append("--").append(BOUNDARY).append(
                                "\r\n");
                        strBuf.append("Content-Disposition: form-data; name=\""
                                + inputName + "\"\r\n\r\n");

                        strBuf.append(inputValue);
                    }

                    i++;
                }
                out.write(strBuf.toString().getBytes());
            }

            // file
            if (fileMap != null) {
                Iterator iter = fileMap.entrySet().iterator();
                while (iter.hasNext()) {
                    Map.Entry entry = (Map.Entry) iter.next();
                    String inputName = (String) entry.getKey();
                    String inputValue = (String) entry.getValue();
                    if (inputValue == null) {
                        continue;
                    }
                    File file = new File(inputValue);
                    String filename = file.getName();
                    String contentType = new MimetypesFileTypeMap().getContentType(file);
                    if (contentType == null || contentType.equals("")) {
                        contentType = "application/octet-stream";
                    }

                    StringBuffer strBuf = new StringBuffer();
                    strBuf.append("\r\n").append("--").append(BOUNDARY).append(
                            "\r\n");
                    strBuf.append("Content-Disposition: form-data; name=\""
                            + inputName + "\"; filename=\"" + filename
                            + "\"\r\n");
                    strBuf.append("Content-Type: " + contentType + "\r\n\r\n");

                    out.write(strBuf.toString().getBytes());

                    DataInputStream in = new DataInputStream(new FileInputStream(file));
                    int bytes = 0;
                    byte[] bufferOut = new byte[1024];
                    while ((bytes = in.read(bufferOut)) != -1) {
                        out.write(bufferOut, 0, bytes);
                    }
                    in.close();
                }
                StringBuffer strBuf = new StringBuffer();
                out.write(strBuf.toString().getBytes());
            }

            byte[] endData = ("\r\n--" + BOUNDARY + "--\r\n").getBytes();
            out.write(endData);
            out.flush();
            out.close();

            // Read the returned data
            StringBuffer strBuf = new StringBuffer();
            BufferedReader reader = new BufferedReader(new InputStreamReader(
                    conn.getInputStream()));
            String line = null;
            while ((line = reader.readLine()) != null) {
                strBuf.append(line).append("\n");
            }
            res = strBuf.toString();
            reader.close();
            reader = null;
        } catch (Exception e) {
            System.err.println("Error in sending a POST request:  " + urlStr);
            throw e;
        } finally {
            if (conn != null) {
                conn.disconnect();
                conn = null;
            }
        }
        return res;
    }

    public static void main(String[] args) throws Exception {
        OssPostObject ossPostObject = new OssPostObject();
        ossPostObject.PostObject();
    }

}

Please note that you must add the following in pom.xml:

<dependency>
    <groupId>com.aliyun.oss</groupId>
    <artifactId>aliyun-sdk-oss</artifactId>
    <version>2.2.1</version>
</dependency>

Conclusion

Post Object makes it possible to upload files to a bucket based on the browser. Encoding the message body of Post Object utilizes multipart/form-data. In the Post Object operation, the program transfers the parameters as the form fields in the message body. Post Object uses AccessKeySecret to compute the signature for the policy. Although the post form field is optional for uploading public-read-write buckets, we recommend you use this field to limit POST requests.