Pre-Ingest File Storage API¶

POST files¶

Next, let’s actually upload files to temporary storage. Let’s begin by creating fake data with commands:

$ mkdir -p data/test1 data/test2
$ echo "This is test file 1" > data/test1/file_1.txt
$ echo "This is test file 2" > data/test1/file_2.txt
$ for i in {00..99}; do echo $i > data/test2/$i.txt; done

This creates directories data/test1/ and data/test2/, which contain 2 and 100 test files respectively. Let’s first look how individual files can be uploaded by uploading the files in directory data/test1/ and then how the whole directory data/test2/ can be uploaded.

Files can be uploaded to temporary storage by sending a POST request to /filestorage/api/v1/files/<project>/path/to/the/file, where <project> is the project identifier and /path/to/the/file is the path to the file on the server relative to your project directory. Files data/test1/file_?.txt can be uploaded with commands:

$ curl https://manage.fairdata.fi/filestorage/api/v1/files/<project>/data/test1/file_1.txt -X POST -T data/test1/file_1.txt -H "Authorization: Bearer <TOKEN>"
$ curl https://manage.fairdata.fi/filestorage/api/v1/files/<project>/data/test1/file_2.txt -X POST -T data/test1/file_2.txt -H "Authorization: Bearer <TOKEN>"

Here, flags -X and -T define request method and the actual data sent respectively. Without any flags provided, curl sends a GET request by default. The aforementioned commands should return responses like:

{
    "file_path": "/data/test1/file_1.txt",
    "status": "created"
}

Directory data/test2/ contains 100 files so uploading them individually doesn’t make sense. Writing a shell script that uploads each of them seperately would work, but even that would accumulate latency and make uploading multiple small files really slow. Thus, it’s best to make a ZIP archive, and upload it. The archive is extracted by the server automatically. Pack data/test2/ into a ZIP archive with command:

$ zip -r test2.zip data/test2/

Upload the ZIP archive to the server:

$ curl https://manage.fairdata.fi/filestorage/api/v1/archives/<project> -X POST -T test2.zip -H "Authorization: Bearer <TOKEN>"

This should return a response like:

{
    "file_path": "/",
    "message": "Uploading archive",
    "polling_url": "https://manage.fairdata.fi/filestorage/api/v1/tasks/5e7df16c2413de7e7b29f263",
    "status": "pending"
}

This response tells that the upload phase has been finished but the archive extraction continues on the server. The polling_url attribute of the response can be used to query the status of the extraction:

$ curl https://manage.fairdata.fi/filestorage/api/v1/tasks/5e7df16c2413de7e7b29f263 -H "Authorization: Bearer <TOKEN>"

As long as the extraction phase is ongoing on the server the command above will return a response like:

{
    "message": "Extracting archive",
    "status": "pending"
}

Finally when the extraction phase is ready on the server the response looks like:

{
    "message": "archive uploaded to /",
    "status": "done"
}

When task is finished, the archive is extracted to the root of the project. If extracting the archive would overwrite files, the task will fail, and in the server. returns 409: Conflict

GET files¶

Now that all the test files have been uploaded to the server let’s check some of them. A list of all the directories and filenames can be requested by sending a GET request to /filestorage/api/v1/files:

$ curl https://manage.fairdata.fi/filestorage/api/v1/files/<project>?all=true -H "Authorization: Bearer <TOKEN>"

GET more info about an individual file with e.g.

$ curl https://manage.fairdata.fi/filestorage/api/v1/files/<project>/data/test1/file_1.txt -H "Authorization: Bearer <TOKEN>"

This should return a response like

{
    "file_path": "/data/test1/file_1.txt",
    "md5": "7dbdc7a8126dcbb55dd383fab5c2d6f8",
    "identifier": "urn:uuid:f7b4913c-7172-44ea-913b-9fa3a426c93d",
    "timestamp": "2019-03-20T14:23:30+00:00"
}

File path is the path where the file was uploaded and is also used when creating the submission information package. MD5 is the checksum of the file and identifier is an UUID4 identifier that can be used for searching the file metadata from Metax. Checksums of the uploaded files should always be checked to make sure the files were not corrupted during the transfer. Checksums returned by the server should always match the local checksums, which can be calculated with command md5sum:

$ md5sum data/test1/file_?.txt

If information about the parent directory of the file is requested:

$ curl https://manage.fairdata.fi/filestorage/api/v1/files/<project>/data/test1/ -H "Authorization: Bearer <TOKEN>"

It can be noticed that also the directory has an identifier:

{
    "directories": [],
    "files": [
        "file_2.txt",
        "file_1.txt"
    ],
    "identifier": "452c6b0be99d3752bc177cd2e8efcb5a"
}

More info about the file metadata stored in Metax can be found on the Metax documentation.

DELETE files¶

Finally, let’s see how files can be deleted from temporary storage. This can be done by sending a DELETE request to the Pre-Ingest File Storage API. DELETE request removes the files from temporary storage and file metadata from Metax, if it is not associated with any dataset. Delete can be requested for the whole project, a single directory or a single file similar to the GET request shown earlier. Following command deletes all the files:

$ curl https://manage.fairdata.fi/filestorage/api/v1/files/<project> -X DELETE -H "Authorization: Bearer <TOKEN>"

The response looks like:

{
    "file_path": "/",
    "message": "Deleting files and metadata",
    "polling_url": "https://manage.fairdata.fi/filestorage/api/v1/tasks/5e7b60b52413de53896900fd",
    "status": "pending"
}

This tells that the server is still deleting metadata and files. The polling_url attribute of the response can be used to query the status of the file and metadata deletion:

$ curl -k https://manage.fairdata.fi/filestorage/api/v1/tasks/5e7b60b52413de53896900fd -H "Authorization: Bearer <TOKEN>"

As long as the file and metadata deletion is ongoing on the server the command above will return a response:

{
    "message": "Deleting files and metadata: /",
    "status": "pending"
}

Finally when the file and metadata deletion is ready on the server the response looks like:

{
    "message": "Deleted files and metadata: /",
    "status": "done"
}

Files can be deleted from temporary storage after the dataset has been accepted for digital preservation. All the files will automatically be cleaned after 30 days based on the timestamp returned by GET /filestorage/api/v1/files/<project>/path/to/the/file.

upload-rest-api-client¶

upload-rest-api-client is a simple python client for sending files to the Pre-Ingest File Storage. The client can be downloaded with command:

$ git clone https://github.com/Digital-Preservation-Finland/upload-rest-api-client.git