Background

I was working on a resource management system, where I needed to upload a large file into the server. The files that I needed to upload were zip files and were in GBs. While uploading and testing those files I got some error with connectivity and the upload failed. The files were already big and to upload them I needed to start them again. So, there began the search for any option to have resumable upload like we do with downloads. I needed a solution to have a resumable upload that could be paused and even if we were interrupted by a poor network connection from a client we could continue where we left off.

The Overlooked Problem: Rethinking File Uploads

In the realm of data transfer, the conventional methods of file uploads often pose a significant challenge that tends to be disregarded—the vulnerability of traditional upload mechanisms. The standard approaches that we use to upload files lack the resilience for handling large file size uploads, handling interruptions, or unreliable and poor network conditions. Unfortunately, this oversight has led us to ignore user experience and increase the frustration.

The Amazing Solution

Enter TUS: An Oasis in the Desert of File Upload Frustration. In the vast expanse of digital challenges, where the struggle for seamless file uploads seemed endless, a Messiah emerged—TUS, an open protocol for resumable file uploads. Finally, amidst the chaos, I found precisely what I needed. Sorry, for being dramatic but the possibility that we could have a resumable file upload was fascinating to me. I got exactly what I needed and through this blog, I am going to share how to use it in Django so, that if you need to upload a large file or need a resumable upload in your project you can do it easily.

About TUS

TUS.IO is an open-source protocol for resumable file uploads built on top of HTTP. It offers a standardized method for managing file transfers, enabling users to upload files of any size with consistency and effectiveness. TUS.IO is a preferred option for developers and companies globally because it removes the annoyances related to interrupted uploads because of network problems, file size restrictions, or browser crashes. TUS.IO enables smooth resumable uploads, guaranteeing that the client can continue an upload even in the event of an interruption. It eases the burden on server resources and permits parallel uploading, improving overall performance, by permitting chunked uploads.

How it works

TUS.IO has a simple workflow. When a file upload starts using TUS, the client sends a HTTP POST request to the server data with some file metadata like filename and type. The server then responds with a generated upload URL where the remaining requests will be made.

After the POST process, the client sends chunks of the file to the server as PATCH requests parallelly if enabled making faster uploads than the traditional way. The requests also send the file’s current state to store the received data and keep track of upload progress. During transfer, if there is any interruption or the connection breaks TUS.IO enables the client to resume where it left off by continuing to send the PATCH requests.

Although TUS.IO supports parallel uploads because we are using Django which has a synchronous nature I don’t think we can take advantage of it in this project.

(Illustration by @transloadit)

Setup

Create a Virtual Environment

Creating a virtual environment is not a mandatory process but is best practice because it provides isolation from other projects, and helps in dependency management, version management, and cleaner development. Open a terminal go to the directory where you want to set up the Django project and run the command below.

#Create a virtual environment

python -m venv env

#I am on Windows so to activate the environment

env\\Scripts\\activate

#If you are on macOS/Linux

source env/bin/activate

You can replace “env” with the environment name of your choice
Install Django project

After activating the environment we create a Django project with the following command where tus_demo is my project name:

#Django 3.2 is [LTS] version so using 3.2 for this project

pip install django==3.2.*

django-admin startproject tus_demo

You can replace “tus_demo” with your project name.
Move the Virtual Environment inside the project

I like the virtual environment inside the Django project but you can completely ignore this step if you don’t want to.

#Deactivate the environment

deactivate

#Move the environment inside the project

# If you are on windows

move env tus_demo

# I you are on macOS/Linux

mv env tus_demo

After this, you can activate the environment from the command above.
Run the Django project

Running the Django project we just created is easy we just need to run this command:

python manage.py runserver

You can open localhost:8000 to get this page which confirms that our project is running successfully.
Install necessary libraries

We only need django-tus as the library for this project.

pip install django-tus

Server Setup

As we talked about above TUS requires a client and a server to upload a file. We set up the server with the help of django-tus library that we installed above which already has the TUS server ready we just need to configure it in our project.

Firstly, We add django_tus in installed apps in settings.py which we can find inside the main folder which will have name same as our project name.

INSTALLED_APPS = [

…

‘django_tus’,

…

]

Add the following URLs in the urls.py file which we can find in the main folder.

#Import the TusUpload view from the django_tus library that we just added

from django_tus.views import TusUpload

…

urlpatterns = [

…

# The first URL lets us upload the whole file without using chunk upload

path(‘upload/’, TusUpload.as_view(), name=’tus_upload’),

# The second URL lets us upload files by breaking them into chunks

path(‘upload/<uuid:resource_id>’, TusUpload.as_view(), name=’tus_upload_chunks’),

…

]

We need to configure some additional settings in settings.py for setting up TUS server.

#We need to import os if not imported in settings.py

import os

…

#additional settings

TUS_UPLOAD_DIR = os.path.join(BASE_DIR, ‘tus_upload’)

TUS_DESTINATION_DIR = os.path.join(BASE_DIR, ‘media’, ‘uploads’)

TUS_FILE_NAME_FORMAT = ‘increment’ # Other options are: ‘random-suffix’, ‘random’, ‘keep’

TUS_EXISTING_FILE = ‘error’ # Other options are: ‘overwrite’, ‘error’, ‘rename’

Django has a setting for maximal memory size for uploaded files. This setting needs to be higher than the chunk size of the TUS client:

DATA_UPLOAD_MAX_MEMORY_SIZE = 5242880 #size is in byte

We have successfully set up the configuration for the TUS server. You can check out the repo if you need more configuration.

TUS Client Setup

We have now set up the TUS server we need to set a view in the Django so we can render a page with the upload page. Django is based on Model View Template (MVT) architecture. We need a template which is a html file and a view to render the html file. For templates, we create a templates folder in our root directory. We need to configure settings to make Django detect templates if not inside an app. Add “templates” inside the list of “DIRS” in TEMPLATES in the settings.py file.

TEMPLATES = [

{

…

‘DIRS’: [‘templates’],

…

]

Create a HTML file named upload.html inside the templates folder that has file input that posts the file. We will be using [tus-client-js](<https://github.com/tus/tus-js-client>) a pure JavaScript client for TUS. We import the js file in the HTML and configure it to work with our server.

<!DOCTYPE html>

<head>

<title>Resumable File Upload with TUS</title>

</head>

<body>

<h2>Normal Upload</h2>

</div>

<div id=”sizeInfo”>Total Size: <span id=”totalSize”>0 MB</span>

| Uploaded Size: <span id=”uploadedSize”>0

MB</span></div>

<button id=”upload-btn”>Upload</button>

</div>

</div>

<!– import tus js client –>

</body>

</html>

We then create a static folder to add our static files CSS and JS. Create a style.css file inside the static folder to make our UI beautiful.

body {

font-family: ‘Arial’, sans-serif;

background-color: #f4f4f4;

margin: 0;

display: flex;

align-items: center;

justify-content: center;

height: 100vh;

}

h2{

text-align: center;

}

.card {

background: rgba(255, 255, 255, 0.1);

border-radius: 10px;

padding: 20px;

box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1), 0 1px 3px rgba(0, 0, 0, 0.1);

backdrop-filter: blur(10px);

max-width: 400px;

width: 100%;

text-align: center;

}

#fileInput {

margin-bottom: 10px;

width: 100%;

}

#progressBar {

width: 0%;

height: 20px;

margin-bottom: 10px;

border-radius: 20px;

background-color: #2980b9;

}

#progressContainer{

border-radius: 20px;

background-color: lightgray;

}

#sizeInfo {

text-align: center;

margin-bottom: 10px;

font-size: 12px;

}

button {

padding: 10px;

background-color: #3498db;

color: #fff;

border: none;

cursor: pointer;

transition: background-color 0.3s ease;

width: 100%;

}

button:hover {

background-color: #2980b9;

}

#uploads {

margin-top: 20px;

}

.upload-item {

border: 1px solid #ccc;

padding: 10px;

margin-bottom: 10px;

border-radius: 5px;

background-color: #fff;

}

.upload-item a {

color: #3498db;

text-decoration: none;

font-weight: bold;

}

Create a JS file script.js that will handle the TUS upload using tus-client-js library.

let upload = null;

let uploadIsRunning = false;

const fileInput = document.getElementById(“fileInput”);

const uploadButton = document.getElementById(“upload-btn”);

const uploadedSizeDiv = document.getElementById(“uploadedSize”);

const totalSizeDiv = document.getElementById(“totalSize”);

const progressBar = document.getElementById(“progressBar”);

const uploadList = document.getElementById(“upload-list”);

// function to start upload file using tus

function startUploadFile() {

const file = fileInput.files[0];

if (file) {

uploadButton.textContent = “Pause upload”;

const options = {

// Endpoint is the upload creation URL from your tus server

endpoint: “/tus/upload/”,

chunkSize: 5242880, //chunk size to breakdown file into

parallelUploads: 10, //no of parallel request

// Retry delays will enable tus-js-client to automatically retry on errors

retryDelays: [0, 3000, 5000, 10000, 20000],

// Attach additional meta data about the file for the server

metadata: {

filename: file.name,

filetype: file.type,

// Callback for errors which cannot be fixed using retries

onError(error) {

if (error.originalRequest) {

if (

window.confirm(`Failed because: ${error}\\nDo you want to retry?`)

) {

upload.start();

uploadIsRunning = true;

return;

}

} else {

window.alert(`Failed because: ${error}`);

}

reset();

// Callback for reporting upload progress

onProgress: function (bytesUploaded, bytesTotal) {

const percentage = ((bytesUploaded / bytesTotal) * 100).toFixed(2);

// Calculate sizes in MB

const totalSizeMB = (bytesTotal / (1024 * 1024)).toFixed(2);

const uploadedSizeMB = (bytesUploaded / (1024 * 1024)).toFixed(2);

progressBar.style.width = `${percentage}%`;

// Update size information

uploadedSizeDiv.textContent = uploadedSizeMB + ” MB”;

totalSizeDiv.textContent = totalSizeMB + ” MB”;

// Callback for once the upload is completed

onSuccess: function () {

console.log(upload);

addUploadItem(upload.file.name, upload.file.size);

reset();

};

// Create a new tus upload

upload = new tus.Upload(file, options);

uploadIsRunning = true;

// Check if there are any previous uploads to continue.

upload.findPreviousUploads().then((previousUploads) => {

askToResumeUpload(previousUploads, upload);

upload.start();

uploadIsRunning = true;

});

} else {

alert(“Please select a file to upload”);

}

//function to reset upload

function reset() {

uploadButton.textContent = “Upload”;

upload = null;

uploadIsRunning = false;

}

function askToResumeUpload(previousUploads, currentUpload) {

if (previousUploads.length === 0) return;

let text = “You tried to upload this file previously at these times:\\n\\n”;

previousUploads.forEach((previousUpload, index) => {

text += `[${index}] ${previousUpload.creationTime}\\n`;

});

text +=

“\\nEnter the corresponding number to resume an upload or press Cancel to start a new upload”;

const answer = prompt(text);

const index = parseInt(answer, 10);

if (!Number.isNaN(index) && previousUploads[index]) {

currentUpload.resumeFromPreviousUpload(previousUploads[index]);

}

// Function to add an upload item to the “uploads” div

function addUploadItem(fileName, fileSize) {

const uploadsDiv = document.getElementById(“uploads”);

// Create a new div for the upload item

const uploadItemDiv = document.createElement(“div”);

uploadItemDiv.classList.add(“upload-item”);

// Create a link for the filename

const fileNameLink = document.createElement(“a”);

fileNameLink.href = “/media/uploads/” + fileName;

fileNameLink.textContent = fileName;

// Create a span for the file size

const fileSizeSpan = document.createElement(“span”);

fileSizeSpan.textContent = ` | Size: ${(fileSize / (1024 * 1024)).toFixed(

)} MB`;

// Append the link and size span to the upload item div

uploadItemDiv.appendChild(fileNameLink);

uploadItemDiv.appendChild(fileSizeSpan);

// Append the upload item div to the “uploads” div

uploadsDiv.appendChild(uploadItemDiv);

}

uploadButton.addEventListener(“click”, (e) => {

e.preventDefault();

if (upload) {

if (uploadIsRunning) {

upload.abort();

uploadButton.textContent = “Resume upload”;

uploadIsRunning = false;

} else {

upload.start();

uploadButton.textContent = “Pause upload”;

uploadIsRunning = true;

}

} else {

startUploadFile();

}

});

We need to do some static and media file setup for the Django project to access these files. Modify this config in settings.py :

STATIC_URL = ‘/static/’

STATICFILES_DIRS = [BASE_DIR / ‘static’]

MEDIA_URL = ‘/media/’

MEDIA_ROOT = BASE_DIR / ‘media’

We need to add media URLs in urls.py so we can access the uploaded file later on.

urlpatterns = [

…

]

urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Finally, we import the CSS and JS file that we just created in our upload.html file.

{% load static %}

<!DOCTYPE html>

<head>

<title>Resumable File Upload with TUS</title>

</head>

<body>

<h2>TUS Resumable Upload</h2>

</div>

<div id=”sizeInfo”>Total Size: <span id=”totalSize”>0 MB</span>

| Uploaded Size: <span id=”uploadedSize”>0 MB</span></div>

<button id=”upload-btn”>Upload</button>

</div>

</div>

<!– import tus js client –>

</body>

</html>

View Setup

We have completed the template part and now we create a view that processes our request and renders the upload.html file, so we can check out the page we just created above. For this project, we are directly creating a views.py file to write our view. We are only using the view to render out template which has tus-client setup.

from django.shortcuts import render

def home(request):

return render(request, ‘upload.html’)

After creating a view we need to assign a URL to the view in urls.py which is in main folder which we do as follows:

…

from tus_demo.views import home

…

urlpatterns = [

…

path(”, home),

…

]

After this, we can now refresh the localhost:8000 page where we can see this page.

Demo

We have finished all the setup to make TUS work with Django, upload a large file pause the upload resume, or disconnect and upload the file again. We don’t need to worry about our network connectivity issue and we don’t have to worry about pausing the upload and continuing.

tus.mp4

Wrapping up

I thought that the upload function was always limited within the traditional approach. I had an idea about chunk uploads that make the upload feature helpful but I had no idea that resumable upload was possible. I looked and found TUS.io so, I can say that don’t let your boundaries hold you back; break through and explore uncharted territories to uncover destinations that exceed your wildest dreams.

TUS.IO has revolutionized the way we handle upload and file transfer. Now uploading large files has become very easy and simple using tus. By using tus we can create an application that handles large files, which supports parallel and chunk upload seamlessly.

Get in touch:

Resumable Uploads in Django with TUS: Rethinking File Uploads

Background

The Overlooked Problem: Rethinking File Uploads

The Amazing Solution

About TUS

How it works

Setup

Server Setup

TUS Client Setup

View Setup

Demo

Wrapping up

Leave a Reply Cancel reply