My first no-code experience

New project, new workflow

To kick start a new subtitling project we had to set up a new workflow in order to:

1. Receive videos to be captioned
2. Process the videos in an automatic transcription system
3. Create subtitles
4. Synchronize subtitles according to client specifications
5. And finally send an SRT file back to each video editor.

All within 48 hours at most.

In order to keep track of the various assignments as they came in, besides entering all the data into our management system, we also created a shared Google spreadsheet to mark:

• Video duration
• Delivery date
• The person in charge of subtitling.

As usual, for all communication within the team we used our internal chat based on Mattermost, an open source alternative to Slack.

At the same time, the client requested that anyone on his team (consisting of dozens of video editors) had to be able to upload videos to a server, with no need to log in.

Had it not been for this requirement, the most immediate solution for us would have been Google Drive. However, Google allows files to be uploaded only to those having Google accounts.

To overcome this problem, we came up with the idea of creating a “file request” on our Synology file server. Video editors can visit a dedicated page where they can upload videos directly onto the server via a simple form.

From there, folders containing videos are synchronized with Google Drive through CloudStation, a tool provided by Synology’s DSM operating system.

In order to notify the project manager, we had our IT guy write a script that would compare the contents of the folders in question every 15 minutes and send an email with the path and name of the newly uploaded file.

Then the project manager had to import the video into our automatic transcription system, Sonix.ai, for one of the collaborators to review.

Sonix allows videos to be grouped hierarchically, so we have reproduced the same folder structure found on Google Drive: very simply, it consists of a main folder for each language and subfolders - one for each video editor - containing the videos.

Which automation tool to use?

From the very first moment it was clear to me that this workflow could be automated, both to save time and to avoid possible mistakes, as the project management involved the repetition of several mechanical actions such as copy-paste, where the risk of human error is always high.

In fact, for each new video received, we had to perform the following actions:

1. Import the video from Google Drive to Sonix
2. Fill in the corresponding language Google Sheet with the video data (date, sender, file name, link on Google Drive, and duration)
3. Notify team members on the appropriate Mattermost channel (one channel for each language).

It seemed like the perfect project to try out some no-code platforms.

I ran initial automation tests with Zapier, which allows you to directly connect Sonix to Google Drive thanks to a ready-made recipe; for each new file uploaded to Drive, Zapier transmits the information to Sonix, which retrieves the file and performs automatic transcription.

This integration is very convenient and works perfectly, but it has one major limitation: Google Drive API does not allow you to monitor a folder along with all of its subfolders. Monitoring is therefore not recursive. This implies that you have to create a “zap” for each folder, instead of being able to use a single zap on a main folder monitoring all subfolders.

This limitation has three disadvantages:

1. The project involves dozens of video makers, so there are dozens of subfolders; creating a zap for each folder involves manual work, which defeats the automation effort we were making.
2. Zapier’s fee plans vary according to the number of zaps needed. As the folders increased, we would soon exceed the limits of the cheapest plans.
3. It was also unclear whether it would be possible to easily integrate with Zapier the other two systems involved in the workflow, Google Sheets and Mattermost, two fundamental ingredients for our management.

Discarding Zapier, we evaluated some alternatives:

• n8n
• Integromat (which has since changed its name to Make and
• Pabbly Connect.

n8n is very powerful, but it is also the most complicated of the three. After installing it on our server, we began to study it, quickly realizing that it was too complex for our purposes.

Integromat seemed to be a good candidate, but we discarded it because we received a rather cold (and late) response from support when we set out our needs.

Pabbly support instead responded in detail, so I registered and began my experiments, following the directions I had received.

This question proved to be one of the most difficult problems to solve, because the data that can be retrieved and used for subsequent steps depend on the trigger event.

Integration development in Pabbly Connect

In Pabbly Connect the triggers available with Google Drive were reduced to only two possibilities: when a new file is uploaded (in any folder), or find the last added files with a recursive search at set time intervals, but only in the folders that contain the videos.

I had to immediately discard the option to monitor newly uploaded files among all Drive folders, i.e., the first item that appears in the “Recent” list, because Pabbly Connect checks for new files every 5 minutes (a setting that cannot be changed). If more than one file arrived within those 5 minutes, it would only detect the most recent one, ignoring others.

Recursive search, on the other hand, supposed asking Drive to list subfolders and listing the contents of each. After that, Pabbly had to compare the upload time with the current time and only continue if it found a file uploaded in the last 15 minutes.

By repeating the workflow every 15 minutes, the system would be able to detect files uploaded during that time interval.

This way, I was confident I would not lose any files, even if several came in within 15 minutes. However, I had to abandon this route as well because it had too great an impact on the number of tasks consumed, which in Pabbly are not unlimited. Even by reducing the frequency to every 30 minutes we would still risk exceeding the maximum number of tasks allowed in our monthly subscription.

Since recursive search was too resource-intensive, at this point I had to consider other solutions.

Our workflow had to start only when a file actually arrived, and it didn't make sense to monitor Drive folders around the clock for a dozen files arriving a day.

If I wanted to disregard Google Drive I had to use the arrival of the notification email addressed to the project manager as a trigger.

Here, however, the problem was we had to separate the text lines in the email and pull from each the data we needed:

• file name
• the name of the folder in which it was uploaded, which in turn is the email address of the subtitles’ recipient,
• and finally the language of the video, because we receive videos in Italian, Spanish, and Portuguese.

All this data can be derived using Pabbly Connect’s function that extracts text with a regular expression. In fact, I had to go from this:

< /volume1/video/Company_ES/enrique.blanco@company.com/@eaDir/video1.mp4/SYNOVIDEO_VIDEO_SCREENSHOT.jpg
< /volume1/video/Company_ES/enrique.blanco@company.com/video1.mp4
< /volume1/video/Company_IT/mario.rossi@company.com/@eaDir/video2.mp4/SYNOVIDEO_VIDEO_SCREENSHOT.jpg
< /volume1/video/Company_IT/mario.rossi@company.com/video2.mp4
< /volume1/video/Company_PT/joao.verdes@company.com/@eaDir/video3.mp4/SYNOVIDEO_VIDEO_SCREENSHOT.jpg
< /volume1/video/Company_PT/joao.verdes@company.com/video3.mp4

to this:

ES
enrique.blanco@company.com
video1.mp4

IT
mario.rossi@company.com
video2.mp4

PT
joao.verdes@company.com
video3.mp4

At first the complexity of the regex had even put me off using the email trigger. However, when the path with Google Drive proved to be a dead end, I rolled up my sleeves and with some patience and persistence managed to create a regular expression that would work in all cases.

The rest of the workflow was already fairly rolled out, and thanks to Pabbly Connect’s reiteration option (Iterator), the steps can be repeated at each input.

Here is a wrap-up of the resulting workflow:

1. The system receives the email to a special address created by Pabbly Connect and the workflow starts.
2. A regex splits the body of the email into as many lines as there are videos uploaded, and the system repeats the following steps (3 to 10) for each video.
3. The system extracts the data I am interested in with another regex: language of the video, sender email address, and file name.
4. It sends an automatic confirmation email to the person who uploaded the video.
5. It searches for the file on Google Drive using the file name as the search key.
6. Drive returns the video data I am interested in, particularly the upload date.
7. The system asks Sonix, via its API, for a list of existing folders:
   • if the sender folder does not yet exist, create it with the same name as the video folder and proceed with the next step.
   • otherwise, if the sender’s folder already exists, proceed with the next step.
8. It uploads the video to the appropriate folder on Sonix.
9. It adds a row with video data to the corresponding shared Google Sheet.
10. It sends a message on Mattermost via a bot to alert team members that there is a new video available.

Conclusion

Initially I had to further adjust the workflow to accommodate some exceptions, which I won’t list here. From beginning to end this no-code-developed automation took in total a couple of weeks of work, which is no small feat. In perspective, however, if the transcription project lasts for at least a few months, I am sure it will save us a lot of time and many potential errors.

Indeed, that of human error is a problem not to be underestimated, because a mistake in typing data or a clumsy copy-paste can mean big problems.

After these initial no-code experiments, the conclusion is that these tools are very powerful and flexible, but they are still not the “set it and forget it” solution they promise.

The two bottlenecks I identified were the trigger event and the regular expression, which in general is by no means simple and in our case was further complicated because it had to consider various combinations of characters in the email address, name, and file path.

Moreover, every system has some design limitations and is also not free of internal bugs, which for Pabbly Connect were several. Support, however, has always been present and responsive on the official forum and has resolved many of the reported problems fairly quickly.

The fundamental problem is that the various links should work on paper, but it is extremely likely that your specific use case has never been tested because it is totally new. This means that the developers were never faced with a certain bug because no one had ever used the system in that way.

For example in the experiment with recursive search on Google Drive, I noticed and reported to support that when listing the contents of folders, Drive also detected files in the trash as if they were still in the folder.

The potential however is there, because it is obviously a way to bring the common user closer to application development, but the upshot of my experience is that no-code is not yet an affordable solution.