FTP is a surprisingly resilient and widely used protocol, even in a world of SFTP and cloud storage, because it’s so simple and universally understood by machines.
Let’s say you need to pull daily sales reports from a partner’s FTP server every morning at 3 AM and put them into your company’s data warehouse. You’re not going to log in manually each day.
Here’s what a typical curl command might look like to download a file:
curl -u username:password -O ftp://ftp.example.com/reports/sales_report_$(date +%Y%m%d).csv
This command, when executed, connects to ftp.example.com using the provided username and password. The -O flag tells curl to save the downloaded file with the same name as it has on the remote server. The $(date +%Y%m%d) part dynamically generates the current date in YYYYMMDD format, so it fetches the report for the specific day.
Now, you need to automate this. For this, we use cron, the standard Linux/Unix job scheduler. You edit your crontab with crontab -e and add a line like this:
0 3 * * * /usr/bin/curl -u myuser:mypassword -O ftp://ftp.example.com/reports/sales_report_$(date +\%Y\%m\%d).csv -o /home/user/downloads/sales_report_$(date +\%Y\%m\%d).csv
Notice the escaped % signs for the date format within the crontab. This cron job will run at 3:00 AM every day. The -o flag is added to explicitly specify the local save path, ensuring the file lands where you want it.
The core problem cron and curl solve here is timing and execution. Instead of a human watching a clock and running commands, cron acts as the attentive system, and curl acts as the obedient client, performing the requested file transfer at the appointed time. This removes human error and ensures consistent data flow.
You can also use lftp, another powerful command-line FTP client, which offers more advanced features like mirroring and parallel transfers. A lftp script might look like this:
open -u username,password ftp.example.com
mirror /reports/ /home/user/downloads/
bye
And the corresponding cron entry:
0 3 * * * /usr/bin/lftp -c "open -u myuser,mypassword ftp.example.com; mirror /reports/ /home/user/downloads/; bye"
This lftp command opens a connection, then uses mirror to recursively download all files from the /reports/ directory on the server to your local /home/user/downloads/ directory. This is particularly useful if there are multiple files or subdirectories to transfer.
The mental model is simple: a scheduler (cron) triggers a client (curl or lftp) at a set interval. The client connects to a server and performs file operations (download, upload, mirror) based on pre-defined instructions.
What most people don’t realize is how robust FTP can be when combined with scripting. You can add error checking, retry logic, and even notifications within your script. For instance, you could wrap your curl command in a shell script that checks the exit code of curl and sends an email if it fails.
#!/bin/bash
REMOTE_HOST="ftp.example.com"
REMOTE_USER="myuser"
REMOTE_PASS="mypassword"
REMOTE_DIR="/reports"
LOCAL_DIR="/home/user/downloads"
DATE_FORMAT=$(date +%Y%m%d)
FILENAME="sales_report_${DATE_FORMAT}.csv"
REMOTE_FILE="${REMOTE_DIR}/${FILENAME}"
LOCAL_FILE="${LOCAL_DIR}/${FILENAME}"
echo "Attempting to download ${REMOTE_FILE} to ${LOCAL_FILE}..."
curl -u ${REMOTE_USER}:${REMOTE_PASS} -o ${LOCAL_FILE} ftp://${REMOTE_HOST}/${REMOTE_FILE}
if [ $? -eq 0 ]; then
echo "Download successful!"
else
echo "Download failed!"
# Add email notification here if desired
fi
Then, your cron job would simply be:
0 3 * * * /home/user/scripts/download_report.sh
This script adds a layer of intelligence, making the automated transfer more reliable.
The next step in automating file transfers often involves dealing with more complex scenarios, like securely transferring files over SSH using scp or sftp, or managing large-scale data synchronization with tools like rsync.