AWS Transfer Family lets you run FTP, FTPS, and SFTP servers without managing any infrastructure. It’s a fully managed service that integrates with S3 and EFS, so your files land directly where you need them.
Let’s see it in action. Imagine we need to receive daily sales reports from a partner via SFTP.
# On your local machine, connect to the Transfer Family SFTP endpoint
sftp -i ~/.ssh/your_private_key.pem user@sftp.transfer.us-east-1.amazonaws.com
# Once connected, upload a file
put sales_report_2023-10-27.csv
That file sales_report_2023-10-27.csv will now appear in your S3 bucket.
Transfer Family handles the complexities of running these protocols. You don’t need EC2 instances, load balancers, or storage provisioning. When you create a Transfer Family server, you choose the protocol (SFTP, FTPS, or FTP), select an identity provider (AWS Directory Service, custom Lambda, or service-managed users), and point it to an S3 bucket or EFS file system.
The key components are:
- Server Endpoint: The public IP address or DNS name clients connect to.
- Protocols: SFTP (SSH File Transfer Protocol), FTPS (FTP over TLS/SSL), and FTP (unencrypted).
- Identity Provider: How users authenticate.
- Service-Managed: You create and manage users and their SSH keys directly within Transfer Family.
- AWS Directory Service: Integrates with AWS Managed Microsoft AD or AD Connector for domain-based authentication.
- Custom Identity Provider (Lambda): A Lambda function handles authentication, allowing for custom logic, integration with existing IdPs (like Okta), or dynamic user provisioning.
- Storage: Your files are stored in an S3 bucket or EFS file system. Transfer Family acts as a gateway to this storage.
The magic is how it maps the file system operations your SFTP client performs to S3 or EFS operations. For SFTP, it’s using the SSH protocol and translating put commands into S3 PutObject API calls. For FTPS, it’s using FTP commands wrapped in TLS, again mapping to S3/EFS.
When setting up a server, you configure things like:
- Endpoint Type: Publicly accessible or VPC hosted (for access within your private network).
- Logging: CloudWatch Logs for auditing and debugging.
- Custom Hostname: CNAME record to a more user-friendly domain.
- Security: IAM roles that grant Transfer Family permissions to access your S3 bucket/EFS.
The most surprising thing about Transfer Family is that it doesn’t actually store the files itself. It’s a highly available, scalable gateway that translates standard file transfer protocols into S3 or EFS API calls. Your data never resides within Transfer Family; it’s always in your chosen S3 bucket or EFS file system. This means you benefit from S3’s durability and EFS’s elasticity directly, without any data movement or complex synchronization.
When you use a service-managed user, you associate an SSH public key with them. Transfer Family then uses this key to authenticate the user when they connect. The user’s home directory is mapped to a specific prefix within your S3 bucket. For example, if a user’s home directory is set to /uploads/partnerA and they upload a file named report.csv, it will be stored in s3://your-bucket-name/uploads/partnerA/report.csv.
The next hurdle is often handling large files or numerous small files efficiently, which might lead you to explore Transfer Family’s asynchronous processing capabilities with SQS.