Media File Migration: rsync, S3, and Automatic URL Updates

When migrating a site with a 15 GB media library and a database of 60,000 records, manual copying inevitably leads to losses. Ten years of experience show that automation with rsync and S3 cuts transfer time by three times and eliminates errors. Without automation, you risk losing data and traffic. We have developed a proven process that includes inventory, delta synchronization, cloud upload, and automatic link updates. This guarantees file integrity and zero downtime. Contact us for a project consultation to assess the scope of work.

What Problems Do We Solve?

Broken links are a common issue. After migration, up to 20% of links remain broken if all occurrences of the old URL are not updated. We replace them automatically, updating content and metadata. Uncontrolled traffic growth is another problem: without image optimization, download sizes remain large. We convert JPEG and PNG to WebP, reducing size by 60% without quality loss. This reduces server load and lowers hosting costs by 30%. Traffic savings reach 60% through CDN caching. Duplicates and garbage: inventory reveals files on disk missing from the database, and vice versa. We delete excess, saving up to 30% of disk space.

How We Do It: A Case Study from Our Practice

For our client with a WordPress site (8 GB media, 40,000 records), we performed migration to a new hosting. First step—inventory:

# Summary by size and type find /var/www/uploads -type f | awk -F. '{print $NF}' | sort | uniq -c | sort -rn du -sh /var/www/uploads 

Next—synchronization via rsync. We ran it multiple times, using --checksum for accuracy. Synchronization took 2 hours instead of 12 hours with manual copying.

# Initial synchronization (can be run multiple times) rsync -avz --progress --checksum user@old-server:/var/www/uploads/ /var/www/new-site/uploads/ # Delta synchronization before final switch rsync -avz --delete user@old-server:/var/www/uploads/ /var/www/new-site/uploads/ 

After that, we uploaded files to S3 for CDN delivery, reducing TTFB by 40%:

aws s3 sync /var/www/uploads/ s3://company-media-bucket/uploads/ --storage-class STANDARD --exclude "*.tmp" --acl public-read 

Link updates were done with a Python script that processed 12,000 records in 3 minutes:

import mysql.connector def update_media_urls_in_content(db_conn, old_base, new_base): cursor = db_conn.cursor() tables_columns = [('posts', 'content'), ('posts', 'excerpt'), ('pages', 'body'), ('users', 'avatar_url')] for table, column in tables_columns: cursor.execute(f"SELECT id, {column} FROM {table} WHERE {column} LIKE %s", (f'%{old_base}%',)) for row_id, content in cursor.fetchall(): if content: new_content = content.replace(old_base, new_base) cursor.execute(f"UPDATE {table} SET {column} = %s WHERE id = %s", (new_content, row_id)) db_conn.commit() print(f"Updated {cursor.rowcount} rows") update_media_urls_in_content(db_conn, 'https://old-site.com/wp-content/uploads', 'https://cdn.new-site.com/uploads') 

For WordPress, we additionally executed SQL queries and configured 301 redirects in nginx:

location ~* ^/wp-content/uploads/(.*)$ { return 301 /uploads/$1; } 

Comparison of Methods and Formats

WebP saves up to 70% space.

rsync is 5x faster than FTP and ensures integrity.

Work Process

1. Analytics—inventory of files and database relationships.
2. Design—choose copy strategy, redirects, and optimization.
3. Implementation—sync, cloud upload, URL updates.
4. Testing—verify integrity, check all links.
5. Deploy—configure 301 redirects, switch DNS.

How to Minimize Downtime During Migration?

Using rsync with delta-sync allows migration without taking the site offline. The final switchover takes minutes. Setting up 301 redirects ensures users do not see broken links. Thus, downtime is virtually eliminated. Contact us for precise planning.

What's Included in the Work?

• Detailed media library audit: report by file types, duplicates, unused resources.
• Writing and executing sync scripts (rsync, S3 CLI).
• Automatic URL updates in all posts and meta fields (WordPress, custom tables).
• Configuration of 301 redirects for old paths.
• Image optimization: conversion to WebP, lossless compression.
• CDN integration (CloudFront or similar) with caching.
• Documentation of the new media storage architecture.
• Technical support for one week after migration.

File Integrity Guarantee

After copying, we run verification: compare md5 hashes of original and copied files. If mismatch, the file is copied again. We also check that all files from the database exist on disk. This eliminates broken files and guarantees full correspondence. Additionally, rsync can be run with the --checksum flag for ongoing verification.

Why Use CDN for Media?

CDN reduces server load and speeds up page loading. We upload files to S3 and connect CloudFront. This reduces TTFB by an average of 40%. Moreover, traffic is saved by 60% through edge caching. When choosing a CDN, consider audience geography and egress costs. For Russian users, Cloudflare or Selectel CDN are optimal. For international, CloudFront or Fastly.

Common Mistakes in Media File Migration

Copying without checksum verification risks broken files. Ignoring files with no database links leads to disk clutter. Lack of redirects loses traffic from old URLs. Skipping meta field updates (e.g., thumbnails in WordPress) breaks site functionality. Omitting image optimization increases traffic costs. Our engineers with ten years of migration experience have successfully transferred over 500 projects. Contact us for a project consultation to assess the scope of work.

Timelines and Cost

Timelines depend on data volume. For a site up to 10 GB—2–3 business days. For larger projects—5–7 days. Cost is calculated individually after analysis and includes support for one week after migration.