Backing up the content of cloud storage volumes

This documentation is for an older version of CCC. You can find the latest version here.
Last updated on April 3, 2024

 

There are several cloud storage solutions available that allow you to synchronize content that's stored locally on your Mac with storage hosted on the Internet. Naturally we want to be able to back up all of your data whether it's stored in the cloud or not. The manner in which cloud syncing solutions store data locally, however, can complicate how you go about backing up and restoring that data. There are two complicating factors that we will address in this article:

  • The actual location of your locally stored data may be hidden, making it difficult to find the files on your backup.
  • Some, or possibly even all, of your cloud-synced files may not be permanently stored on your Mac; content stored only in the cloud is not available for making a local backup.

Local storage of cloud content is kept in a hidden location

You're typically used to accessing your cloud-synced content in the Finder's sidebar. In some cases (e.g., Microsoft OneDrive), the cloud storage solution may place an alias in your home folder that conveniently points to the location of the local copy of your data. Typically, though, that content is not stored in an obvious location, rather it's stored in the hidden "Library" folder in your home folder. Knowing where that data "lives" is key to understanding how to access that content on your backups.

Finding your cloud-synced content on the backup

If you make an ordinary backup of your startup disk, all of your locally-stored cloud content is on the backup. That content is in a hidden location, though, so follow these steps to locate that content on your backup disk:

  1. Choose Computer from the Finder's Go menu
  2. Select your backup disk, then navigate to Users > (yourname)
  3. Press Command+Shift+Period to toggle the Finder's display of hidden items
  4. iCloud: Navigate to Library > Mobile Documents
  5. Other cloud storage: Navigate to Library > CloudStorage

"iCloud Drive" is not a volume nor folder; it's actually a collection of many disparate folders

When you open "iCloud Drive" in the Finder sidebar, you see a simple list of files and folders. Some of those folders may have special icons representing the application that stores data in that folder, e.g. for Preview, Pages, TextEdit, etc. Looking at the content of iCloud Drive in the Finder, you might assume that there is a folder somewhere ("in the sidebar") that has all of those items collected together.

iCloud Drive does not work like that. What you see in the Finder is a Finder trick. iCloud Drive is actually a collection of folders hidden away in the Library folder in your home directory. Files and folders that you manually add to iCloud Drive are stored here:

Macintosh HD --> Users > (yourname) > Library > Mobile Documents > com~apple~CloudDocs

Application storage folders are kept elsewhere. If you have a Pages folder in iCloud Drive, for example, that content would be stored here:

Macintosh HD --> Users > (yourname) > Library > Mobile Documents > com~apple~Pages > Documents

Making matters even more complicated, if you choose to sync your Desktop & Documents folders (i.e., System Settings > Apple ID > iCloud Drive > Options), the Finder will make it appear as if your Desktop and Documents folders actually reside within iCloud Drive. In fact, those folders still exist in their normal locations:

Macintosh HD --> Users > (yourname) > Desktop
Macintosh HD --> Users > (yourname) > Documents

But you won't see those folders in those locations when you navigate there in the Finder — Finder hides them.

Backing up cloud-only content

Some cloud storage service providers offer features that allow (or even encourage/force) you to store your files only online, thus freeing up space on your hard drive. Some services that currently offer this functionality include:

  • Dropbox Professional's "Smart Sync" feature
  • Microsoft OneDrive's "Free up space" feature
  • iCloud Drive's "Optimize Mac Storage" feature
  • Google Drive's "File Stream" feature
  • Box.com's cloud-only files

Files that are only available online will typically have a "cloud" icon or badge in the Finder, e.g., iCloud: File only available in iCloud and Dropbox: File only available in Dropbox online

When a file stored by one of these storage services is flagged to reside only online, the local copy of the file is deleted from your Mac and replaced with a 0-byte placeholder file. While this is a convenient feature that allows you to free up some space on your Mac, it imposes a logistical challenge to create a local backup of those files. If you want to have a local backup of these cloud-only files, CCC must temporarily download these files to your startup disk. CCC can do this, but because this involves downloading a potentially large amount of data from the Internet, this functionality is disabled by default. Likewise, allowing this data to co-mingle with your startup disk's backup could lead to a situation where it is impossible to restore your entire backup to the original disk due to space constraints. To avoid that, we recommend making backups of your cloud-only storage to a separate volume on your backup disk.

Best practice for setting up a CCC task to back up cloud-only data

See Preparing your destination disk for a backup or restore for guidance if your backup disk is not APFS-formatted.

  1. Open Disk Utility and select your APFS-formatted primary backup volume in the sidebar.
  2. Click + in the toolbar to add a volume; name it something like "Cloud Storage Backup".
  3. Open CCC and click New Task in the toolbar; name the new task something like "Local Backup of Cloud Storage".
  4. Click on the Source selector and choose Cloud Storage
  5. Click on the Destination selector and choose the "Cloud Storage Backup" volume as the destination.
  6. Click Done, then schedule the task or run it immediately.

When this setting is enabled, CCC will temporarily download cloud-only files that are not yet on the destination, or that are newer than the corresponding file on the destination. After copying the temporarily downloaded files, CCC will "evict" the files to free up the space that they consumed. CCC attempts to retain no more than 100 files and no more than 2GB of temporarily-downloaded content at a time.

This functionality requires macOS Monterey 12.5+.

The Cloud Storage task creates a custom filter

The Cloud Storage source will create a filter that automatically includes the following folders in your home directory:

  • Desktop
  • Documents
  • Library > Mobile Documents
  • Library > CloudStorage

The first three folders are specific to iCloud, and the first two folders are only applicable if you have iCloud Drive configured to sync your Desktop and Documents folders. If you don't sync your Desktop and Documents folders to iCloud, you can click Task Filter at the bottom of the CCC window, then uncheck the boxes next to your Desktop and Documents folder to not include those folders in this backup task.

Some iCloud cloud-only content will not be temporarily downloaded

Starting in macOS Monterey (12.3, January 2022), Apple disallowed cloud syncing by means of a System Extension. Cloud-syncing service providers like Google, Microsoft, and Dropbox were "encouraged" to adopt the "FileProvider" service within macOS instead. Prior to macOS Sonoma, Apple had not yet adopted that service for their own iCloud Drive cloud syncing solution. Rather, Apple continued to use a proprietary syncing service on macOS Monterey and Ventura that relied on proprietary placeholder files.

Lack of adoption of their own standard leads to some idiosyncrasies when dealing with iCloud cloud-only content on Monterey and Ventura. The only notable problem that proved too difficult for us to work around involves bundle files. iCloud uses a single file placeholder for bundle files (vs. dataless folders via FileProvider). This (lack of) structure posed logistical and practical issues that we decided were too costly to resolve, especially in light of the fact that Apple migrated iCloud to FileProvider in macOS Sonoma. As such, cloud-only iCloud files that are bundle files will not be downloaded on Monterey and Ventura, rather the placeholder file will be copied instead.

Other notable caveats regarding the temporary downloading of cloud-only content

CCC's helper tool must have access to icloud.com for the entire duration of the task event

If connectivity is not available or is lost to that host (regardless of the service provider that hosts your cloud-backed content), CCC will suspend downloading activity in the task. We do this because it's not possible to revoke requests to download files from the cloud. If we plowed through all of the files and requested all of them, iCloud/FileProvider would resume downloading all of those files as soon as connectivity was restored. Making matters worse, this would happen outside of the purview of the task, and CCC would not be able to evict the downloaded files, likely leading to space constraints on the startup disk.

CCC's Dashboard application must be running for the entire duration of the backup task

Only the logged-in user is allowed to "evict" file content, so CCC passes those requests to the CCC Dashboard service. If CCC cannot reliably evict files, it will not download files. If/when this occurs, CCC will report an error for the task event.

Verification discrepancies

If you verify a source that has cloud-only content, the cloud-only placeholder files will fail verification. This is the correct result, because the (empty) content of the cloud-only placeholder file does not match the content of the "rehydrated" file that was temporarily downloaded. CCC's transaction only retains a checksum of the actual data that was downloaded, not the empty placeholder file.

Cloud-only placeholder files will not be downloaded for checksum analysis

When using the "Find and replace corrupted files on the destination" setting (aka Backup Health Check), CCC will not download cloud-only placeholder files from the Internet just to calculate their checksums. If CCC copied these files in the past (either via temporary download or prior to eviction), then the task audit will already have the checksum for those files. To verify those files, click on the Destination selector for your dedicated CloudStorage backup task and choose Verify files copied by this task.

Ad hoc verification: Verify the source or the destination against the "last known state"

Cloud service problems can prevent CCC from downloading cloud-only files

Naturally your own Internet connectivity problems can prevent CCC from downloading cloud-only files, but so can service problems on the cloud-provider's end. Most cloud-service providers offer a dashboard showing their server status. Here are a few for your convenience: