Problem

After upgrading your Hipchat Server to 2.2.3 and newer, some files (such as shared files, avatars, and custom emoticons) are missing.  

One or more of the following WARN messages appear in /var/log/chef.log: 

WARN: File tmpzGYVKifiles__photos__1146__kqcDFWxvJK8kYRa.png exists neither in posix_store nor on FS!"
WARN: File /file_store/cumulus/posixdata/f/tmpzVMizYfiles__1__22__3Qq8RiRNuJVdFkY__koala.png failed to move to /file_store/local/files/1/22/3Qq8RiRNuJVdFkY/koala.png
WARN: Some files could not be migrated. See /var/log/hipchat/cumulus.log and https://confluence.atlassian.com/x/Mi3tN for details.

Cause

When you upgrade to 2.2.3, the upgrade process performs a file migration to a new directory (/file_store/local/files). In certain situations the migration script can fail to migrate some of the available files. For example if some files were manually deleted, or are temporarily unavailable (for example due to a lock on the file), or if the file database is corrupted, or if the destination location doesn't have enough space, the script will produce an error. Files that are not migrated are unavailable to your users after the upgrade.

Diagnostics

Check that:

• /file_store/local (the post-migration) directory exists and is owned by hipchat:www-data

• /file_store/cumulus (the pre-migration source directory) exists

• the partition where /file_store/local resides has free space (The migration script does not need much extra space, as this is a move command instead of a copy. However it does require a little working room.)
• the missing files were manually deleted (if they no longer exist in the file system, they were deleted)

Workaround / Resolution

Re-run the migration

If the error was caused by a temporary issue, you can re-run the migration script using the following command:

/opt/atlassian/hipchat/virtualenv/bin/python /opt/atlassian/hipchat/sbin/migrate_cumulus_to_fs.py --cumulus_db /file_store/cumulus/etc/authz.db.bak

If you still get migration errors after running the command, you can copy it to the destination manually.

Placeholders for deleted files

If the file doesn't exist in the original location, this most likely means it was deleted manually.

• If you have the original file, you can manually copy it to the destination mentioned in the error message. 
  (You must create the directories and rename the file to the expected destination in the error message from the log)
• If you don't have the original file, you can create a placeholder file. If you do not, any chat messages referencing the missing file display an error.

# an example error message might look like this: 
WARN: File /file_store/cumulus/posixdata/f/tmpb_BJBTfiles__1__234__9QTSoePwsjBDpjn__example.jpg exists neither in posix_store nor on FS!

# to create an empty placeholder file:
touch "/file_store/local/files/1/234/9QTSoePwsjBDpjn/example.jpg"

# to migrate the original file manually:
cp "tmpb_BJBTfiles__1__234__9QTSoePwsjBDpjn__example.jpg" "/file_store/local/files/1/234/9QTSoePwsjBDpjn/example.jpg"

# to ensure that HipChat owns the file so it can be served to users:
chown hipchat:www-data "/file_store/local/files/1/234/9QTSoePwsjBDpjn/example.jpg"
chmod 660 "/file_store/local/files/1/234/9QTSoePwsjBDpjn/example.jpg"