Problem

Something went wrong. Contact administrator pop-up message in Assets can be seen when making one of the following actions:

• Filtering on or searching for objects in Assets 

• When triggering an Assets Reindex or when attempting to edit the configuration of an Object-referencing attribute
• When attempting to delete an Assets Schema from the UI.

The following appears in the atlassian-jira.log:

2021-05-06 14:27:55,871 Assets-AssetsThreadGroup-worker-thread-6 ERROR admin 863x2041x1 1gmljw3 127.0.0.1 /rest/Assets/1.0/index/reindex/start [i.r.i.index.model.ObjectIndexImpl] Index not completed successfully. objectIds: ([1, 2, 5, 6, 85])
com.google.common.cache.CacheLoader$InvalidCacheLoadException: loadAll failed to return a value for 3
    at com.google.common.cache.LocalCache.getAll(LocalCache.java:4025)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getAll(LocalCache.java:4972)
    at io.riada.Assets.index.model.AssetsIndexBase.loadAll(AssetsIndexBase.java:81)
    at io.riada.Assets.index.model.ObjectIndexImpl.reindexObjects(ObjectIndexImpl.java:318)
    at com.riadalabs.jira.plugins.Assets.services.core.index.ReindexServiceImpl$ReindexObjectsJob.executeTask(ReindexServiceImpl.java:543)
    at com.riadalabs.jira.plugins.Assets.services.core.index.ReindexServiceImpl$ReindexObjectsJob.executeTask(ReindexServiceImpl.java:520)
    at com.riadalabs.jira.plugins.Assets.services.core.multithreadservice.AssetsServiceJob.call(AssetsServiceJob.java:42)
    at com.atlassian.sal.core.executor.ThreadLocalDelegateCallable.call(ThreadLocalDelegateCallable.java:38)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)

• The "objectIds" list shows all of the Object IDs that Assets was attempting to re-index at that specific operation (your list may be way longer).
• The "return a value for" will indicate the Object ID where Assets first encountered problems from the above list.

Diagnosis

Environment

• This problem has been detected on Assets major versions > 8.4.x

• It causes the Assets Index not completed successfully due to some corrupted objects due to some kind of race condition. This happens after operations as
  • Deletion of objects
  • Deletion/change of attributes
  • Deletion of ObjectTypes
  • Deletion of ObjectShemas

Diagnostic Steps

To diagnose this problem:

1. Review InvalidCacheLoadException: loadAll failed to return a value for xxx KB and run the SQL queries suggested there. If none rows are returned, please proceed with the following steps shared in this article.
2. You might look for the above-mentioned errors in the Jira logs and run the below query on the Jira database:

3. A row should be returned and if you have null values for all the right tables indicates that object got corrupted and needs to be deleted from your database.

   FOR POSTGRES:

   SELECT
     OAV."TEXT_VALUE",
     O."NAME",
     O."OBJECT_TYPE_ID",
     OTA."NAME",
     OA."ID",
     OA."OBJECT_TYPE_ATTRIBUTE_ID",
     OA."OBJECT_ID",
     OAV."ID",
     OAV."OBJECT_ATTRIBUTE_ID"FROM "AO_8542F1_IFJ_OBJ" O
     LEFT OUTER JOIN "AO_8542F1_IFJ_OBJ_ATTR" OA ON O."ID" = OA."OBJECT_ID"
     LEFT OUTER JOIN "AO_8542F1_IFJ_OBJ_TYPE_ATTR" OTA ON OTA."ID" = OA."OBJECT_TYPE_ATTRIBUTE_ID"
     LEFT OUTER JOIN "AO_8542F1_IFJ_OBJ_ATTR_VAL" OAV ON OA."ID" = OAV."OBJECT_ATTRIBUTE_ID"
   WHERE O."ID" = ?;

   ? needs to be replaced by the the Object ID you found after the following message "com.google.common.cache.CacheLoader$InvalidCacheLoadException: loadAll failed to return a value for X "

   You may have more than one exception so you will need to perform the same steps for each of the values. 

4. Example of corrupted object

    TEXT_VALUE |    NAME     | OBJECT_TYPE_ID | NAME | ID | OBJECT_TYPE_ATTRIBUTE_ID | OBJECT_ID | ID | OBJECT_ATTRIBUTE_ID 
   ------------+-------------+----------------+------+----+--------------------------+-----------+----+------------
   
         null  | webex        |  97 |  null    | null   |           null               |     null      |  null  |            
   (1 row)

5. Before we move to delete those objects, run the following query. It should return four standard Object Type attributes, just making sure that the problem is not on the object type level.

   SELECT * FROM "AO_8542F1_IFJ_OBJ_TYPE_ATTR" WHERE "OBJECT_TYPE_ID" = ? AND ("NAME" IN ('Key', 'Created', 'Updated') OR "LABEL"='t');

   ? needs to be replaced by the Object Type ID from the first query output

   1. Query should return exactly 4 attributes (Key, Created, Updated and the current Label), and all additional Attributes, if configured. 

      1. If none attributes are retuned, there is an issue with the Object Type and needs to be removed by following Cannot Delete Object Types on Assets Asset Management due to database constraint violation exception
      2. If the 4 attributes are returned, proceed with the Resolution steps below:

Cause

Some manual database changes is required for Assets 

Resolution

1. Stop Jira
2. Execute the SQL queries below in Jira database.

3. Check all inbound references pointing to the corrupt object (if any), as these may have been the initial reason for the object not deleting:

   SELECT * FROM "AO_8542F1_IFJ_OBJ_ATTR_VAL" WHERE "REFERENCED_OBJECT_ID"= ?

4. Delete these references (attribute values) if any.

   DELETE FROM "AO_8542F1_IFJ_OBJ_ATTR_VAL" WHERE "REFERENCED_OBJECT_ID"=?

5. Once the inbound references to the corrupt objects are removed, you should be able to DELETE the corrupt object correctly, using:

   DELETE FROM "AO_8542F1_IFJ_OBJ" WHERE "ID" = ?;

   ? needs to be replaced by the Object ID you found after the following message "com.google.common.cache.CacheLoader$InvalidCacheLoadException: loadAll failed to return a value for X "

   You may have more than one exception so you will need to perform the same steps for each of the values. 

6. Run a clean reindex of Assets.
7. Verify that the issue is solved by checking the atlassian-jira.log for the completion of Assets indexing without error.