Assets CSV import job does not import all assets from the file
Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Assets CSV import job does not import all assets from the file. For example, only 2000 assets are imported from a CSV file containing more than 10000 assets.
Environment
Assets 8.10.15 or newer
Diagnosis
There are no errors in insight_import.log files. However, after enabling debugging for package com.riadalabs.jira.plugins.insight.services.imports.schema.log.ImportLogger
,
the following appears in the import log while the import task is running:
2023-10-16 13:49:08,777 [insight-InsightImportThreadGroup-worker-thread-1] | Row not read on line number (heading row included) 4622 due to The number of columns to be processed (22) must match the number of CellProcessors (21): check that the number of CellProcessors you have defined matches the expected number of columns being read/written
Furthermore, analyzing the file, we can see there are many data rows parsed as a single row for a data locator:
2023-10-16 13:49:13,462 [insight-InsightImportThreadGroup-worker-thread-6] | dataLocatorsValue: value1;value2;value3;value4;;
000000000000001;12/10/2023;01:15:16;0003000000;1000;60;10;X;;*test1*;;;;;;;;;;;;;
000000000000002;12/10/2023;01:15:16;0003000001;1001;20;10;X;;*test2*;;;;;;;;;;;;;
...
...
000000000009999;12/10/2023;01:37:19;00030000002;1002;50;10;;;*test9999*;;;;;;;;;;;;;; for dataLocator [DataLocator [locator=test-locator]]
Cause
This problem could occur when a CSV column contains a single quotation (") char. A CSV file should be escaped properly with the quotes and if not, the application would consider it as a single parameter until it finds the escape character.
Debug lines show many rows are being treated as a single data locator value for a data locator, effectively treating number of assets as a single one; this is a consequence of the single / double quote not being escaped.
Solution
Check the CSV file and see where are quotes that don't have its closing match, or if there are semicolon ";" characters that are being part of the value, instead of being a delimiter. After fixing the cause in the CSV file, run the Assets import task again.