Assets CSV import job does not import all assets from the file


Platform notice: Server and Data Center only. This article only applies to Atlassian products on the Server and Data Center platforms.

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

    

Summary

Assets CSV import job does not import all assets from the file. For example, only 2000 assets are imported from a CSV file containing more than 10000 assets.

Environment

Assets 8.10.15 or newer

Diagnosis

There are no errors in insight_import.log files. However, after enabling debugging for package com.riadalabs.jira.plugins.insight.services.imports.schema.log.ImportLogger,
the following appears in the import log while the import task is running:

2023-10-16 13:49:08,777 [insight-InsightImportThreadGroup-worker-thread-1] | Row not read on line number (heading row included) 4622 due to The number of columns to be processed (22) must match the number of CellProcessors (21): check that the number of CellProcessors you have defined matches the expected number of columns being read/written

Furthermore, analyzing the file, we can see there are many data rows parsed as a single row for a data locator:

2023-10-16 13:49:13,462 [insight-InsightImportThreadGroup-worker-thread-6] | dataLocatorsValue: value1;value2;value3;value4;;
000000000000001;12/10/2023;01:15:16;0003000000;1000;60;10;X;;*test1*;;;;;;;;;;;;;
000000000000002;12/10/2023;01:15:16;0003000001;1001;20;10;X;;*test2*;;;;;;;;;;;;;
...
...
000000000009999;12/10/2023;01:37:19;00030000002;1002;50;10;;;*test9999*;;;;;;;;;;;;;; for dataLocator [DataLocator [locator=test-locator]]


Cause

This problem could occur when a CSV column contains a single quotation (") char. A CSV file should be escaped properly with the quotes and if not, the application would consider it as a single parameter until it finds the escape character.

Debug lines show many rows are being treated as a single data locator value for a data locator, effectively treating number of assets as a single one; this is a consequence of the single / double quote not being escaped.

Solution

Check the CSV file and see where are quotes that don't have its closing match, or if there are semicolon ";" characters that are being part of the value, instead of being a delimiter. After fixing the cause in the CSV file, run the Assets import task again.


Last modified on Oct 30, 2023

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.