Difference between revisions of "Flat File Importer"

From LearnLab
Jump to: navigation, search
(New page: '''Status: Prioritization Needed''' == User Story == As a DataShop administrator, I want to redesign Dataset Import Tool to load the dataset into database by processing it column by colu...)
 
Line 10: Line 10:
 
* The import sometime has failed for several large datasets.  
 
* The import sometime has failed for several large datasets.  
 
* We'd like to rewrite this import tool to allow column by column process and avoid Hibernate layer to make the import faster.  
 
* We'd like to rewrite this import tool to allow column by column process and avoid Hibernate layer to make the import faster.  
* The goal is to process 1 million rows per minute on import only. [[User:Shanwen|Shanwen]]  
+
* The goal is to process 1 million rows per minute on import only. [[User:Shanwen|Shanwen]]10:09, 12 October 2010 (EDT)
 
<br>
 
<br>
 
----
 
----

Revision as of 14:09, 12 October 2010

Status: Prioritization Needed

User Story

As a DataShop administrator, I want to redesign Dataset Import Tool to load the dataset into database by processing it column by column so that I can speed up the import process.

Notes/Comments

  • Current Dataset Import Tool processes teh dataset row by row and uses Hibernate layer, which takes a long time to import a dataset.
  • The import sometime has failed for several large datasets.
  • We'd like to rewrite this import tool to allow column by column process and avoid Hibernate layer to make the import faster.
  • The goal is to process 1 million rows per minute on import only. Shanwen10:09, 12 October 2010 (EDT)



See complete DataShop Feature Wish List.