BUG: Fix Import DREAM3D Memory Usage #1341
Open
+1,838
−101
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* ImportH5ObjectPathsAction allocation improvements. * EmptyListStore created for preflight and unimported NeighborLists. * Allow StringArray to change its underlying store. * Update HDF5 DataStructure IO to finish importing preflight DataObjects. * Update DataStoreUtilities to create arbitrary AbstractListStores based on memory usage. * Update Dream3dIO to finish importing DataObjects from DREAM3D files. * Update DatasetIO to read values as an AbstractDataStore. Updates BlueQuartzSoftware#1312 Updates BlueQuartzSoftware#1273
* Import Legacy DREAM3D file always imports geometry arrays as if running in execute. * Fixing importing DREAM3D data so that data importing can be delayed until running execute.
* Fixed PartitionGeometryTest * Fixed importing Legacy DREAM.3D files. * Fixed importing current DREAM.3D files for filters that require data during preflight and updated DataStructureReader for partially imported data. * added additional readAsDataStore method for DatasetIO with specific tuple and component dimensions. * Commented out ImportH5ObjectPathsAction errors related to partial data imports as they are no longer viable with the rework.
mmarineBlueQuartz
force-pushed
the
feature/1273_NeighborListIO
branch
from
d322f2c to
d5ffed0
Compare
|
@mmarineBlueQuartz Initial testing on a 75 GB .dream3d file allocated over 100GB of RAM just trying to add the "Read DREAm3D" filter to the pipeline. If you need this test file let me know and I can post it to Google Drive |
…cated or not Signed-off-by: Michael Jackson <[email protected]>
Geometries only add required data ID if using empty data stores
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Broke up importing DREAM3D files into two stages so that only DataObjects requested are fully imported. Other data is imported using empty data stores to serve as placeholders. Geometries will always import the corresponding arrays completely during preflight due to other filters using those arrays during their calculations.
Added ability to import HDF5 dataset as an AbstractDataStore with or without predetermined tuple and component shapes. This will use the appropriate type of data store based on the required memory size and user preferences. Reading as a vector would not work when using out-of-core data.