Difference between revisions of "ViSUS Convert"

From
Jump to: navigation, search
(Compression)
 
(29 intermediate revisions by the same user not shown)
Line 7: Line 7:
 
In the following examples we will use the variable <code>$CONVERT</code> as the visus convert executable.
 
In the following examples we will use the variable <code>$CONVERT</code> as the visus convert executable.
 
You can set this variable using your OS commands like:
 
You can set this variable using your OS commands like:
 +
<syntaxhighlight lang="bash">
 
  export CONVERT=/path/to/visus_executable
 
  export CONVERT=/path/to/visus_executable
 +
</syntaxhighlight>
 
or
 
or
 +
<syntaxhighlight lang="bash">
 
  setenv CONVERT /path/to/visus_executable
 
  setenv CONVERT /path/to/visus_executable
 
+
</syntaxhighlight>
 
The variable <code>$RESOURCES</code> is used instead to indicate a folder that contains the data to convert.
 
The variable <code>$RESOURCES</code> is used instead to indicate a folder that contains the data to convert.
  
Line 16: Line 19:
 
* a ''box'' defines the extent of the domain where the samples are placed for each dimension
 
* a ''box'' defines the extent of the domain where the samples are placed for each dimension
 
* the ''dims'' are the number of samples for each dimension
 
* the ''dims'' are the number of samples for each dimension
* the ''dtype'' (i.e. datatype) is defined as a combination of the number type and the number of components (e.g. int8 with 3 components will be <code>int8[3]</code>, for single component a simple <code>int8</code> can be used)
+
* the ''dtype'' (i.e. datatype) is defined as a combination of the number type and the number of components (e.g. an RGB vector uint8 with 3 components will be <code>uint8[3]</code>, for single component a simple <code>uint8</code> can be used)
  
 
== Examples ==
 
== Examples ==
Line 22: Line 25:
 
=== Simple conversion ===
 
=== Simple conversion ===
 
For very simple conversions, all you need is:
 
For very simple conversions, all you need is:
 
+
<syntaxhighlight lang="bash">
  $CONVERT import $RESOURCES/cat_gray.tga create temp/cat_gray.idx  
+
  $CONVERT import $RESOURCES/cat_gray.tga create temp/cat_gray.idx
 +
</syntaxhighlight>
  
 
=== Get information ===  
 
=== Get information ===  
 
To *get information* about any file (including non-idx), use info
 
To *get information* about any file (including non-idx), use info
  
$CONVERT info $RESOURCES/cat_gray.tga
+
<syntaxhighlight lang="bash">
$CONVERT info $RESOURCES/cat_rgb.tga
+
$CONVERT info $RESOURCES/cat_gray.tga
$CONVERT info temp/cat_gray.idx
+
$CONVERT info $RESOURCES/cat_rgb.tga
$CONVERT info temp/cat_rgb.idx
+
$CONVERT info temp/cat_gray.idx
 +
$CONVERT info temp/cat_rgb.idx
 +
</syntaxhighlight>
  
 
=== Create the IDX dataset ===
 
=== Create the IDX dataset ===
Line 38: Line 44:
  
 
You can define a set of fields:
 
You can define a set of fields:
 +
<syntaxhighlight lang="bash">
 +
fields="scalar uint8 compressed + vector uint8[3] compressed"
 +
</syntaxhighlight>
 +
and use the following to create the dataset
 +
<syntaxhighlight lang="bash">
 +
$CONVERT create temp/test_formats.idx --box "0 511 0 511" --fields "$fields" --time 0 0 time%03d/
 +
</syntaxhighlight>
  
  fields="scalar uint8 compressed + vector uint8[3] compressed"
+
=== Convert from a RAW dataset ===
 +
 
 +
With the following example we convert an array of ''int16'' to IDX:
 +
<syntaxhighlight lang="bash">
 +
$CONVERT import data.raw --dtype 1*int16 --dims "995 501 401" export data.idx --dims "995 501 401" --field my_field --time 0 --box "0 994 0 500 0 400"
 +
</syntaxhighlight>
 +
Or simply:
 +
<syntaxhighlight lang="bash">
 +
  $CONVERT import data.raw --dtype 1*int16 --dims "995 501 401 1" create data.idx
 +
</syntaxhighlight>
 +
 
 +
=== Interleave multiple fields into one ===
 +
 
 +
With the following example we import 2 raw datasets into one interleaved IDX dataset:
 +
<syntaxhighlight lang="bash">
 +
$CONVERT --import /dev/null --dims "995 501 401 2" --dtype "int16" \
 +
--paste data0.raw --dtype 1*int16 --dims "995 501 401 1" --destination-box "0 994 0 500 0 400 1 1" --source-box "0 994 0 500 0 400" \
 +
--paste data1.raw --dtype 1*int16 --dims "995 501 401 1" --destination-box "0 994 0 500 0 400 0 0" --source-box "0 994 0 500 0 400" \
 +
--resize --dtype "int16[2]" --dims "995 501 401 1" --interleave --create data0_data1.idx
 +
</syntaxhighlight>
 +
The new field created with this example will be a vector with 2 components.
 +
 
 +
=== Convert a stack of images ===
 +
 
 +
Very often microscopy data or MRI scans are simply big stacks of 2D images. Here is how you can easily convert them in a single IDX dataset using visus tools:
 +
<syntaxhighlight lang="bash">
 +
# path to your brand new IDX file
 +
IDXFILE="G:/visus_dataset/2kbit1/visus.idx"
 +
 
 +
# box is specified as [x1 x2] [y1 y2] [z1 z2], min-max included (in this case the dataset has size 2048^3)
 +
BOX="0 2047 0 2047 0 2047"
 +
 
 +
# change your data DTYPE (in this case a single Uint8)
 +
DTYPE="uint8[1]"
 +
 
 +
$CONVERT create "$IDXFILE" --box "$BOX" --fields "data $DTYPE"
  
and use the following to create the dataset
+
# example of conversion of png slices to IDX
$CONVERT create temp/test_formats.idx --box "0 511 0 511" --fields "$fields"
+
for sliceid in $(seq -f "%04g" 0 2047)
 +
do
 +
  $CONVERT \
 +
    import "G:/visus_dataset/2kbit1/png/$sliceid.png" \
 +
    export "$IDXFILE" --box "0 2047 0 2047 $sliceid $sliceid"  
 +
done
 +
</syntaxhighlight>
  
 
=== Tiled dataset ===  
 
=== Tiled dataset ===  
Multiple images can be combined (as tiles) to compose a panorama or volume (e.g. stack of image from RMI scans).
+
Multiple images can be combined (as tiles) to compose a panorama or volume (e.g. stack of images from an MRI scan).
  
 
This can be easily performed specifying the destination ''box'' of each image, as following:
 
This can be easily performed specifying the destination ''box'' of each image, as following:
 +
<syntaxhighlight lang="bash">
 +
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "  0 255    0 255"
 +
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "256 511    0 255"
 +
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "  0 255  256 511"
 +
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "256 511  256 511"
 +
</syntaxhighlight>
 +
 +
=== Conversion from IDX to image ===
 +
 +
Conversion from IDX to image formats can be performed as well using the same technique, as following:
 +
<syntaxhighlight lang="bash">
 +
$CONVERT import temp/test_formats.idx --field scalar --box "  0 255    0 255"  export temp/test_formats_scalar.tga
 +
$CONVERT import temp/test_formats.idx --field scalar --box "256 511    0 255"  export temp/test_formats_scalar.jpg 
 +
$CONVERT import temp/test_formats.idx --field scalar --box "  0 255  256 511"  export temp/test_formats_scalar.tif 
 +
$CONVERT import temp/test_formats.idx --field scalar --box "256 511  256 511"  export temp/test_formats_scalar.bmp
 +
$CONVERT import temp/test_formats.idx --field scalar --box "128 383  128 383"  export temp/test_formats_scalar.png
 +
</syntaxhighlight>
 +
 +
== Compression ==
 +
 +
You can compress your dataset (already created) using zip or lz4, as following:
 +
<syntaxhighlight lang="bash">
 +
$CONVERT compress-dataset your_dataset.idx lz4
 +
</syntaxhighlight>
 +
 +
You can also compress float values using the zfp lossy compression specifying the number of bits you want to store each sample, for example for 16 bits it will:
 +
<syntaxhighlight lang="bash">
 +
$CONVERT compress-dataset your_dataset.idx zfp-16
 +
</syntaxhighlight>
 +
 +
Note: The maximum number of bit planes to compress is the same as the bit-size of the type (e.g. 32 for float32, 64 for float64, 16 for for int16, etc).
 +
 +
When importing directly to a new dataset you can use:
 +
 +
<syntaxhighlight lang="bash">
 +
$CONVERT import neon.tif create neon.idx --fields "data float32 default_compression(zfp-16) format(1)"
 +
$CONVERT import neon.tif export neon.idx --box "0 999 0 999"
 +
</syntaxhighlight>
  
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box " 0 255    0 255"  
+
Note the "format(1)" part which is important to achieve good compression.  
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "256 511    0 255"
 
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "  0 255  256 511"
 
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "256 511  256 511"
 
  
=== Conversion from image to IDX ===
+
To decompress, use:
  
Conversion from IDX to image formats can be performed as well using the same technique, as following:
+
<syntaxhighlight lang="bash">
 +
$CONVERT import neon.idx --box "0 999 0 999" export neon-out.tif
 +
</syntaxhighlight>
 +
 
 +
Also, RGB fields can be compressed with JPG instead of ZFP, with the following parameters:
  
$CONVERT import temp/test_formats.idx --field scalar --box " 0 255    0 255" export temp/test_formats_scalar.tga
+
<syntaxhighlight lang="bash">
$CONVERT import temp/test_formats.idx --field scalar --box "256 511    0 255" export temp/test_formats_scalar.jpg
+
$CONVERT import neon.tif create neon.idx --fields "data uint8[3] default_compression(jpg-JPEG_QUALITYSUPERB-JPEG_SUBSAMPLING_420-JPEG_OPTIMIZE) format(1)"
$CONVERT import temp/test_formats.idx --field scalar --box " 0 255  256 511"  export temp/test_formats_scalar.tif 
+
$CONVERT import neon.tif export neon.idx --box "0 999 0 999"
$CONVERT import temp/test_formats.idx --field scalar --box "256 511  256 511"  export temp/test_formats_scalar.bmp
+
</syntaxhighlight>
$CONVERT import temp/test_formats.idx --field scalar --box "128 383  128 383" export temp/test_formats_scalar.png
 

Latest revision as of 23:43, 26 September 2019

Introduction

ViSUS convert is the tool that allows to convert data from different file formats (i.e. mainly image formats and raw) to IDX.

The supported formats are the ones supported by FreeImage.

In the following examples we will use the variable $CONVERT as the visus convert executable. You can set this variable using your OS commands like:

export CONVERT=/path/to/visus_executable

or

setenv CONVERT /path/to/visus_executable

The variable $RESOURCES is used instead to indicate a folder that contains the data to convert.

Some useful definitions:

  • a box defines the extent of the domain where the samples are placed for each dimension
  • the dims are the number of samples for each dimension
  • the dtype (i.e. datatype) is defined as a combination of the number type and the number of components (e.g. an RGB vector uint8 with 3 components will be uint8[3], for single component a simple uint8 can be used)

Examples

Simple conversion

For very simple conversions, all you need is:

$CONVERT import $RESOURCES/cat_gray.tga create temp/cat_gray.idx

Get information

To *get information* about any file (including non-idx), use info

$CONVERT info $RESOURCES/cat_gray.tga
$CONVERT info $RESOURCES/cat_rgb.tga
$CONVERT info temp/cat_gray.idx
$CONVERT info temp/cat_rgb.idx

Create the IDX dataset

To explicitly create the idx volume

You can define a set of fields:

fields="scalar uint8 compressed + vector uint8[3] compressed"

and use the following to create the dataset

$CONVERT create temp/test_formats.idx --box "0 511 0 511" --fields "$fields" --time 0 0 time%03d/

Convert from a RAW dataset

With the following example we convert an array of int16 to IDX:

$CONVERT import data.raw --dtype 1*int16 --dims "995 501 401" export data.idx --dims "995 501 401" --field my_field --time 0 --box "0 994 0 500 0 400"

Or simply:

$CONVERT import data.raw --dtype 1*int16 --dims "995 501 401 1" create data.idx

Interleave multiple fields into one

With the following example we import 2 raw datasets into one interleaved IDX dataset:

$CONVERT --import /dev/null --dims "995 501 401 2" --dtype "int16" \
 --paste data0.raw --dtype 1*int16 --dims "995 501 401 1" --destination-box "0 994 0 500 0 400 1 1" --source-box "0 994 0 500 0 400" \
 --paste data1.raw --dtype 1*int16 --dims "995 501 401 1" --destination-box "0 994 0 500 0 400 0 0" --source-box "0 994 0 500 0 400" \
 --resize --dtype "int16[2]" --dims "995 501 401 1" --interleave --create data0_data1.idx

The new field created with this example will be a vector with 2 components.

Convert a stack of images

Very often microscopy data or MRI scans are simply big stacks of 2D images. Here is how you can easily convert them in a single IDX dataset using visus tools:

# path to your brand new IDX file
IDXFILE="G:/visus_dataset/2kbit1/visus.idx"

# box is specified as [x1 x2] [y1 y2] [z1 z2], min-max included (in this case the dataset has size 2048^3)
BOX="0 2047 0 2047 0 2047"

# change your data DTYPE (in this case a single Uint8)
DTYPE="uint8[1]"

$CONVERT create "$IDXFILE" --box "$BOX" --fields "data $DTYPE"

# example of conversion of png slices to IDX
for sliceid in $(seq -f "%04g" 0 2047)
do
  $CONVERT \
     import "G:/visus_dataset/2kbit1/png/$sliceid.png" \
     export "$IDXFILE" --box "0 2047 0 2047 $sliceid $sliceid" 
done

Tiled dataset

Multiple images can be combined (as tiles) to compose a panorama or volume (e.g. stack of images from an MRI scan).

This can be easily performed specifying the destination box of each image, as following:

$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "  0 255     0 255" 
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "256 511     0 255"
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "  0 255   256 511"
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "256 511   256 511"

Conversion from IDX to image

Conversion from IDX to image formats can be performed as well using the same technique, as following:

$CONVERT import temp/test_formats.idx --field scalar --box "  0 255     0 255"  export temp/test_formats_scalar.tga 
$CONVERT import temp/test_formats.idx --field scalar --box "256 511     0 255"  export temp/test_formats_scalar.jpg  
$CONVERT import temp/test_formats.idx --field scalar --box "  0 255   256 511"  export temp/test_formats_scalar.tif  
$CONVERT import temp/test_formats.idx --field scalar --box "256 511   256 511"  export temp/test_formats_scalar.bmp 
$CONVERT import temp/test_formats.idx --field scalar --box "128 383   128 383"  export temp/test_formats_scalar.png

Compression

You can compress your dataset (already created) using zip or lz4, as following:

$CONVERT compress-dataset your_dataset.idx lz4

You can also compress float values using the zfp lossy compression specifying the number of bits you want to store each sample, for example for 16 bits it will:

$CONVERT compress-dataset your_dataset.idx zfp-16

Note: The maximum number of bit planes to compress is the same as the bit-size of the type (e.g. 32 for float32, 64 for float64, 16 for for int16, etc).

When importing directly to a new dataset you can use:

$CONVERT import neon.tif create neon.idx --fields "data float32 default_compression(zfp-16) format(1)"
$CONVERT import neon.tif export neon.idx --box "0 999 0 999"

Note the "format(1)" part which is important to achieve good compression.

To decompress, use:

$CONVERT import neon.idx --box "0 999 0 999" export neon-out.tif

Also, RGB fields can be compressed with JPG instead of ZFP, with the following parameters:

$CONVERT import neon.tif create neon.idx --fields "data uint8[3] default_compression(jpg-JPEG_QUALITYSUPERB-JPEG_SUBSAMPLING_420-JPEG_OPTIMIZE) format(1)"
$CONVERT import neon.tif export neon.idx --box "0 999 0 999"