ViSUS Convert

Introduction

ViSUS convert is the tool that allows to convert data from different file formats (i.e. mainly image formats and raw) to IDX.

The supported formats are the ones supported by FreeImage.

In the following examples we will use the variable $CONVERT as the visus convert executable. You can set this variable using your OS commands like:

export CONVERT=/path/to/visus_executable

or

setenv CONVERT /path/to/visus_executable

The variable $RESOURCES is used instead to indicate a folder that contains the data to convert.

Some useful definitions:

• a box defines the extent of the domain where the samples are placed for each dimension
• the dims are the number of samples for each dimension
• the dtype (i.e. datatype) is defined as a combination of the number type and the number of components (e.g. an RGB vector uint8 with 3 components will be uint8[3], for single component a simple uint8 can be used)

Examples

Simple conversion

For very simple conversions, all you need is:

$CONVERT import $RESOURCES/cat_gray.tga create temp/cat_gray.idx

Get information

To *get information* about any file (including non-idx), use info

$CONVERT info $RESOURCES/cat_gray.tga
$CONVERT info $RESOURCES/cat_rgb.tga
$CONVERT info temp/cat_gray.idx
$CONVERT info temp/cat_rgb.idx

Create the IDX dataset

To explicitly create the idx volume

You can define a set of fields:

fields="scalar uint8 compressed + vector uint8[3] compressed"

and use the following to create the dataset

$CONVERT create temp/test_formats.idx --box "0 511 0 511" --fields "$fields" --time 0 0 time%03d/

Convert from a RAW dataset

With the following example we convert an array of int16 to IDX:

$CONVERT import data.raw --dtype 1*int16 --dims "995 501 401" export data.idx --dims "995 501 401" --field my_field --time 0 --box "0 994 0 500 0 400"

Or simply:

$CONVERT import data.raw --dtype 1*int16 --dims "995 501 401 1" create data.idx

Interleave multiple fields into one

With the following example we import 2 raw datasets into one interleaved IDX dataset:

$CONVERT --import /dev/null --dims "995 501 401 2" --dtype "int16" \
 --paste data0.raw --dtype 1*int16 --dims "995 501 401 1" --destination-box "0 994 0 500 0 400 1 1" --source-box "0 994 0 500 0 400" \
 --paste data1.raw --dtype 1*int16 --dims "995 501 401 1" --destination-box "0 994 0 500 0 400 0 0" --source-box "0 994 0 500 0 400" \
 --resize --dtype "int16[2]" --dims "995 501 401 1" --interleave --create data0_data1.idx

The new field created with this example will be a vector with 2 components.

Convert a stack of images

Very often microscopy data or MRI scans are simply big stacks of 2D images. Here is how you can easily convert them in a single IDX dataset using visus tools:

# path to your brand new IDX file
IDXFILE="G:/visus_dataset/2kbit1/visus.idx"

# box is specified as [x1 x2] [y1 y2] [z1 z2], min-max included (in this case the dataset has size 2048^3)
BOX="0 2047 0 2047 0 2047"

# change your data DTYPE (in this case a single Uint8)
DTYPE="uint8[1]"

$CONVERT create "$IDXFILE" --box "$BOX" --fields "data $DTYPE"

# example of conversion of png slices to IDX
for sliceid in $(seq -f "%04g" 0 2047)
do
  $CONVERT \
     import "G:/visus_dataset/2kbit1/png/$sliceid.png" \
     export "$IDXFILE" --box "0 2047 0 2047 $sliceid $sliceid" 
done

Tiled dataset

Multiple images can be combined (as tiles) to compose a panorama or volume (e.g. stack of images from an MRI scan).

This can be easily performed specifying the destination box of each image, as following:

$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "  0 255     0 255" 
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "256 511     0 255"
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "  0 255   256 511"
$CONVERT import $RESOURCES/cat_gray.tga export temp/test_formats.idx --field scalar --box "256 511   256 511"

Conversion from IDX to image

Conversion from IDX to image formats can be performed as well using the same technique, as following:

$CONVERT import temp/test_formats.idx --field scalar --box "  0 255     0 255"  export temp/test_formats_scalar.tga 
$CONVERT import temp/test_formats.idx --field scalar --box "256 511     0 255"  export temp/test_formats_scalar.jpg  
$CONVERT import temp/test_formats.idx --field scalar --box "  0 255   256 511"  export temp/test_formats_scalar.tif  
$CONVERT import temp/test_formats.idx --field scalar --box "256 511   256 511"  export temp/test_formats_scalar.bmp 
$CONVERT import temp/test_formats.idx --field scalar --box "128 383   128 383"  export temp/test_formats_scalar.png

Compression

You can compress your dataset (already created) using zip or lz4, as following:

$CONVERT compress-dataset your_dataset.idx lz4

You can also compress float values using the zfp lossy compression specifying the number of bits you want to store each sample, for example for 16 bits it will:

$CONVERT compress-dataset your_dataset.idx zfp-16

Note: The maximum number of bit planes to compress is the same as the bit-size of the type (e.g. 32 for float32, 64 for float64, 16 for for int16, etc).

When importing directly to a new dataset you can use:

$CONVERT import neon.tif create neon.idx --fields "data float32 default_compression(zfp-16) format(1)"
$CONVERT import neon.tif export neon.idx --box "0 999 0 999"

Note the "format(1)" part which is important to achieve good compression.

To decompress, use:

$CONVERT import neon.idx --box "0 999 0 999" export neon-out.tif

Also, RGB fields can be compressed with JPG instead of ZFP, with the following parameters:

$CONVERT import neon.tif create neon.idx --fields "data uint8[3] default_compression(jpg-JPEG_QUALITYSUPERB-JPEG_SUBSAMPLING_420-JPEG_OPTIMIZE) format(1)"
$CONVERT import neon.tif export neon.idx --box "0 999 0 999"