MIDX examples

MIDX is an XML metadata file which defines how multiple IDX files are combined in on single dataset. This, for example, allows dynamic blending of the content of different volumes or also to perform operations (e.g., scripting) using different fields from different data sources.

Affine transformation of a dataset

<dataset typename='IdxMultipleDataset'>

  <dataset url='file://$(CurrentFileDirectory)/datasets/visus.idx' name='first'>
    <M value="2.5 0.0 0.0 0.0 0.0 1.875 0.0 0.0 0.0 0.0 1 0.0 0.0 0.0 0.0 1.0"/>
  </dataset>
  
</dataset>

Two overlapping datasets

<dataset typename='IdxMultipleDataset'>

  <dataset url='file://$(CurrentFileDirectory)/../2kbit1/zip/rowmajor/visus.idx' name='first'>
    <M><translate x='0' y='0' z='0'/></M>
  </dataset>

  <dataset url='file://$(CurrentFileDirectory)/../2kbit1/zip/rowmajor/visus.idx' name='second'>
    <M><translate x='0' y='0' z='0'/></M>
  </dataset>
  
</dataset>

Two datasets side by side:

<dataset typename='IdxMultipleDataset'>

  <dataset url='file://$(CurrentFileDirectory)/../male/visus.idx' name='first'>
    <M>
      <translate x='0' y='0' z='0'/>
      <scale     x='1' y='1' z='3'/>
    </M>
  </dataset>

  <dataset url='file://$(CurrentFileDirectory)/../male/visus.idx' name='second'>
    <M>
      <translate x='2048' y='0' z='0'/>
      <scale     x='1'    y='1' z='3'/>
    </M>
  </dataset>
  
</dataset>

Datasets overlapping with time query

<dataset typename='IdxMultipleDataset'>

  <dataset url='file://$(CurrentFileDirectory)/../2kbit1/zip/rowmajor/visus.idx' name='first'>
    <M><translate x='0' y='0' z='0'/></M>
  </dataset>

  <dataset url='file://$(CurrentFileDirectory)/../borg/visus.idx?time=13' name='second'>
    <M><translate x='0' y='0' z='0'/></M>
  </dataset>
  
</dataset>

Blending multiple timesteps

<dataset typename='IdxMultipleDataset'>
  <dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=0' name='0' />
  <dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=1' name='1' />
  <dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=2' name='2' />
  <dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=3' name='3' />
</dataset>

Local MIDX. No caching

<dataset name="Example" url="file://./docs/examples/midx/visus.midx">
  <access name="MultipleAccess" type="MultipleAccess">
    <A type='disk' chmod='r' url='file://./docs/examples/midx/A/visus.idx' />
    <B type='disk' chmod='r' url='file://./docs/examples/midx/B/visus.idx' />
    <C type='disk' chmod='r' url='file://./docs/examples/midx/C/visus.idx' />
    <D type='disk' chmod='r' url='file://./docs/examples/midx/D/visus.idx' />
  </access>  
</dataset>  

Local MIDX. Local IDX block caching

Note that is useless to create two local copies of child datasets. It would make more sense if you had remote datasets.

<dataset name="Example" url="file://./docs/examples/midx/visus.midx">
  <access name="MultipleAccess" type="MultipleAccess">
    <A type='multiplex'>
      <access type='disk' chmod='rw' url="file://$(VisusCacheDirectory)/A/visus.idx" />
      <access type='disk' chmod='r'  url='file://./docs/examples/midx/A/visus.idx' />
    </A>
    <B type='multiplex'>
      <access type='disk' chmod='rw' url="file://$(VisusCacheDirectory)/B/visus.idx" />
      <access type='disk' chmod='r'  url='file://./docs/examples/midx/B/visus.idx' />
    </B>
    <C type='multiplex'>
      <access type='disk' chmod='rw' url="file://$(VisusCacheDirectory)/C/visus.idx" />
      <access type='disk' chmod='r'  url='file://./docs/examples/midx/C/visus.idx' />
    </C>
    <D type='multiplex'>
      <access type='disk' chmod='rw' url="file://$(VisusCacheDirectory)/D/visus.idx" />
      <access type='disk' chmod='r'  url='file://./docs/examples/midx/D/visus.idx' />
    </D>
  </access> 
</dataset>   

Local MIDX. Local MIDX block caching

All datasets must have the same IDX bitmask. Otherwise it won't work.

<dataset name="Example" url="file://./docs/examples/midx/visus.midx">
    
  <access name="DefaultAccess" type="multiplex" >
        
    <!-- here I do the up-midx caching -->
      <access type="DiskAccess" chmod='rw'  dir="./cached" compression="zip" />
        
    <!-- child access -->
    <access name="MultipleAccess" type="MultipleAccess">
      <A type='disk' chmod='r' url='file://./docs/examples/midx/A/visus.idx' />
      <B type='disk' chmod='r' url='file://./docs/examples/midx/B/visus.idx' />
      <C type='disk' chmod='r' url='file://./docs/examples/midx/C/visus.idx' />
      <D type='disk' chmod='r' url='file://./docs/examples/midx/D/visus.idx' />
    </access> 
  </access> 
</dataset>

Remote MIDX. Local MIDX block caching

Example for remote MIDX pin2d on atlantis:

<dataset typename="IdxMultipleDataset" url="file:///usr/sci/cedmav/LDAV/midx/pin2d.midx">
  <dataset name="A" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/A" />
  <dataset name="B" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/B" />
  <dataset name="C" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/C" />
  <dataset name="D" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/D" />
  <dataset name="E" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/E" />
</dataset>

The MIDX processing is done server side (example: fieldname is 'avg'). The client caches the MIDX blocks got from server.

<dataset name="Example" url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d">
  <access name="DiskAccess"  type='multiplex'>
    <access name="Example" type="DiskAccess" dir="./cached" compression="zip" />
    <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d" />
  </access> 
</dataset>   

Remote MIDX. Local IDX block caching

Example for remote MIDX pin2d on atlantis:

<dataset typename="IdxMultipleDataset" url="file:///usr/sci/cedmav/LDAV/midx/pin2d.midx">
  <dataset name="A" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/A" />
  <dataset name="B" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/B" />
  <dataset name="C" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/C" />
  <dataset name="D" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/D" />
  <dataset name="E" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/E" />
</dataset>

The server serves child IDX datasets. The child caches the child IDX datasets. The MIDX processing is done client side.

<dataset name="Example" url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d" >
 <access name="MultipleAccess" type="MultipleAccess">
  <A type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/A/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/A" />
  </A>
  <B type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/B/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/B" />
  </B>
  <C type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/C/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/C" />
  </C>
  <D type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/D/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/D" />
  </D>
  <E type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/E/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/E" />
  </E>  
 </access>
</dataset>