Difference between revisions of "MIDX examples"

From
Jump to: navigation, search
(→‎Two visible males side by side:)
(→‎Affine transformation of a dataset)
 
(7 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
MIDX is an XML metadata file which defines how multiple IDX files are combined in on single dataset. This, for example, allows dynamic blending of the content of different volumes or also to perform operations (e.g., scripting) using different fields from different data sources.
 
MIDX is an XML metadata file which defines how multiple IDX files are combined in on single dataset. This, for example, allows dynamic blending of the content of different volumes or also to perform operations (e.g., scripting) using different fields from different data sources.
 +
 +
=== Affine transformation of a dataset ===
 +
 +
<pre>
 +
&lt;dataset typename='IdxMultipleDataset'&gt;
 +
 +
  &lt;dataset url='file://$(CurrentFileDirectory)/datasets/visus.idx' name='first'&gt;
 +
    <M value="2.5 0.0 0.0 0.0 0.0 1.875 0.0 0.0 0.0 0.0 1 0.0 0.0 0.0 0.0 1.0"/>
 +
  &lt;/dataset&gt;
 +
 
 +
&lt;/dataset&gt;</pre>
  
 
=== Two overlapping datasets ===
 
=== Two overlapping datasets ===
Line 36: Line 47:
 
</pre>
 
</pre>
  
−
=== 2kbit1 overlapping Borg?time=13 ===
+
=== Datasets overlapping with time query ===
  
 
<pre>&lt;dataset typename='IdxMultipleDataset'&gt;
 
<pre>&lt;dataset typename='IdxMultipleDataset'&gt;
Line 49: Line 60:
 
    
 
    
 
&lt;/dataset&gt;</pre>
 
&lt;/dataset&gt;</pre>
−
=== Blend David timesteps ===
+
 
 +
=== Blending multiple timesteps ===
  
 
<pre>&lt;dataset typename='IdxMultipleDataset'&gt;
 
<pre>&lt;dataset typename='IdxMultipleDataset'&gt;
Line 57: Line 69:
 
   &lt;dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=3' name='3' /&gt;
 
   &lt;dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=3' name='3' /&gt;
 
&lt;/dataset&gt;</pre>
 
&lt;/dataset&gt;</pre>
 +
 
=== Local MIDX. No caching ===
 
=== Local MIDX. No caching ===
  

Latest revision as of 16:11, 21 September 2018

MIDX is an XML metadata file which defines how multiple IDX files are combined in on single dataset. This, for example, allows dynamic blending of the content of different volumes or also to perform operations (e.g., scripting) using different fields from different data sources.

Affine transformation of a dataset

<dataset typename='IdxMultipleDataset'>

  <dataset url='file://$(CurrentFileDirectory)/datasets/visus.idx' name='first'>
    <M value="2.5 0.0 0.0 0.0 0.0 1.875 0.0 0.0 0.0 0.0 1 0.0 0.0 0.0 0.0 1.0"/>
  </dataset>
  
</dataset>

Two overlapping datasets

<dataset typename='IdxMultipleDataset'>

  <dataset url='file://$(CurrentFileDirectory)/../2kbit1/zip/rowmajor/visus.idx' name='first'>
    <M><translate x='0' y='0' z='0'/></M>
  </dataset>

  <dataset url='file://$(CurrentFileDirectory)/../2kbit1/zip/rowmajor/visus.idx' name='second'>
    <M><translate x='0' y='0' z='0'/></M>
  </dataset>
  
</dataset>

Two datasets side by side:

<dataset typename='IdxMultipleDataset'>

  <dataset url='file://$(CurrentFileDirectory)/../male/visus.idx' name='first'>
    <M>
      <translate x='0' y='0' z='0'/>
      <scale     x='1' y='1' z='3'/>
    </M>
  </dataset>

  <dataset url='file://$(CurrentFileDirectory)/../male/visus.idx' name='second'>
    <M>
      <translate x='2048' y='0' z='0'/>
      <scale     x='1'    y='1' z='3'/>
    </M>
  </dataset>
  
</dataset>

Datasets overlapping with time query

<dataset typename='IdxMultipleDataset'>

  <dataset url='file://$(CurrentFileDirectory)/../2kbit1/zip/rowmajor/visus.idx' name='first'>
    <M><translate x='0' y='0' z='0'/></M>
  </dataset>

  <dataset url='file://$(CurrentFileDirectory)/../borg/visus.idx?time=13' name='second'>
    <M><translate x='0' y='0' z='0'/></M>
  </dataset>
  
</dataset>

Blending multiple timesteps

<dataset typename='IdxMultipleDataset'>
  <dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=0' name='0' />
  <dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=1' name='1' />
  <dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=2' name='2' />
  <dataset url='file://$(CurrentFileDirectory)/../david_subsampled/visus.idx?time=3' name='3' />
</dataset>

Local MIDX. No caching

<dataset name="Example" url="file://./docs/examples/midx/visus.midx">
  <access name="MultipleAccess" type="MultipleAccess">
    <A type='disk' chmod='r' url='file://./docs/examples/midx/A/visus.idx' />
    <B type='disk' chmod='r' url='file://./docs/examples/midx/B/visus.idx' />
    <C type='disk' chmod='r' url='file://./docs/examples/midx/C/visus.idx' />
    <D type='disk' chmod='r' url='file://./docs/examples/midx/D/visus.idx' />
  </access>  
</dataset>

Local MIDX. Local IDX block caching

Note that is useless to create two local copies of child datasets. It would make more sense if you had remote datasets. 

<dataset name="Example" url="file://./docs/examples/midx/visus.midx">
  <access name="MultipleAccess" type="MultipleAccess">
    <A type='multiplex'>
      <access type='disk' chmod='rw' url="file://$(VisusCacheDirectory)/A/visus.idx" />
      <access type='disk' chmod='r'  url='file://./docs/examples/midx/A/visus.idx' />
    </A>
    <B type='multiplex'>
      <access type='disk' chmod='rw' url="file://$(VisusCacheDirectory)/B/visus.idx" />
      <access type='disk' chmod='r'  url='file://./docs/examples/midx/B/visus.idx' />
    </B>
    <C type='multiplex'>
      <access type='disk' chmod='rw' url="file://$(VisusCacheDirectory)/C/visus.idx" />
      <access type='disk' chmod='r'  url='file://./docs/examples/midx/C/visus.idx' />
    </C>
    <D type='multiplex'>
      <access type='disk' chmod='rw' url="file://$(VisusCacheDirectory)/D/visus.idx" />
      <access type='disk' chmod='r'  url='file://./docs/examples/midx/D/visus.idx' />
    </D>
  </access> 
</dataset>

Local MIDX. Local MIDX block caching

All datasets must have the same IDX bitmask. Otherwise it won't work. 

<dataset name="Example" url="file://./docs/examples/midx/visus.midx">
    
  <access name="DefaultAccess" type="multiplex" >
        
    <!-- here I do the up-midx caching -->
      <access type="DiskAccess" chmod='rw'  dir="./cached" compression="zip" />
        
    <!-- child access -->
    <access name="MultipleAccess" type="MultipleAccess">
      <A type='disk' chmod='r' url='file://./docs/examples/midx/A/visus.idx' />
      <B type='disk' chmod='r' url='file://./docs/examples/midx/B/visus.idx' />
      <C type='disk' chmod='r' url='file://./docs/examples/midx/C/visus.idx' />
      <D type='disk' chmod='r' url='file://./docs/examples/midx/D/visus.idx' />
    </access> 
  </access> 
</dataset>

Remote MIDX. Local MIDX block caching

Example for remote MIDX pin2d on atlantis: 

<dataset typename="IdxMultipleDataset" url="file:///usr/sci/cedmav/LDAV/midx/pin2d.midx">
  <dataset name="A" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/A" />
  <dataset name="B" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/B" />
  <dataset name="C" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/C" />
  <dataset name="D" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/D" />
  <dataset name="E" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/E" />
</dataset>

The MIDX processing is done server side (example: fieldname is 'avg'). The client caches the MIDX blocks got from server. 

<dataset name="Example" url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d">
  <access name="DiskAccess"  type='multiplex'>
    <access name="Example" type="DiskAccess" dir="./cached" compression="zip" />
    <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d" />
  </access> 
</dataset>

Remote MIDX. Local IDX block caching

Example for remote MIDX pin2d on atlantis: 

<dataset typename="IdxMultipleDataset" url="file:///usr/sci/cedmav/LDAV/midx/pin2d.midx">
  <dataset name="A" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/A" />
  <dataset name="B" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/B" />
  <dataset name="C" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/C" />
  <dataset name="D" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/D" />
  <dataset name="E" url="$(protocol)://$(hostname):$(port)/mod_visus?action=readdataset&amp;dataset=pin2d/E" />
</dataset>

The server serves child IDX datasets. The child caches the child IDX datasets. The MIDX processing is done client side. 

<dataset name="Example" url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d" >
 <access name="MultipleAccess" type="MultipleAccess">
  <A type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/A/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/A" />
  </A>
  <B type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/B/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/B" />
  </B>
  <C type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/C/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/C" />
  </C>
  <D type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/D/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/D" />
  </D>
  <E type='multiplex'>
   <access type='disk' chmod='rw' url='file://./cached/E/visus.idx'  />
   <access type='network' chmod='r' compression='zip' url="http://atlantis.sci.utah.edu/mod_visus?dataset=pin2d/E" />
  </E>  
 </access>
</dataset>
Retrieved from "http://wiki.visus.org/index.php?title=MIDX_examples&oldid=365"