Write and read data with PIDX - Revision history

Admin: /* How to read */

2018-05-29T22:52:45Z

‎How to read

Admin: /* How to write */

2018-05-29T22:51:22Z

‎How to write

Admin: /* How to write */

2018-05-29T22:50:57Z

‎How to write

Admin: /* How to write */

2018-05-29T22:50:04Z

‎How to write

Admin: /* How to write */

2018-05-29T22:49:50Z

‎How to write

Admin: /* How to write */

2018-05-29T22:48:30Z

‎How to write

Admin: /* How to write */

2018-05-29T22:47:47Z

‎How to write

Admin: /* How to write */

2018-05-29T22:47:11Z

‎How to write

Admin at 22:46, 29 May 2018

2018-05-29T22:46:46Z

Admin at 20:51, 31 July 2017

2017-07-31T20:51:52Z

@@ Line 84: / Line 84: @@
 After the initialization the first step is to know how many variables are in the dataset, we can do this using:
-<pre>PIDX_get_variable_count(file, &amp;variable_count);</pre>
+<syntaxhighlight lang="C++">
 With this information we can try to gather more information about a specific variable, in the following case the variable number ''variable_index'':
-<pre>PIDX_set_current_variable_index(file, variable_index);
+<syntaxhighlight lang="C++">
 PIDX_get_current_variable(file, &amp;variable);
-PIDX_values_per_datatype(variable-&gt;type_name, &amp;values_per_sample, &amp;bits_per_sample);</pre>
+PIDX_values_per_datatype(variable-&gt;type_name, &amp;values_per_sample, &amp;bits_per_sample);
 Using these functions we retrieved alse the type name, the number of values per sample and the number of bits per sample.
 Now it is time to read the data using:
-<pre>PIDX_variable_read_data_layout(variable, local_offset, local_size, data, PIDX_row_major);</pre>
+<syntaxhighlight lang="C++">
 Also in this case we need to indicate the per-process local box information (i.e. local offset and local size) and the array where we want to store the data of this variable. When the <code>PIDX_close(file)</code> will be called the data of this variable will be copied to our local ''data'' array in row major order.
 Example source: [https://github.com/sci-visus/PIDX/blob/master/examples/basic-io/idx_read.c idx_read.c]

@@ Line 48: / Line 48: @@
 and the current '''number of variables''' that we want to write, as following:
-<pre>PIDX_set_variable_count(file, variable_count);</pre>
+<syntaxhighlight lang="C++">PIDX_set_variable_count(file, variable_count);</syntaxhighlight>
 We are now ready to define these variables. A variable in PIDX is defined by: - a name - a number of bits, e.g. 32 for a 32 bit integer variable - a typename, e.g. &quot;1*int32&quot;

@@ Line 19: / Line 19: @@
 PIDX uses a ''struct'' called <code>PIDX_point</code> to represent a value in n-dimensional space. We can use this ''struct'' to '''define the global domain size, and the local box information'''.
-<syntaxhighlight lang="javascript">
+<syntaxhighlight lang="C++">
 PIDX_set_point(global_size, global_box_size[X], global_box_size[Y], global_box_size[Z]);
 PIDX_set_point(local_offset, local_box_offset[X], local_box_offset[Y], local_box_offset[Z]);
@@ Line 55: / Line 55: @@
 With these information we can '''create our variables''' as following:
-<pre>PIDX_variable_create(var_name[var], bpv[var] * vps[var], type_name[var], &amp;variable[var]);</pre>
+<syntaxhighlight lang="C++">
 where ''variable'' is an array of <code>PIDX_variable</code>.
 We want to tell to PIDX where this variable's data are in memory in the per-process '''local domain''', we do this as following:
-<pre>PIDX_variable_write_data_layout(variable[var], local_offset, local_size, data[var], PIDX_row_major);</pre>
+<syntaxhighlight lang="C++">
 where ''data'' is a ''char'' pointer to the data array (of the specific variable) and the last parameter (<code>PIDX_row_major</code>) indicates the order of the data in this array (e.g. row major).
 Finally we can tell PIDX to submit this '''variable for the parallel write''', we do this using:
-<pre>PIDX_append_and_write_variable(file, variable[var]);</pre>
+<syntaxhighlight lang="C++">
 For performance improvement PIDX tries to gather data from multiple variables and dump them to disk at once only when the function <code>PIDX_close(file)</code> is called.

@@ Line 19: / Line 19: @@
 PIDX uses a ''struct'' called <code>PIDX_point</code> to represent a value in n-dimensional space. We can use this ''struct'' to '''define the global domain size, and the local box information'''.
-<syntaxhighlight lang="C++">
+<syntaxhighlight lang="javascript">
 PIDX_set_point(global_size, global_box_size[X], global_box_size[Y], global_box_size[Z]);
 PIDX_set_point(local_offset, local_box_offset[X], local_box_offset[Y], local_box_offset[Z]);

@@ Line 34: / Line 34: @@
 Now it is time to '''create the file''' using:
-<pre>PIDX_file file;
+<syntaxhighlight lang="C++">
-PIDX_file_create(output_file_name, PIDX_MODE_CREATE, p_access, global_size, &amp;file);</pre>
+PIDX_file file;
 The parameters that we use are: - ''output_file_name'', the name of the file that we want to create - ''file_access_mode'', in this case we create the file - ''pidx_access'', the access configuration that we defined - ''global_size'', the global domain size as we defined - ''file'', the PIDX_file struct that we are creating
 Other general settings that we want to do are '''setting the current timestep''' (i.e. logical time)using:
-<pre>PIDX_set_current_time_step(file, ts);</pre>
+<syntaxhighlight lang="C++">
 and the current '''number of variables''' that we want to write, as following:

@@ Line 19: / Line 19: @@
 PIDX uses a ''struct'' called <code>PIDX_point</code> to represent a value in n-dimensional space. We can use this ''struct'' to '''define the global domain size, and the local box information'''.
-<syntaxhighlight lang="cpp">
+<syntaxhighlight lang="c">
 PIDX_set_point(global_size, global_box_size[X], global_box_size[Y], global_box_size[Z]);
 PIDX_set_point(local_offset, local_box_offset[X], local_box_offset[Y], local_box_offset[Z]);
@@ Line 27: / Line 27: @@
 The next step is to '''create an access''' to tell PIDX how we want to access the data, in this case in parallel. We can do this as following:
-<syntaxhighlight lang="cpp">
+<syntaxhighlight lang="c">
 PIDX_create_access(&p_access);
 PIDX_set_mpi_access(p_access, MPI_COMM_WORLD);

@@ Line 19: / Line 19: @@
 PIDX uses a ''struct'' called <code>PIDX_point</code> to represent a value in n-dimensional space. We can use this ''struct'' to '''define the global domain size, and the local box information'''.
-<pre>PIDX_set_point(global_size, global_box_size[X], global_box_size[Y], global_box_size[Z]);
+<syntaxhighlight lang="c++">
 PIDX_set_point(local_offset, local_box_offset[X], local_box_offset[Y], local_box_offset[Z]);
-PIDX_set_point(local_size, local_box_size[X], local_box_size[Y], local_box_size[Z]);</pre>
+PIDX_set_point(local_size, local_box_size[X], local_box_size[Y], local_box_size[Z]);
 The next step is to '''create an access''' to tell PIDX how we want to access the data, in this case in parallel. We can do this as following:
-<pre>PIDX_create_access(&amp;p_access);
+<syntaxhighlight lang="c++">
-PIDX_set_mpi_access(p_access, MPI_COMM_WORLD);</pre>
+PIDX_create_access(&amp;p_access);
 Now it is time to '''create the file''' using:

@@ Line 1: / Line 1: @@
-= Introduction =
+== Introduction ==
 In these examples we describe how to write and read data using the PIDX library. We consider a global 3D regular grid domain that we will call global domain (g). This global domain represents the grid space where all the data resides.
@@ Line 13: / Line 13: @@
 If you want to write and read data in PIDX you just need to follow a few simple steps: - identify the per-process local domain - create a <code>PIDX_access</code>, it defines how you want to access the data (serial or parallel) - create or access a <code>PIDX_file</code>, it defines the dataset - define or read one or more <code>PIDX_variable</code> - write/read the variables
-= How to write =
+== How to write ==
 The data that we want to write from each processor can be considered as a ''box'' in the global domain. This ''box'' is represented by and ''offset'' (i.e. the position in the global domain) and a ''size'' (i.e. the number of elements in the box).
@@ Line 61: / Line 61: @@
 Example source: [https://github.com/sci-visus/PIDX/blob/master/examples/basic-io/idx_write.c idx_write.c]
-= How to read =
+== How to read ==
 To read data written with PIDX we follow the same initialization process to create the <code>PIDX_access</code> and <code>PIDX_file</code> that we saw in the previous section.