Skip to content

Commit

Permalink
Implemented starts_with()
Browse files Browse the repository at this point in the history
  • Loading branch information
hosseinmoein committed Jul 27, 2024
1 parent 2a1323a commit 6af066b
Show file tree
Hide file tree
Showing 9 changed files with 437 additions and 20 deletions.
49 changes: 33 additions & 16 deletions docs/HTML/DataFrame.html
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@
</UL>
</UL>

<H2 ID="1"><font color="blue">Summary</font></H2>
<H2 ID="1"><font color="blue">Summary <font size="+3">&#9730;</font></font></H2>
<P>
<font size="+1">DataFrame</font> is a templatized and heterogeneous C++ container designed for data analysis for statistical, machine-learning, or financial applications. You can think of data-frame as a two-dimensional data structure of columns and rows just like an Excel spreadsheet, or a SQL table. But in case of C++ DataFrame, your data needn't be two-dimensional necessarily. Columns in the C++ DataFrame could be vectors of any type, including DataFrames or other containers. So, a C++ DataFrame can be of any dimension. That's the logical layout of the data. C++ DataFrame also includes an intuitive API for data analysis and analytics. The API is designed to be open-ended meaning you can easily include your own custom algorithms.<BR>
Any data-frame inherently includes a schema. C++ DataFrame schema is either built dynamically at run-time or it comes from a file. Currently C++ DataFrame could be shared between different nodes (e.g. computers) in a couple of ways. It can be written into a file, or it can be serialized into a buffer and sent across and reconstituted on the other side.
Expand Down Expand Up @@ -136,7 +136,7 @@ <H2 ID="1"><font color="blue">Summary</font></H2>

<BR><HR COLOR="Gray" SIZE="5">

<H2 ID="2"><font color="blue">API Reference with code samples</font></H2>
<H2 ID="2"><font color="blue">API Reference with code samples <font size="+3">&#128477;</font></font></H2>
DataFrame library interface is separated into two main categories:
<OL>
<LI>Accessing, adding, slicing &amp; dicing, joining &amp; groupby'ing ... <B>(The first column in the table below)</B></LI>
Expand Down Expand Up @@ -488,6 +488,10 @@ <H2 ID="2"><font color="blue">API Reference with code samples</font></H2>
<td title="Sorts DataFrame"><a href="https://htmlpreview.github.io/?https://github.com/hosseinmoein/DataFrame/blob/master/docs/HTML/sort.html">sort( 5 )<BR>sort_async( 5 )</a></td>
</tr>

<tr class="item" onmouseover="this.style.backgroundColor='#ffff66';" onmouseout="this.style.backgroundColor='#d4e3e5';">
<td title="Returns a vector of booleans indicating if values start with the given pattern"><a href="https://htmlpreview.github.io/?https://github.com/hosseinmoein/DataFrame/blob/master/docs/HTML/starts_with.html">starts_with( )</a></td>
</tr>

<tr class="item" onmouseover="this.style.backgroundColor='#ffff66';" onmouseout="this.style.backgroundColor='#d4e3e5';">
<td title="Swaps data between two DataFrames"><a href="https://htmlpreview.github.io/?https://github.com/hosseinmoein/DataFrame/blob/master/docs/HTML/remove_column.html">swap</a>()</td>
</tr>
Expand Down Expand Up @@ -1487,7 +1491,7 @@ <H2 ID="2"><font color="blue">API Reference with code samples</font></H2>

<BR><HR COLOR="Gray" SIZE="5">

<H2 ID="3"><font color="blue">Multithreading</font></H2>
<H2 ID="3"><font color="blue">Multithreading <font size="+3">&#128640;</font></font></H2>
In general, multithreading could be very unintuitive. Often you think by using multithreading you enhance the performance of your program. But in fact, you are hindering it. To do effective multithreading, you must do two things repeatedly; measure and adjust. In general (rule of thumb), you should use multithreading in two contradictory situations. First, when you have intensive CPU-bound operations like mathematical equations that can independently utilize different cores. Second, when you have multiple I/O-bound operations that can go on independently while they wait for each other. The key word here is <I>independently</I>. You must also realize that multithreading has an inherent overhead that not only affects your process but also other processes running on the same node. It is recommended to start with a single-threaded version and when that is <I>working correctly</I>, establish a baseline, take measurements, and implement a multithreaded solution.<BR>
DataFrame uses multithreading extensively and provides granular tools to adjust your environment. Let's divide the multithreading subject in DataFrame into two categories:<BR>

Expand All @@ -1508,7 +1512,7 @@ <H4>2. DataFrame Internal Multithreading</H4>

<BR><HR COLOR="Gray" SIZE="5">

<H2 ID="4"><font color="blue">Views</font></H2>
<H2 ID="4"><font color="blue">Views <font size="+3">&#127749;</font></font></H2>
<P>
Views have useful and practical use-cases. A view is a slice of a DataFrame that is a reference to the original DataFrame. It appears exactly the same as a DataFrame, but if you modify any data in the view, the corresponding data point(s) in the original DataFrame will also be modified and vice versa. There are certain things you cannot do in views. For example, you cannot add or delete columns, extend the index column, ...<BR><BR>

Expand All @@ -1520,7 +1524,7 @@ <H2 ID="4"><font color="blue">Views</font></H2>

Why would you use views
<UL>
<LI>To run algorithms/slicing on subsets of data, <I>without copying</I> data</LI>
<LI>To run algorithms/slicing on smaller subsets of data, <I>without copying</I> data</LI>
<LI>To mix and compare different data subsets, <I>without copying</I> data</LI>
<LI>To have one source of truth, while having different datasets <I>without copying</I> data </LI>
</UL>
Expand All @@ -1530,7 +1534,7 @@ <H2 ID="4"><font color="blue">Views</font></H2>

<BR><HR COLOR="Gray" SIZE="5">

<H2 ID="5"><font color="blue">Visitors</font></H2>
<H2 ID="5"><font color="blue">Visitors <font size="+3">&#128109;</font></font></H2>
<P>
Visitors are the main mechanism to implement analytical (i.e. statistical, financial, machine-learning) algorithms. You can easily follow the visitor's interface to add your custom algorithm by which you will extend the DataFrame package. Visitors also play several roles that in other packages maybe handled by separate interfaces. Visitors play the role of <I>apply</I>, <I>transformer</I>, and <I>algorithms</I>. For example, a visitor can transform column(s) or it may take the column(s) as read-only and implement an algorithm.<BR>
There are two visitor interfaces:<BR>
Expand Down Expand Up @@ -1559,7 +1563,7 @@ <H2 ID="5"><font color="blue">Visitors</font></H2>

<BR><HR COLOR="Gray" SIZE="5">

<H2 ID="6"><font color="blue">Memory Alignment</font></H2>
<H2 ID="6"><font color="blue">Memory Alignment <font size="+3">&#10803;</font></font></H2>
<P>
DataFrame gives you the ability to allocate memory on custom alignment boundaries.<BR>
You can use this feature to take advantage of <I>SIMD</I> instructions in modern CPU's. Since DataFrame algorithms are all done on vectors of data &#8212; columns, this can come handy in conjunction with compiler optimizations. Also, you can use alignment to prevent false cache-line sharing between multiple columns.<BR>
Expand All @@ -1569,38 +1573,51 @@ <H2 ID="6"><font color="blue">Memory Alignment</font></H2>

<BR><HR COLOR="Gray" SIZE="5">

<H2 ID="7"><font color="blue">Numeric Generators</font></H2>
<H2 ID="7"><font color="blue">Numeric Generators <font size="+3">&Sum;</font></font></H2>
<P>
Random generators, and a few other numeric generators, were added as a series of convenient stand-alone functions to generate random numbers (it covers all C++ standard distributions). You can seamlessly use these routines to generate random DataFrame columns.<BR>
Random generators, and a few other numeric generators, were added as a series of convenient stand-alone functions to generate random numbers (it covers all C++ standard distributions). You can seamlessly use these routines to generate random DataFrame columns. The result vectors are space-optimized and and you can choose different memory alignments.<BR>
See this document and file <I>RandGen.h</I> and <I>dataframe_tester.cc.</I>
For the definition and defaults of <I>RandGenParams</I>, see this document and file <I>DataFrameTypes.h</I>
</P>

<BR><HR COLOR="Gray" SIZE="5">

<H2 ID="8"><font color="blue">Code Structure</font></H2>
<H2 ID="8"><font color="blue">Code Structure <font size="+3">&#128193;</font></font></H2>
<P>
The DataFrame library is <I>almost</I> a header-only library. Currently the only library source file is <I>DateTime.cc.</I><BR>
<BR>
Starting from the root directory:<BR>
<BR>
<font size="+1"><I>include</I></font> directory contains almost all of the code. It includes <I>.h</I> and <I>.tcc</I> files. The latter are C++ template code files (they are mostly located in the <I>Internals</I> subdirectory). The main header file is <I>DataFrame.h.</I> It contains the DataFrame class and its public interface. There are comprehensive comments for each public interface call in that file. The rest of the files will show you how the sausage is made. <I>Include</I> directory also contains subdirectories that contain mostly internal DataFrame implementation. One exception is the <I>Utils</I> subdirectory<BR>
<font size="+1"><I>include/DataFrame</I></font> subdirectory contains almost all of the code. It includes <I>.h</I> and <I>.tcc</I> files. The latter are C++ template code files (they are mostly located in the <I>Internals</I> subdirectory). The main header file is <I>DataFrame.h.</I> It contains the DataFrame class and its public interface. There are comprehensive comments for each public interface call in that file. The rest of the files will show you how the sausage is made. <I>Include</I> directory also contains subdirectories that contain mostly internal DataFrame implementation. One exception is the <I>Utils</I> subdirectory<BR>
<BR>
<font size="+1"><I>src</I></font> directory contains Linux-only make files and <I>Utils</I> subdirectory.<BR>
<BR>
<font size="+1"><I>test</I></font> directory contains all the test source files, mocked data files, and test output files. The main test source files are <I>dataframe_tester.cc</I> and <I>dataframe_tester_2.cc</I>. It contains test cases for all functionalities of DataFrame. It is not in a very organized structure. I plan to make the test cases more organized.
<font size="+1"><I>test</I></font> directory contains all the test source files. Mocked data files are in the <I>data</I> subdirectory. The main test source files are <I>dataframe_tester.cc</I>, <I>dataframe_tester_2.cc</I>, and <I>dataframe_tester_3.cc</I>. It contains test cases for all functionalities of DataFrame. <I>benchmarks</I> subdirectory contains a bunch of benchmarks. All documnetations are under <I>docs</I> subdirectory.
</P>

<BR><HR COLOR="Gray" SIZE="5">

<H2 ID="9"><font color="blue">Build Instructions</font></H2>
<H2 ID="9"><font color="blue">Build Instructions <font size="+3">&#x1F6E0;</font></font></H2>
<P>
<font size="+1"><B>Using plain make and make-files:</B></font><BR>
Go to the <I>src</I> subdirectory, and execute build_all.sh. This will build the library and test executables for <I>Linux/Unix flavors only</I><BR>
Go to the <I>src</I> subdirectory, and execute build_all.sh. This will build the library and test executables for <I>Linux/Unix flavors only</I><BR><BR>
<font size="+1"><B>Using CMake:</B></font><BR>
You would be able to build this in Linux, Windows, MacOS, and more &#8212; see README<BR>
<PRE>
mkdir [Debug|Release]
cd [Debug|Release]

cmake -DCMAKE_BUILD_TYPE=Release -DHMDF_BENCHMARKS=1 -DHMDF_EXAMPLES=1 -DHMDF_TESTING=1 ..
cmake -DCMAKE_BUILD_TYPE=Debug -DHMDF_SANITY_EXCEPTIONS=1 -DHMDF_BENCHMARKS=1 -DHMDF_EXAMPLES=1 -DHMDF_TESTING=1 ..

make
make install

cd [Debug|Release]
make uninstall
</PRE>
<font size="+1"><B>Using Package Managers:</B></font><BR>
You can also use <I>Conan</I> or <I>VCPKG</I> &#8212; see README
DataFrame is available on <a href="https://conan.io/center/recipes/dataframe"<I>Conan</I></a> platform. See the <a href="https://docs.conan.io/en/latest/"<I>Conan docs</I></a> for more information.<BR>
DataFrame is also available on <a href="https://vcpkg.link/ports/dataframe"<I>VCPKG</I></a> platform. See <a href="https://learn.microsoft.com/en-us/vcpkg/"<I>VCPKG docs</i></a> for more information<BR>
</P>

<BR><a href="https://github.com/hosseinmoein/DataFrame?tab=readme-ov-file">&#8592; Back to Github</a><BR>
Expand Down
2 changes: 1 addition & 1 deletion docs/HTML/copying_data.html
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@

<body style="font-family: Georgia, serif">

Copying data obviously causes space and time being wasted. But it also could have other side effects. For example, the construction and/or destruction of objects being copied may have side effects. Also, some object may be consumers of expensive resources and copying them may increase resource consumption.
Copying data obviously causes space and time being wasted. But it also could have other side effects. For example, the construction and/or destruction of objects being copied may have side effects. Also, some object may be consumers of expensive resources and copying them may increase resource consumption. DataFrame <I>Views</I> play an important role in this regard.
<BR>
<BR>
<BR>
Expand Down
Loading

0 comments on commit 6af066b

Please sign in to comment.