README.md

June 9, 2023 · View on GitHub

Highlights

Search recursively for a regex pattern using Intel Hyperscan.
When a git repository is detected, the repository index is searched using libgit2.
Similar to grep, ripgrep, ugrep, The Silver Searcher etc.
C++17, Multi-threading, SIMD.
USAGE GUIDE
Implementation notes here.
Not cross-platform. Tested in Linux.

Performance

The following tests compare the performance of hypergrep against:

ripgrep v13.0.0
ag 2.2.0 (The Silver Searcher) v2.2.0
ugrep v3.11.2

System Details

Type	Value
Processor	11th Gen Intel(R) Core(TM) i9-11900KF @ 3.50GHz 3.50 GHz
Instruction Set Extensions	Intel® SSE4.1, Intel® SSE4.2, Intel® AVX2, Intel® AVX-512
Installed RAM	32.0 GB (31.9 GB usable)
SSD	ADATA SX8200PNP
OS	Ubuntu 20.04 LTS
C++ Compiler	g++ (Ubuntu 11.1.0-1ubuntu1-20.04) 11.1.0

Vcpkg Installed Libraries

vcpkg commit: 662dbb5

Library	Version
argparse	2.9
concurrentqueue	1.0.3
fmt	10.0.0
hyperscan	5.4.2
libgit2	1.6.4

Single Large File Search: `OpenSubtitles.raw.en.txt`

The following searches are performed on a single large file cached in memory (~13GB, OpenSubtitles.raw.en.gz).

Regex	Line Count	ag	ugrep	ripgrep	hypergrep
Count number of times Holmes did something `hgrep -c 'Holmes did \w'`	27	n/a	1.820	1.022	0.696
Literal with Regex Suffix `hgrep -nw 'Sherlock [A-Z]\w+' en.txt`	7882	n/a	1.812	1.509	0.803
Simple Literal `hgrep -nw 'Sherlock Holmes' en.txt`	7653	15.764	1.888	1.524	0.658
Simple Literal (case insensitive) `hgrep -inw 'Sherlock Holmes' en.txt`	7871	15.599	6.945	2.162	0.650
Alternation of Literals `hgrep -n 'Sherlock Holmes\|John Watson\|Irene Adler\|Inspector Lestrade\|Professor Moriarty' en.txt`	10078	n/a	6.886	1.836	0.689
Alternation of Literals (case insensitive) `hgrep -in 'Sherlock Holmes\|John Watson\|Irene Adler\|Inspector Lestrade\|Professor Moriarty' en.txt`	10333	n/a	7.029	3.940	0.770
Words surrounding a literal string `hgrep -n '\w+[\x20]+Holmes[\x20]+\w+' en.txt`	5020	n/a	6m 11s	1.523	0.638

Git Repository Search: `torvalds/linux`

The following searches are performed on the entire Linux kernel source tree (after running make defconfig && make -j8). The commit used is f1fcb.

Regex	Line Count	ag	ugrep	ripgrep	hypergrep
Simple Literal `hgrep -nw 'PM_RESUME'`	9	2.807	0.316	0.147	0.140
Simple Literal (case insensitive) `hgrep -niw 'PM_RESUME'`	39	2.904	0.435	0.149	0.141
Regex with Literal Suffix `hgrep -nw '[A-Z]+_SUSPEND'`	536	3.080	1.452	0.148	0.143
Alternation of four literals `hgrep -nw '(ERR_SYS\|PME_TURN_OFF\|LINK_REQ_RST\|CFG_BME_EVT)'`	16	3.085	0.410	0.153	0.146
Unicode Greek `hgrep -n '\p{Greek}'`	111	3.762	0.484	0.345	0.146

Git Repository Search: `apple/swift`

The following searches are performed on the entire Apple Swift source tree. The commit used is 3865b.

Regex	Line Count	ag	ugrep	ripgrep	hypergrep
Function/Struct/Enum declaration followed by a valid identifier and opening parenthesis `hgrep -n '(func\|struct\|enum)\s+[A-Za-z_][A-Za-z0-9_]\s\('`	59026	1.148	0.954	0.154	0.090
Words starting with alphabetic characters followed by at least 2 digits `hgrep -nw '[A-Za-z]+\d{2,}'`	127858	1.169	1.238	0.156	0.095
Workd starting with Uppercase letter, followed by alpha-numeric chars and/or underscores `hgrep -nw '[A-Z][a-zA-Z0-9_]*'`	2012372	3.131	2.598	0.550	0.482
Guard let statement followed by valid identifier `hgrep -n 'guard\s+let\s+[a-zA-Z_][a-zA-Z0-9_]\s=\s*\w+'`	839	0.828	0.174	0.054	0.047

Directory Search: `/usr`

The following searches are performed on the /usr directory.

Regex	Line Count	ag	ugrep	ripgrep	hypergrep
Any HTTPS or FTP URL `hgrep "(https?\|ftp)://[^\s/$.?#].[^\s]*"`	13682	4.597	2.894	0.305	0.171
Any IPv4 IP address `hgrep -w "(?:\d{1,3}\.){3}\d{1,3}"`	12643	4.727	2.340	0.324	0.166
Any E-mail address `hgrep -w "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}"`	47509	5.477	37.209	0.494	0.220
Any valid date `MM/DD/YYYY` `hgrep "(0[1-9]\|1[0-2])/(0[1-9]\|[12]\d\|3[01])/(19\|20)\d{2}"`	116	4.239	1.827	0.251	0.163
Count the number of HEX values `hgrep -cw "(?:0x)?[0-9A-Fa-f]+"`	68042	5.765	28.691	1.439	0.611
Search any C/C++ for a literal `hgrep --filter "\.(c\|cpp\|h\|hpp)$" test`	7355	n/a	0.505	0.118	0.079

Build

Install Dependencies with `vcpkg`

git clone https://github.com/microsoft/vcpkg
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg install concurrentqueue fmt argparse libgit2 hyperscan

Build `hypergrep` using `cmake` and `vcpkg`

Clone the repository

git clone https://github.com/p-ranav/hypergrep
cd hypergrep

If `cmake` is older than `3.19`

mkdir build
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=<path_to_vcpkg>/scripts/buildsystems/vcpkg.cmake ..
make

If `cmake` is newer than `3.19`

Use the release preset:

export VCPKG_ROOT=<path_to_vcpkg>
cmake -B build -S . --preset release
cmake --build build

To build the binary for x86_64 portability, invoke cmake with -DBUILD_PORTABLE=on option. This will use -march=x86-64 -mtune=generic and -static-libgcc -static-libstdc++, and link the C++ standard library and GCC runtime statically into the binary, reducing dependencies on the target system.