Skip to content

Commit 09f424a

Browse files
lemireclaude
lemire
and
claude
committed
Add project landing page for GitHub Pages
A clean, simdjson.org-style single-page site presenting the library: hero with throughput stats, highlights, quick start, codec features, publications, related libraries and footer. Self-contained (no CDNs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 1e61945 commit 09f424a

3 files changed

Lines changed: 499 additions & 0 deletions

File tree

‎.nojekyll‎

Whitespace-only changes.

‎index.html‎

Lines changed: 284 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,284 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="utf-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1">
6+
<title>SIMDCompressionAndIntersection — Decoding billions of integers per second</title>
7+
<meta name="description" content="A C/C++ library for fast SIMD compression and intersection of sorted lists of integers. Decode at least 4 billion compressed integers per second (15 GB/s).">
8+
<meta property="og:title" content="SIMDCompressionAndIntersection">
9+
<meta property="og:description" content="Fast SIMD compression and intersection of sorted lists of integers — decode billions of integers per second.">
10+
<meta property="og:type" content="website">
11+
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' rx='20' fill='%232563eb'/><text x='50' y='70' font-size='62' font-family='monospace' font-weight='bold' text-anchor='middle' fill='white'>&lt;/&gt;</text></svg>">
12+
<link rel="stylesheet" href="styles.css">
13+
</head>
14+
<body>
15+
16+
<header class="nav">
17+
<div class="container nav-inner">
18+
<a class="brand" href="#top">
19+
<span class="brand-mark">SIMD</span><span class="brand-rest">Compression&amp;Intersection</span>
20+
</a>
21+
<nav class="nav-links">
22+
<a href="#highlights">Highlights</a>
23+
<a href="#quickstart">Quick&nbsp;start</a>
24+
<a href="#features">Features</a>
25+
<a href="#publications">Publications</a>
26+
<a class="nav-gh" href="https://github.com/fast-pack/SIMDCompressionAndIntersection" target="_blank" rel="noopener">GitHub ↗</a>
27+
</nav>
28+
</div>
29+
</header>
30+
31+
<main id="top">
32+
33+
<!-- Hero -->
34+
<section class="hero">
35+
<div class="container hero-inner">
36+
<div class="hero-copy">
37+
<h1>Decoding <span class="grad">billions of integers</span> per second.</h1>
38+
<p class="lede">
39+
A C/C++ library for fast <strong>compression</strong> and <strong>intersection</strong>
40+
of sorted lists of integers using SIMD instructions — with particular attention to
41+
differential coding and novel schemes such as SIMD&nbsp;Galloping.
42+
</p>
43+
<div class="hero-stats">
44+
<div class="stat"><span class="num">4&nbsp;billion+</span><span class="lbl">integers decoded / second</span></div>
45+
<div class="stat"><span class="num">15&nbsp;GB/s</span><span class="lbl">decompression throughput</span></div>
46+
<div class="stat"><span class="num">&gt; gzip · LZ4 · Snappy</span><span class="lbl">far faster than generic codecs</span></div>
47+
</div>
48+
<div class="hero-cta">
49+
<a class="btn btn-primary" href="https://github.com/fast-pack/SIMDCompressionAndIntersection" target="_blank" rel="noopener">★ Star on GitHub</a>
50+
<a class="btn btn-ghost" href="#quickstart">Get started</a>
51+
</div>
52+
<div class="hero-badges">
53+
<img alt="Ubuntu CI" src="https://github.com/lemire/SIMDCompressionAndIntersection/actions/workflows/ubuntu22.yml/badge.svg">
54+
<img alt="Visual Studio CI" src="https://github.com/lemire/SIMDCompressionAndIntersection/actions/workflows/visualstudio.yml/badge.svg">
55+
</div>
56+
</div>
57+
58+
<div class="hero-code">
59+
<div class="code-card">
60+
<div class="code-head">
61+
<span class="dot"></span><span class="dot"></span><span class="dot"></span>
62+
<span class="code-file">example.cpp</span>
63+
</div>
64+
<pre class="code"><code><span class="c1">// Pick a codec and compress sorted integers</span>
65+
<span class="t">IntegerCODEC</span> &amp;codec =
66+
*<span class="t">CODECFactory</span>::getFromName(<span class="s">"s4-fastpfor-d1"</span>);
67+
68+
codec.<span class="fn">encodeArray</span>(input.data(), input.size(),
69+
out.data(), outSize);
70+
71+
<span class="c1">// SIMD intersection of two sorted lists</span>
72+
<span class="t">auto</span> inter =
73+
<span class="t">IntersectionFactory</span>::getFromName(<span class="s">"simd"</span>);
74+
<span class="t">size_t</span> n = <span class="fn">inter</span>(a.data(), a.size(),
75+
b.data(), b.size(), a.data());</code></pre>
76+
</div>
77+
</div>
78+
</div>
79+
</section>
80+
81+
<!-- Highlights -->
82+
<section id="highlights" class="section">
83+
<div class="container">
84+
<h2 class="section-title">Why this library</h2>
85+
<div class="cards">
86+
<article class="card">
87+
<div class="ic">⚡</div>
88+
<h3>Fast</h3>
89+
<p>Decodes at least 4&nbsp;billion compressed integers per second on most desktop and laptop CPUs — roughly 15&nbsp;GB/s.</p>
90+
</article>
91+
<article class="card">
92+
<div class="ic">▦</div>
93+
<h3>SIMD-accelerated</h3>
94+
<p>Built on SSE/AVX vector instructions to pack, unpack and intersect integers many lanes at a time.</p>
95+
</article>
96+
<article class="card">
97+
<div class="ic">↗</div>
98+
<h3>Sorted-integer focused</h3>
99+
<p>Specialized for sorted lists with first-class differential (delta) coding — ideal for inverted indexes and posting lists.</p>
100+
</article>
101+
<article class="card">
102+
<div class="ic">⨯</div>
103+
<h3>SIMD Galloping</h3>
104+
<p>Introduces new vectorized intersection schemes, including SIMD&nbsp;Galloping, for very fast set intersection.</p>
105+
</article>
106+
<article class="card">
107+
<div class="ic">⌥</div>
108+
<h3>Many codecs</h3>
109+
<p>FastPFOR, BP128, StreamVByte, Frame-of-Reference, Masked&nbsp;VByte, VarintGB and more — selectable by name.</p>
110+
</article>
111+
<article class="card">
112+
<div class="ic">🎓</div>
113+
<h3>Peer reviewed</h3>
114+
<p>Backed by peer-reviewed research in <em>Software: Practice &amp; Experience</em>, <em>ACM TOIS</em> and more.</p>
115+
</article>
116+
</div>
117+
</div>
118+
</section>
119+
120+
<!-- Quick start -->
121+
<section id="quickstart" class="section section-alt">
122+
<div class="container quickstart">
123+
<div class="quickstart-copy">
124+
<h2 class="section-title">From clone to running in three commands</h2>
125+
<p>The library builds a static archive plus a header set you can drop into your own project.
126+
On Linux, macOS and similar systems:</p>
127+
<div class="terminal">
128+
<pre><code><span class="pr">$</span> git clone https://github.com/fast-pack/SIMDCompressionAndIntersection
129+
<span class="pr">$</span> cd SIMDCompressionAndIntersection
130+
<span class="pr">$</span> make && ./unit</code></pre>
131+
</div>
132+
<p class="muted">
133+
A static library <code>libSIMDCompressionAndIntersection.a</code> is produced, with headers
134+
in <code>include/</code>. Build the demo with <code>make example &amp;&amp; ./example</code>.
135+
Windows users build with <code>nmake -f .\makefile.vc</code>.
136+
</p>
137+
</div>
138+
<div class="quickstart-code">
139+
<div class="code-card">
140+
<div class="code-head">
141+
<span class="dot"></span><span class="dot"></span><span class="dot"></span>
142+
<span class="code-file">roundtrip.cpp</span>
143+
</div>
144+
<pre class="code"><code><span class="pp">#include</span> <span class="s">"codecfactory.h"</span>
145+
<span class="pp">#include</span> <span class="s">"intersection.h"</span>
146+
<span class="k">using namespace</span> SIMDCompressionLib;
147+
148+
<span class="t">IntegerCODEC</span> &amp;codec =
149+
*<span class="t">CODECFactory</span>::getFromName(<span class="s">"s4-fastpfor-d1"</span>);
150+
151+
<span class="c1">// compress</span>
152+
<span class="t">size_t</span> outSize = out.size();
153+
codec.<span class="fn">encodeArray</span>(in.data(), in.size(),
154+
out.data(), outSize);
155+
156+
<span class="c1">// decompress — exact, lossless round-trip</span>
157+
<span class="t">size_t</span> recovered = back.size();
158+
codec.<span class="fn">decodeArray</span>(out.data(), outSize,
159+
back.data(), recovered);</code></pre>
160+
</div>
161+
</div>
162+
</div>
163+
</section>
164+
165+
<!-- Features -->
166+
<section id="features" class="section">
167+
<div class="container">
168+
<h2 class="section-title">What's inside</h2>
169+
<p class="section-sub">Pick any scheme by name through <code>CODECFactory</code> or <code>IntersectionFactory</code>.</p>
170+
<div class="feature-grid">
171+
<div class="feature">
172+
<h4>Bit packing</h4>
173+
<p>SIMD bit-packing (BP128 / s4-bp128) with optional delta coding in 1, 2 or 4 lanes.</p>
174+
</div>
175+
<div class="feature">
176+
<h4>FastPFOR</h4>
177+
<p>Patched frame-of-reference codec tuned for sorted data, including <code>s4-fastpfor-d1</code>.</p>
178+
</div>
179+
<div class="feature">
180+
<h4>Stream VByte</h4>
181+
<p>Faster byte-oriented integer compression using control streams and shuffles.</p>
182+
</div>
183+
<div class="feature">
184+
<h4>Varint family</h4>
185+
<p>VByte, Masked VByte, VarintGB and VarintG8IU variable-length encodings.</p>
186+
</div>
187+
<div class="feature">
188+
<h4>Frame of Reference</h4>
189+
<p>Classic and SIMD frame-of-reference compression for tightly clustered values.</p>
190+
</div>
191+
<div class="feature">
192+
<h4>SIMD intersection</h4>
193+
<p>Vectorized set intersection, including SIMD Galloping, for sorted posting lists.</p>
194+
</div>
195+
<div class="feature">
196+
<h4>Differential coding</h4>
197+
<p>In-place and on-the-fly delta coding to exploit the structure of sorted integers.</p>
198+
</div>
199+
<div class="feature">
200+
<h4>Cross-platform</h4>
201+
<p>Tested on Linux, macOS and Windows; SSE2 minimum, SSE&nbsp;4.1 / AVX recommended.</p>
202+
</div>
203+
</div>
204+
</div>
205+
</section>
206+
207+
<!-- Publications -->
208+
<section id="publications" class="section section-alt">
209+
<div class="container">
210+
<h2 class="section-title">Backed by research</h2>
211+
<p class="section-sub">The techniques in this library are described in peer-reviewed publications.</p>
212+
<ul class="pubs">
213+
<li>
214+
<a href="https://arxiv.org/abs/1709.08990" target="_blank" rel="noopener">Stream VByte: Faster Byte-Oriented Integer Compression</a>
215+
<span class="pub-venue">Daniel Lemire, Nathan Kurz, Christoph Rupp — Information Processing Letters 130, 2018</span>
216+
</li>
217+
<li>
218+
<a href="http://arxiv.org/abs/1401.6399" target="_blank" rel="noopener">SIMD Compression and the Intersection of Sorted Integers</a>
219+
<span class="pub-venue">Daniel Lemire, Leonid Boytsov, Nathan Kurz — Software: Practice &amp; Experience 46(6), 2016</span>
220+
</li>
221+
<li>
222+
<a href="http://arxiv.org/abs/1209.2137" target="_blank" rel="noopener">Decoding billions of integers per second through vectorization</a>
223+
<span class="pub-venue">Daniel Lemire, Leonid Boytsov — Software: Practice &amp; Experience 45(1), 2015</span>
224+
</li>
225+
<li>
226+
<a href="http://arxiv.org/abs/1503.07387" target="_blank" rel="noopener">Vectorized VByte Decoding</a>
227+
<span class="pub-venue">Jeff Plaisance, Nathan Kurz, Daniel Lemire — International Symposium on Web Algorithms, 2015</span>
228+
</li>
229+
<li>
230+
<a href="http://arxiv.org/abs/1502.01916" target="_blank" rel="noopener">A General SIMD-based Approach to Accelerating Compression Algorithms</a>
231+
<span class="pub-venue">W. X. Zhao, X. Zhang, D. Lemire, et al. — ACM Transactions on Information Systems 33(3), 2015</span>
232+
</li>
233+
</ul>
234+
</div>
235+
</section>
236+
237+
<!-- Related -->
238+
<section class="section">
239+
<div class="container">
240+
<h2 class="section-title">Related libraries</h2>
241+
<div class="related">
242+
<a class="rel" href="https://github.com/lemire/simdcomp" target="_blank" rel="noopener"><strong>simdcomp</strong><span>A simpler companion library written in pure C.</span></a>
243+
<a class="rel" href="https://github.com/lemire/FastPFor" target="_blank" rel="noopener"><strong>FastPFor</strong><span>C++ integer compression, not specialized for sorted lists.</span></a>
244+
<a class="rel" href="https://github.com/lemire/JavaFastPFOR" target="_blank" rel="noopener"><strong>JavaFastPFOR</strong><span>Integer compression for the JVM.</span></a>
245+
<a class="rel" href="https://github.com/lemire/FrameOfReference" target="_blank" rel="noopener"><strong>FrameOfReference</strong><span>Dedicated frame-of-reference compression in C++.</span></a>
246+
</div>
247+
</div>
248+
</section>
249+
250+
<!-- CTA -->
251+
<section class="cta">
252+
<div class="container cta-inner">
253+
<h2>Ready to compress at SIMD speed?</h2>
254+
<p>Apache 2.0 licensed, patent-free as far as the authors know, and battle-tested across platforms.</p>
255+
<a class="btn btn-primary btn-lg" href="https://github.com/fast-pack/SIMDCompressionAndIntersection" target="_blank" rel="noopener">Get it on GitHub ↗</a>
256+
</div>
257+
</section>
258+
259+
</main>
260+
261+
<footer class="footer">
262+
<div class="container footer-inner">
263+
<div class="footer-col">
264+
<div class="brand"><span class="brand-mark">SIMD</span><span class="brand-rest">Compression&amp;Intersection</span></div>
265+
<p class="muted small">Fast SIMD compression and intersection of sorted integers.<br>Apache License 2.0.</p>
266+
</div>
267+
<div class="footer-col">
268+
<h5>Project</h5>
269+
<a href="https://github.com/fast-pack/SIMDCompressionAndIntersection" target="_blank" rel="noopener">Source code</a>
270+
<a href="https://github.com/fast-pack/SIMDCompressionAndIntersection/issues" target="_blank" rel="noopener">Issue tracker</a>
271+
<a href="https://github.com/fast-pack/SIMDCompressionAndIntersection/blob/master/LICENSE" target="_blank" rel="noopener">License</a>
272+
</div>
273+
<div class="footer-col">
274+
<h5>Authors</h5>
275+
<p class="muted small">Leonid Boytsov, Nathan Kurz, Daniel Lemire, Owen Kaser, Andrew Consroe, Shlomi Vaknin, Christoph Rupp, Bradley Grainger, and others.</p>
276+
</div>
277+
</div>
278+
<div class="container footer-bottom muted small">
279+
Supported by NSERC grant 26143. Built with vector instructions and care.
280+
</div>
281+
</footer>
282+
283+
</body>
284+
</html>

0 commit comments

Comments
 (0)