Skip to content

Instantly share code, notes, and snippets.

View shaneharter's full-sized avatar

Shane Harter shaneharter

View GitHub Profile
@shaneharter
shaneharter / README.md
Created November 8, 2026 09:47
DuckDB 2.0.0 vs 1.5.6: parse/prepare cost on large generated statements (repro scripts + measurements)

DuckDB 2.0.0 vs 1.5.6: parse and prepare cost on large generated statements

We generate SQL statements of tens to hundreds of KB (CTE chains and wide UNION ALLs over Parquet) and prepare many of them per publish cycle. Moving from DuckDB v1.5.6 (069cc9f9b5) to v2.0.0 (d54cbbe3df) roughly doubled that cycle, and the whole difference is in statement parse/prepare; catalog work, capture and chunk fetches are unchanged or faster on 2.0.

Everything below is reproducible with the three scripts in this gist. Environment for the micro numbers: macOS 15.3 arm64 (Apple Silicon, 16 cores), Bun 1.4.3, @duckdb/node-api 1.5.6-r.2 vs 2.0.0-r.1, in-memory database, default settings, runs alternated on the same host within the same minute. The host was loaded (load average 26 to 49 during the runs), so absolute times are inflated; the ratios reproduce across runs.

1. Two statement shapes (parser-repro.ts)

| statement | bytes | 1.5.6 extractStatements | 2.0.0 extractStatements | 1.5.6 prepare | 2.0.0 prepare | |--

with make_temp_dir(listing_id) as tmpdir:
logger.console('Downloading assets for %s to %s' % (listing_id, tmpdir))
photoset = photos.Photoset(filenames, tmpdir, limit=MAXIMUM_PHOTOS, subprocessor=logged_popen)
photoset.load()
if photoset.count() < MINIMUM_PHOTOS:
raise RuntimeWarning('Too few conforming photos')
photoset.process()
@shaneharter
shaneharter / gist:bc7d33a513728f5bf44c
Created September 12, 2014 19:18
PHP ImmutableObject class
<?php
/**
* An immutable value object.
*/
final class ImmutableObject
{
private $state = array();
@shaneharter
shaneharter / backbone_pushstate_router.js
Last active August 29, 2015 14:06 — forked from wololodev/backbone_pushstate_router.js
Play nice with Backbone.history.root when attaching pushstate click handler
// Only need this for pushState enabled browsers
if (Backbone.history && Backbone.history._hasPushState) {
var $document = $(window.document);
var openLinkInTab = false;
// Links like <a href="some/thing/here"> are relative to the page.
// We want to run these links thru the router
var is_relative_to_page = function(href) {
return href.match(/^\/|(http:|https:|ftp:|mailto:|javascript:)/) === null;
};
@shaneharter
shaneharter / gist:5863839
Created June 26, 2013 00:50
PHP Namespace Autoloader
<?php
$namespaces = array(
'Sheldon\examples\SignalConsole\*' => __DIR__,
'Sheldon\*' => dirname(dirname(__DIR__)) . '/src/',
);
spl_autoload_register(function ($class) use ($namespaces) {
$NAMESPACE_SEPARATOR = '\\';
@shaneharter
shaneharter / sqrt.go
Created May 13, 2013 18:21
Gotour: Newtonian square roots in Go
package main
import (
"fmt"
"math"
)
const min_delta = 0.000000000001
func Sqrt(x float64) float64 {
#!/usr/bin/php
<?php
ini_set('memory_limit', '256M');
set_time_limit(0);
require_once('include/local.conf');
require_once(COMMON_INCLUDE_PATH . 'db_mysql.class.php');
$ymds = array(20130107, 20130108, 20130109, 20130110, 20130111, 20130112, 20130113, 20130114, 20130115, 20130116,
<?php
ini_set('display_errors', 1);
require_once 'include/local.conf';
require_once (COMMON_INCLUDE_PATH.'email/EmailTypes.php');
$mutex = sem_get(ftok(__file__, 'D'), 1, 0666, 1);
register_shutdown_function(function() use($mutex) {
@sem_release($mutex);
});
@shaneharter
shaneharter / tap.js
Created December 18, 2012 18:43 — forked from evilbuck/tap.js
Object.defineProperty(Object.prototype, 'tap', {
value: function(fun){
fun.call( this );
return this;
},
enumerable: false
});
// Usage:
// a = [];
@shaneharter
shaneharter / gist:4211173
Created December 5, 2012 01:32
Progress Display for PHP CLI scripts
for ($i=1; $i<101; $i++) {
progress($i, 100, 'Percent Complete: ');
usleep(25000);
}
function progress ($step_x, $of_y, $label = '') {
$p = number_format($step_x / $of_y * 100, 1) . '%';
$p = str_pad($p, 5, ' ', STR_PAD_LEFT);
if ($label)
$p = "{$label} {$p}";