Skip to content

Instantly share code, notes, and snippets.

View olizilla's full-sized avatar

Oli Evans olizilla

340 followers · 24 following
View GitHub Profile
@olizilla
olizilla / what-is-parq.md
Last active October 7, 2026 15:26
parq: homebrew for data

What is parq

parq is homebrew for data.

  • Its input is wild data. Any csv, xls, json, xml source.
  • Its output is a reproducible parquet dataset saved in an OCI registry, like docker images or [homebrew] bottles.

It's a low co-ordination way to let anyone who cares about a dataset publish a tidier version of it in parquet: a format designed to be nice to query locally or over http. (vs e.g. xml, a format designed for modelling hierarchies)

@olizilla
olizilla / portable-storge-specs.md
Created June 12, 2026 10:50

Generating Portable Storage Specs

How we get from the S3 API specifications to a vendor-neutral, machine-readable subset.

The gist

  • Filter the AWS smithy spec down to the operations defined in tier-1.yml
  • Rewrite the API docs to be vendor neutral and coherent for that subset
  • Merge the new docs with the bare bones smithy spec to create the new one
  • Translate the smithy spec into an openAPI spec.
@olizilla
olizilla / w3s-writes-pipeline-legacy.md
Created December 5, 2023 11:12 
---
title: w3 writes
---
flowchart TD
    w3infra --> |upload/add rootCid,carCid|r2-dudewhere 
    api.web3.storage -->|PUT carCid/carCid.idx|r2-satnav[(R2 satnav \n bucket)]
    api.web3.storage -->|PUT carCid/carCid.car|r2-carpark[(R2 carpark \n bucket)]
    api.web3.storage -->|PUT carCid/carCid.idx|s3-carpark[(S3 carpark \n bucket)]    
    api.web3.storage -->|PUT rootCid/carCid|r2-dudewhere[(R2 dudewhere \n bucket)]
@olizilla
olizilla / w3up-client-upload-sequence.md
Created December 4, 2023 17:54 
---
title: w3s upload
# w3up-client upload sequence
---
sequenceDiagram 
    Agent ->> w3up: store/add (carCid)
    w3up -->> Agent: signed carURL
    Agent ->> R2: PUT to carURL
 Agent ->> w3up: store/add (indexCarCid)
@olizilla
olizilla / w3s-to-ipni-pipeline.md
Last active December 4, 2023 15:42 
---
title: w3 indexing pipeline
---
flowchart TD
    s3-dotstorage[(S3 dotstorage\nbucket)] -->|ObjectCreated fa:fa-car| bucket-to-indexer(bucket-to-indexer ƛ)
    bucket-to-indexer -->|region/bucket/cid/cid.car| indexer-queue[/indexer queue/]
    indexer-queue --> indexer(Indexer ƛ)
    indexer --> |zQmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn| multihash-queue[/multihash queue/]
    multihash-queue --> ipni-advertisment-content(ipni advertisment content ƛ)
@olizilla
olizilla / got.js
Created March 13, 2023 09:20
backup multi-progress bar cli from logs
import readline from 'node:readline'
import cliProgress from 'cli-progress'
const COUNT_PER_FILE = 1_000_000
const inputCounts = new Map([
['nft-30.json', 196294],
['web3-4.json', 576770]
])
function getCountPerFile (file) {
return inputCounts.get(file) ?? COUNT_PER_FILE
@olizilla
olizilla / ipfs-create-car-sync.js
Created March 1, 2023 08:30
CarBufferWriter for synchronous in-memory CAR creation
import fs from 'fs'
import { CarBufferWriter } from '@ipld/car'
import { CID } from 'multiformats/cid'
import * as json from 'multiformats/codecs/json'
import * as raw from 'multiformats/codecs/raw'
import { sha256 } from 'multiformats/hashes/sha2'
/**
* @param {CID} root
* @param {Block[]} blocks
@olizilla
olizilla / car-reader-idb.ts
Created June 30, 2021 13:08
CarReaderIdb - A CarReader impl that decodes blocks directly to idb via idb-keyval
import * as idb from 'idb-keyval'
import { CID } from 'multiformats'
import { BytesReader } from '@ipld/car/lib/coding'
import { createDecoder, asyncIterableReader } from '@ipld/car/lib/decoder.js'
export class CarReaderIdb {
_version: number;
_roots: Array<CID>;
constructor (version: number, roots: Array<CID>) {
@olizilla
olizilla / full.json
Created March 30, 2021 15:05
Lotus OpenRPC definitions from 1.7.0-dev https://github.com/filecoin-project/lotus/tree/f4a2c8fa49dfc456090f5a5ea1b4ae62d33b4f85/build/openrpc
{
"openrpc": "1.2.6",
"info": {
"title": "Lotus RPC API",
"version": "1.7.0-dev"
},
"methods": [
{
"name": "Filecoin.BeaconGetEntry",
"description": "```go\nfunc (s *FullNodeStruct) BeaconGetEntry(p0 context.Context, p1 abi.ChainEpoch) (*types.BeaconEntry, error) {\n\treturn s.Internal.BeaconGetEntry(p0, p1)\n}\n```",
@olizilla
olizilla / what-are-ipfs-gateways.md
Created January 16, 2020 20:06
The HTTP to IPFS Gateway - Explained

The HTTP to IPFS Gateway

The gateway hosted at https://ipfs.io allows browsers and tools that speak http to access content from the IPFS network. It provides the canonical way to address IPFS content via http scheme urls.

A request for:

https://ipfs.io/ipfs/QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN

...will hit the gateway and get forwarded to an IPFS node, which fetches the block for the CID QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN and returns the content via http.