ajsutton

Adaptive eth_getLogs range: teku's method for withdrawal_l2_presence

This note is for the authors of ethereum-optimism/monitorism#234.

Problem

The withdrawal_l2_presence monitor scans WithdrawalProvenExtension1 logs in fixed ranges. The --log.scan.blocks.per.iteration flag sets the range size. If one eth_getLogs call fails, the whole run fails.

The next run retries from the same start block, with the same range size. Some providers reject a range that is too large or that returns too many logs. Some providers time out on such a range. With a fixed range, the monitor can then fail on every run and never move forward. The heartbeat alert fires, but the monitor does not recover without a config change.

ajsutton

package fault

import (
	"context"
	"errors"
	"math/big"
	"slices"
	"sync"
	"testing"
	"time"

ajsutton

#!/bin/bash
set -euo pipefail
SCRIPTS_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
STATES_DIR="${SCRIPTS_DIR}/../temp/states"
BIN_DIR="${SCRIPTS_DIR}/../bin/"

mkdir -p "${STATES_DIR}"

cd "${SCRIPTS_DIR}/../../"

ajsutton

Mar 17 19:41:01.323 WARN Peer sent invalid response to parent request., reason: ExtraBlocksReturned, peer_id: 16Uiu2HAmFq7aPgGKNQwmiNqEvsf51T6Yj7eL77y4YRVz46ohcnHZ, service: sync
Mar 17 19:40:53.001 INFO Synced                                  slot: 8656702, block:    …  empty, epoch: 270521, finalized_epoch: 270519, finalized_root: 0x3dfc…aeeb, exec_hash: 0x0fcf…2fe5 (verified), peers: 20, service: slot_notifier
Mar 17 19:40:46.776 WARN Beacon chain re-org                     reorg_distance: 2, new_slot: 8656698, new_head: 0x281ade7abae27402007394990218517e50b30ac742a333c2fead84c9c6862a74, previous_slot: 8656699, previous_head: 0xd7cd8aebe5e04d21de102f0f48c5d678166db4b7a3fc67954da8ff522981fcad, service: beacon
Mar 17 19:40:43.394 INFO New RPC block received                  hash: 0x281a…2a74, slot: 8656698
Mar 17 19:40:41.001 INFO Synced                                  slot: 8656701, block:    …  empty, epoch: 270521, finalized_epoch: 270519, finalized_root: 0x3dfc…aeeb, exec_hash: 0x333a…cc2f (verified), 

ajsutton

2022-04-23 09:04:14.364+00:00 | beaconchain-async-4 | WARN  | teku-p2p-log | Rejecting invalid block at slot 7196 with root 0x92ca30ad606a98862102536afc7984c927209ee2720a157f11ff096e11fafccd because FAILED_STATE_TRANSITION. Full block data: 0x64000000a020e190150002c17f370cf8b0d9454827ae757de2e73c1299dcff0ea3697b78464de1184ea88e7940b77233d92b706b03f21cf1e3cdfbd246d39794449bcfee701a56fbb38ca65870445c280dd5cacc64ea6898e6f0360bb83c56bab288c4551c1c0000000000004d5c000000000000752c3f0eb0522ff5aeafd027ebc021f368705812f363948fe0a95f1e47d8fd6977536e6f422afb83e7605e467ea816025a38130ba1f413238aaba917c686567d540000008f36c0d37e2dceed3d639655deb70941764761aa9566b892daff249b91f42cf48ca0732becb45344923075a5b4026fbe13138c6aac174b4e53345468db85352400bc961aa038e1deb612b1d141f6dbdfc601483ee232ca152da7e08964e49b9fae72d2ba5818fac4ec1ce0ae75679b69a8bbcd79345a88db8d04d5c83ab9c96276b60500000000009bc862535d4592c7d997874be746f919162206acb0e08703bd7f6c422e8214fb6e696d6275732d6e65746865726d696e642df09f90bc000000000000000000008001000080010

ajsutton

{
  "data": [
    {
      "slot": "2074112",
      "index": "0",
      "validators": [
        "262116",
        "164445",
        "78337",
        "247952",

ajsutton

# Extends the mainnet preset
PRESET_BASE: 'mainnet'

# For backwards compatibility in the config/spec API
CONFIG_NAME: "prater"

# Prater config
# Genesis
# ---------------------------------------------------------------
# `2**14` (= 16,384)

ajsutton

######################################
# Teku Application Alert Rules
######################################
groups:

  - name: Teku-Slot-Metrics
    rules:
      - alert: 'Teku Beacon Slot Increase'
        expr: sum (irate(beacon_slot{job="prod-teku-node"}[40s])) by (instance) == 0
        for: 1m

ajsutton

0x6400000096c203da93af6a3dfab08eda96d7740e7c1205c1cea62c977cb09c9bea6a4fc33ecf3553f8648ed449be0db9acfcad4c09ae32a19fce45f1da256f70fed3c9406d8dbe095b17d9128abd2aa56e8230af575494d583e3e31461e9c6d2a181782363cd0000000000002f02000000000000e2e80da60b648b566a798cd3d807000ffe99c3a17c0ed44663a6a02feb8ac6c1f84dbb36e6175bb0a25ab6fe4549207b3fbf9cf244f1d3c15c97f10b74d95ec054000000b1850641ed49edb9d7f31e053f73564e574f8a68ceaa79018e475009eb5b5721002f26d252885cee8c09a7818f7caa2604a9b388b72ce8a69d45eadb3ffa6564689aed264d4efe993390c04813cc6a36a81e77f17e47882ded54f1bd0127d800d70a234731285c6804c2a4f56711ddb8c82c99740f207854891028af34e27e5e000000000000000002a639204448dc1bab7a47dc4c0c0398b3c2b85c49b631fbdfcca6a15b23d214636861696e736166652f6c6f6465737461722d302e32362e312d6465762e32307c0100007c0100007c010000903b0000903b0000fffffffffffffffffffffffffffffffffffebdffe7b7ffcbbfcff0fdfb3effcfe74dbaef3717accd7bf5c7f3db76b16fffffffffffffffffffffffffffffffff9683ec3ea12a3dbbec6f4f7dba1ecfa62d4cc6952efe0be9af566a7d64f6b8804666f0a588b67312d3cdb9

ajsutton

33600 0x510c5129aedd91f8a2c4defbc5a4bccf075e82b5e1dde9e95a3951986b7d67dc2ec345e1ce58620f56d14f7508f3fdf181113677ebdfef73ba9d754188c07f47
33601 0x7d4f712ffedfbdfba6c5fffffdfcbefffffffffffffffffffffffffffbffeffffffffffffffffffffffffffffdffffffffffffffffffffffffffffffffffffff
33602 0x00000000000000000000000000000000e77f8bf5f77dafe9dabdf79b6bfde7bc2feb7fa9fedb732dd6d7cf7508fffff0f9b537f6ebdfbf7feffff4cebfcefffb
33603 0x00000000000000000000000000000000e77f8bf5d77dafc8dabdb70b6bfde7bc2feb7fa9fedb732d96d7cf7508fffff0f9b537f68bdfbf7feffff4cebf8effff
33604 0x7d4f712ffedfbdfba6857ffefdf8beffe77f8bf5d77dafc8dabdb70b6bfde7bc2feb7fa9fedb732d96d7cf7508fffff0f9b537f68bdfbf7ffffff5cfbf8effff
33605 0x7d4f712ffedfbdfba6857ffefdf8beffe77f8bf5d77dafc8dabdb70b6bfde7bc2feb7fa9fedb732d96d7cf7508fffff0f9b537f68bdfbf7ffffff5cfbf8effff
33606 0x7d4f602f725fadfba6857fb6fdd8ae7fe77f0bf5d77d8fc0da9db70b6bfde7bc2fe97f29fecb732c9657cf7108ffdfe0f9b537f68bcfbf6ffffff5cfbf8effff
null null
33608 0xffffeeff737fedffffbf7fb6ffdbef7fe77f0bf5d77d8fc