Decentralized RPC

milestones and todo checklist along the way of fully decentralized ops: deploys, monitors, supports of testnets / rpc / gateways:

Weekly Milestones

Pocket Network - Unblocked all known issues, dialed up to 10% over the weekend, 25% by end of the month
Covalent Network - still pushing through, some members in Lisbon. Need to sync on the indexing solution with Sam
ChainStack - will explore with them next week

Pocket Network - Traffic shifting testing began, SSL certificates being put in place on Pocket gateways
Covalent Network - agreement drawn up

Pocket Network - Finalized closing plan, to discuss internally on traffic shifting next week
Covalent Network - Kickoff discussion on RPC and indexing mechanisms, to follow up on drafting an agreement between teams
QuickNode - spoke to Solana (no outages), Polygon (team members were new).

Community Feedback: QuickNode made similar promises to both Polygon and Fantom, and failed in part because of the use of LevelDB by the archival nodes. LevelDB causes the database to compact quite frequently, and without notice when not triggering it manually. To ensure availability, they have to run 3-4 high end servers, and rotate compacting manually in order to have a solid uptime. This will get worse as the chain grow and that’s to support a single geographic location.

Pocket Network - met with them in NYC, plan to go live 1st week of October. Marketing plan in place

One time setup cost token swap worth $50K POKT tokens, rest is incentivized by POKT's staking-and-reward mechanism
Two planned phases:

Phase 1 — route 25% RPC traffic to Pocket Network via token swap agreement, fully online by end of October
Phase 2 — add Harmony Validators into the Pocket Network of nodes, lead by a Validator node runner, to onramp 8 to 16 Validators, with POKT tokens sponsored by Harmony worth ~$4,500 per node

Latency & routing concern answered — client requests are routed to the nearest API Gateway, from there they measure the lowest latency nodes from the 5 nodes that are elected in a 15-min session window. There are likely more than 1 session running at any point in time for our level of traffic
Goal - Pocket node runners take 10% of RPC network traffic by the first week of October, and 25% by end of October
Scope - Pocket Network takes RPC and GraphQL services

begun discussion with QuickNode
they provide full-service with pay-as-you-go model
$250K setup fee plus a 5c/10K req cost, equiv. to $7.5K/month for 50% of our RPC traffic ~ 50M relays/day)
when signed by end of this month, they'll be ready to deliver by Dec 30

Weekly On-Call Summary

RPC node stabilization

RPC node are reaching critical level of disk space usage - upgrade completion on 30th September 2021

need to follow up with dapp projects if issue is still ongoing since 4.2.1 upgrade

New self-hosted API gateway - 15th October 2021