Dump of a 5 day small scale data scraping project on the io game "arras.io"

Related tags

Learning resource db
Overview

Arras DB

A bot built to scan all arras.io game servers (besides sandbox) in search of bulk server and player data. This bot was run 5 days straight, althought arras.io#oc and arras.io#ob were not processed for majority of the run (due lack of IPv6 support on those servers).

About the Dump

For perspective, every server in arras.io (identifiable by a unique id, for example #oc, #ha, #wa etc) runs a single gamemode at a time, and each gamemode has a series of players. Our bot collected gamemode meta data from the arras.io gamemode "status" API, and it collected player data from both the leaderboard as well as in game. The database is accessable in JSON format (zipped) at /arras.json.zip, while the more compressed formats are located in the /db/ folder.

Layout of the Database

(Section of the README is WIP - this will be completed and written to text for non coders soon)

// arras.json is an array of `Game`s
interface Game {
  // Contains the identifier for the server / vps
  // (insert id into arras.io#${id} to get the server link)
  // Example:
  //   "id": "vcx"
  id: string; // ascii (len is 1 to 3)
  // The start time of the server / when the server first
  // started being scanned
  // Example:
  //   "start": 1657424582585
  start: number; // float64
  // Meta includes data like mspt, uptime, and game/server information
  meta: GameMeta;
  // The amount of players retrieved from the scanning
  // Example:
  //   "playerCount": 272
  playerCount: number; // uint16
  // Array of player datas (array.length === playerCount)
  players: PlayerData[];
}

interface GameMeta {
  // The average milliseconds per tick in the game (retrieved from API)
  // Example:
  //   "mspt": 4.856888294219971
  mspt: number; // float32
  // The uptime of the game in seconds
  // Example:
  //   "uptime": 17781.03125
  uptime: number; // float32
  // The unparsed code of the server, you can use lib/parseCode
  // to parseCode(code: string): string and see a more readable format
  // Example:
  //   "code": "vultr-sgp-mf"
  code: string; // ascii
  // The host server of the game (its technically specific to
  // server ID, but it was put here anyway)
  // Example:
  //   "host": "f6jklqqdgr0l0kbm.uvwx.xyz:5001"
  host: string; // ascii
}

interface PlayerData {
  // The player's body's entity id - part of the in game
  // protocol and can be used for identifying players per game
  // Example:
  //   "id": 197956
  id: number;
  // The player's in game name
  // Example:
  //   "name": "!"
  name: string; // utf8
  // The player's team color (part of the arras protocol)
  // Example:
  //   "teamColor": 12;
  teamColor: number; // uint8
  // Whether or not the player data was collected from
  // the leaderboard or not (if false, found in from fov)
  // Example:
  //   "onLb": true
  onLb: boolean; // bool
  // Name of the player's last tank
  // Example:
  //   "tankName": "Falcon"
  tankName: string; // utf8
  // Uptime of the server when the player was first found
  // Example:
  //   "firstSeen": 9854.556640625
  firstSeen: number; // float32
  // Uptime of the server when the player was last seen
  // Example:
  //   "lastSeen": 9948.310546875
  lastSeen: number; // float32
  // The last seen score of the player
  // Example:
  //   "lastSeenScore": 283571
  lastSeenScore: number; // float32
}

Executing

Be warned, the headless bot code is now outdated as of July 18th - use at your own risk

After putting your CaptchaV3Solver™ API_TOKEN into config.js, see below

Standalone

npm install
npm run start

Using Docker

docker build .

docker start

Credits

  • Thank you to Altanis for preliminaries on the socket interface, as well as for providing servers to host recaptcha solvers.

  • Thank you to bird for running the bot for more than 80% of its lifetime (he saved the project).

  • Thank you to cx88 for being the developer of arras

You might also like...

This an CRUD app built were users can create, update and delete specific day to day tasks as they wish. The app is built using webpack and served with webpack dev server

This an CRUD app built were users can create, update and delete specific day to day tasks as they wish. The app is built using webpack and served with webpack dev server

Todo-List This an CRUD app built were users can create, update and delete specific day to day tasks as they wish. The app is built using webpack and s

Sep 28, 2022

A simple application used to organize your day to day activity. Tasks can be added and removed from this list.

Minimalist ToDo List A minimalist list of TODOs. Built With HTML, CSS, JavaScript Webpack Jest NPM Node Getting Started In order to obtain a local cop

Sep 9, 2022

Today began learn about MEAN stack first with Nodejs that is a server side form of javascript. Repository for track of Day by Day improvement track in new Skill.

Today began learn about MEAN stack first with Nodejs that is a server side form of javascript. Repository for track of Day by Day improvement track in new Skill.

NodeJs Today began learn about MEAN stack first with Nodejs that is a server side form of javascript. Repository for track of Day by Day improvement t

Oct 14, 2022

aka Scaletor, take screenshots of a piece of a map and scale/compare with other parts of the map

scale-a-tron A quick-and-dirty map that lets you compare one area to another. Draw a shape around a region, zoom in to another place on the map, and c

Nov 7, 2022

🍎 A simple application which enables you to sync readings from your Withings scale to a Notion page.

🍎 A simple application which enables you to sync readings from your Withings scale to a Notion page.

weight-logger weight-logger is a simple application which enables you to sync readings from your Withings scale to a Notion page. Preview Installation

Jan 14, 2022

A Kubernetes monitoring tool to visualize large-scale activity and real-time comprehensive metrics within your cluster.

A Kubernetes monitoring tool to visualize large-scale activity and real-time comprehensive metrics within your cluster.

Armada A light-weight Kubernetes health monitoring tool. Summary Armada is an open-source tool for monitoring the health of your Kubernetes cluster. I

Nov 2, 2022

A large scale simulation which pits millions of space ships against each other in a virtual universe all running directly in SingleStore.

A large scale simulation which pits millions of space ships against each other in a virtual universe all running directly in SingleStore.

Wasm Space Program In this demo we simulate a fake universe full of thousands of solar systems. In each solar system there are many space ships and en

Nov 2, 2022

T3 is a client-side JavaScript framework for building large-scale web applications

Box has migrated using react, webpack, and the latest version of ECMAScript for our frontend projects as of 2018. We no longer support chan

Dec 8, 2022

multi-channel fulfillment at scale

Openship is an operations platform that enables multi-channel fulfillment Website · Documentation · Openship Cloud Running locally To get Openship run

Dec 31, 2022
Owner
ABC
ABC
A web scraping / data mining script for extracting beginner-friendly github repos from Y Combinator's company database

A web scraping / data mining script for extracting beginner-friendly github repos from Y Combinator's company database

Oscar Mier 27 Nov 24, 2022
Tools for Dump NUXT JS Environment Config Through Browser Window Object

Introduction Nuxt JS is an open source javascript framework making web development simple and powerful. This tool is used to get environment configura

Agus Setya R 7 Sep 5, 2022
Custom endpoint to backup the PG database and upload the `.dump` to Directus.

Directus Backup Endpoint Custom Directus endpoint to backup Postgres database using pg_dump and upload the .dump file into Directus files. Usage Clone

Guilherme Oliveira 7 Dec 21, 2022
Custom operation to backup the PG database and upload the .dump to Directus.

Directus Backup Operation Custom Directus operation to backup Postgres database using pg_dump and upload the .dump file into Directus storage. Prerequ

Guilherme Oliveira 7 Dec 21, 2022
✏️ A small jQuery extension to turn a static HTML table into an editable one. For quickly populating a small table with JSON data, letting the user modify it with validation, and then getting JSON data back out.

jquery-editable-table A small jQuery extension to turn an HTML table editable for fast data entry and validation Demo ?? https://jsfiddle.net/torrobin

Tor 7 Jul 31, 2022
🧾 My personal CLI app to manage my invoices via Web Scraping.

ALM Invoices CLI My personal CLI (Command Line Interface) app to manage my invoices via Web Scraping. WIP (Work in Progress): For now only the list co

Alexandre Marques 4 Nov 29, 2022
A simple library that I use for web scraping. Uses htmlparser2 to parse dom.

Docpa A simple library that I use for web scraping. Uses htmlparser2 to parse dom. Usage const Docpa = require("docpa") const doc = new Docpa(`<html>

Roshan Acharya 2 Jan 21, 2022
College project done for the CG Artwork lecture in 2022. Used HTML for the indexes and mainly JavaScript (using to THREE.js). Ended with the final A grade (17.3 in scale grade).

CG Artwork Project 2022 This project was done by a group of 3 in 2022 with an educational purpose for the CG Artwork lecture in Instituto Superior Téc

filipe_neves 3 Sep 19, 2022
Chrome Extensions are something which we use very frequently in our day-to-day life for example Grammarly

Chrome-Extension Chrome Extensions are something which we use very frequently in our day-to-day life for example Grammarly, In this project similarly,

Harshita sharma 97 Dec 14, 2022
Obsidian plugin: get things dead day by day

Obsidian Daily Todo Pro Plugin 中文说明 inspired by https://github.com/shichongrui/obsidian-rollover-daily-todos get things dead day by day - enhance your

null 11 Sep 18, 2022