Ethereum Transactions Crawler

A web application to fetch, track, and analyze Ethereum wallet transactions and ETH and token balances over time.

---

🛠 Tech Stack

• Backend / API: Node.js, TypeScript, Next.js (API Routes)
• Database: MongoDB Atlas, Mongoose ORM
• Ethereum Integration: Ethers.js, Etherscan API, Multicall3
• Testing: Jest, ts-jest
• Environment Management: dotenv
• Version Control: Git

---

📦 Features

• Fetch all transactions for a given wallet address.
• Store transaction history in MongoDB.
• Supports fetching and storing ETH and tokens balance snapshot at specific date.
• Paginated transaction history and total count per wallet.
• Retry mechanism for failed API calls.

---

🎬 Demo & Screenshots

• Live Demo: Live Demo
• Video Demo: Video

Screenshots:

---

⚡ Getting Started

Prerequisites

• Node.js >= 18
• Yarn or NPM
• MongoDB Atlas CLI
• Etherscan API key
• Docker

Setup

1. Clone the repository

git clone git@github.com:Addis-Belete/ethereum-transactions-crawler.git
cd ethereum-transactions-crawler

2. Install dependencies

yarn install
# or
npm install

3. Install MongoDB Atlas CLI

# macOS
brew install mongodb-atlas-cli
# Windows
choco install mongodb-atlas-cli

4. Install Docker

# macOS
brew install docker
# Windows
choco install docker-desktop

5. Start Docker Engine

6. Deploy Mongo Cluster

atlas deployments setup ethcrawler --type local \
--mdbVersion 8.0 --port 27018 --connectWith skip

7. Create .env file in the root:

MONGODB_URI=mongodb://localhost:27018/?directConnection=true
ETHERSCAN_API_KEY=<your_etherscan_api_key>
RPC_URL=<your_rpc_node_url>

8. Run the app locally

yarn dev
# or
npm run dev

The app will run at http://localhost:3000.

---

🧪 Running Tests

yarn test
# or
npm run test

• Tests use Jest and cover repositories, services, and validation functions.
• Mocked dependencies allow testing without a real blockchain or database connection.

---

🏗 Project Structure

src/
├─ lib/
│  ├─ model/           # Mongoose models
│  ├─ repositories/    # Database access layer
│  ├─ services/        # Etherscan & Ethereum services
│  └─ utils/           # Helpers & error handling
├─ types/              # TypeScript interfaces
└─ app/                # Next.js app

---

⚠ Notes

• Free Etherscan API keys have rate limits; parallel calls for large block ranges are not supported.
• Only ERC20 tokens and ETH are supported for token balances.
• Historical balances are available starting from 2024-07-14 using Multicall3.