Data Processing

Data Processing

The data processing layer in Currencia is handled by a standalone Nitro server running scheduled tasks for data transformation and synchronization.

Formatter Service

The formatter service operates through scheduled tasks:

scheduledTasks: {
  '* * * * *': ['sync:mongo', 'format:sentiments', 'format:data']
}

Task Structure

Each task in the formatter follows a common pattern:

1. Initialization

• RabbitMQ client setup
• MongoDB client connection
• Cleanup handlers configuration

2. Message Processing

• Consume messages from RabbitMQ
• Fetch corresponding data from MongoDB
• Format data using @currencia/cryptos package
• Save to PostgreSQL
• Delete processed data from MongoDB

Error Handling

• Maximum retry attempts: 3
• Retry delay: 5000ms
• Dead Letter Queue (DLQ) processing
• Graceful shutdown handling
• Connection cleanup

Performance Configuration

{
  maxRetries: 3,
  retryDelay: 5000,
  prefetchCount: 5
}

Data Transformation

The formatter:

• Standardizes cryptocurrency data format
• Validates against @currencia/cryptos package
• Converts timestamps
• Normalizes price values
• Handles missing or invalid data

Task Monitoring

• Console logging for task progress
• Error tracking
• Success rate monitoring
• Processing statistics