Skip to content

VeryNiceOrg/anomaly-detector-python

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GitHub Anomaly Detector

A Flask-based webhook service that detects suspicious behavior in GitHub organizations by analyzing webhook events in real-time.

Features

  • Real-time anomaly detection from GitHub webhook events
  • Extensible rule-based system for detecting various suspicious behaviors
  • Multiple notification channels (currently console, easily extensible)
  • Type-safe domain models with comprehensive type hints
  • Clean architecture designed for multi-developer collaboration

Architecture

The application follows a clean, layered architecture:

github-anomaly-detector/
├── app/              # Application layer (Flask routes, webhook parsing)
├── domain/           # Domain layer (events, anomalies, rules)
├── services/         # Business logic (anomaly engine, state management)
├── notifiers/        # Notification handlers
└── tests/            # Test suite

Design Patterns

  • Strategy Pattern: Rules and notifiers are implemented via abstract base classes, making it trivial to add new detection rules or notification methods
  • Adapter Pattern: github_event_parser.py adapts GitHub webhook format to our domain model
  • Dependency Injection: Components are injected at initialization for easy testing and extensibility

Detected Anomalies

  1. Push Time Anomaly: Detects pushes between 14:00-16:00 UTC (configurable)
  2. Hacker Team Anomaly: Detects teams created with suspicious naming patterns (e.g., "hacker-*")
  3. Repository Lifetime Anomaly: Detects repositories created and deleted within 10 minutes

Setup

Prerequisites

  • Python 3.10+
  • pip
  • ngrok (for local testing with GitHub webhooks)

Installation

  1. Clone the repository
  2. Create and activate virtual environment:
    python3 -m venv venv
    source venv/bin/activate  # On Windows: venv\Scripts\activate
  3. Install dependencies:
    pip install -r requirements.txt
  4. Run the application:
    python run.py
    Or use the startup script:
    ./start.sh

The server will start on http://localhost:8000

Adding New Detection Rules

To add a new anomaly detection rule:

  1. Create a new file in domain/rules/ (e.g., new_rule.py)
  2. Inherit from BaseRule and implement required methods:
from typing import Any, Iterable
from domain.rules.base import BaseRule
from domain.anomalies import Anomaly
from domain.events import YourEventType

class YourNewRule(BaseRule):
    def name(self) -> str:
        return "your_rule_name"
    
    def evaluate(self, event: Any) -> Iterable[Anomaly]:
        if not isinstance(event, YourEventType):
            return []
        
        # Your detection logic here
        if suspicious_condition:
            return [
                Anomaly(
                    rule_name=self.name(),
                    description="Description of the anomaly",
                    # ... other fields
                )
            ]
        return []
  1. Register the rule in app/server.py:
engine = AnomalyEngine([
    PushTimeRule(),
    HackerTeamRule(),
    RepoLifetimeRule(repo_store),
    YourNewRule(),  # Add here
])

Adding New Notifiers

To add a new notification method:

  1. Create a new file in notifiers/ (e.g., email_notifier.py)
  2. Inherit from Notifier and implement notify():
from typing import List
from notifiers.base import Notifier
from domain.anomalies import Anomaly

class EmailNotifier(Notifier):
    def notify(self, anomalies: List[Anomaly]) -> None:
        # Send email notifications
        pass
  1. Replace or add the notifier in app/server.py:
notifier = EmailNotifier()  # Or use multiple notifiers

Testing

Run the integration tests:

PYTHONPATH=. python tests/run_integration_tests.py

Or run individual test files:

PYTHONPATH=. python tests/test_webhook_integration.py
PYTHONPATH=. python tests/test_anomaly_engine_demo.py

Webhook Configuration

  1. Start ngrok to expose your local server:

    ngrok http 8000
  2. Copy the HTTPS URL (e.g., https://xxxx-xxxx.ngrok.io)

  3. Configure GitHub webhook:

    • Go to your organization/repository Settings → Webhooks
    • Click "Add webhook"
    • Payload URL: https://your-ngrok-url.ngrok.io/webhook
    • Content type: application/json
    • Select events: Push events, Team events, Repository events
    • Click "Add webhook"

Code Quality

The codebase follows best practices:

  • Type hints throughout for better IDE support and maintainability
  • Comprehensive docstrings explaining design decisions and extensibility
  • Constants instead of magic numbers for easy configuration
  • Error handling with graceful degradation
  • Clean separation of concerns between layers

Project Structure

github-anomaly-detector/
├── app/
│   ├── server.py                 # Flask application and routes
│   └── github_event_parser.py    # GitHub webhook → domain event adapter
├── domain/
│   ├── events.py                 # Domain event models
│   ├── anomalies.py              # Anomaly data model
│   └── rules/
│       ├── base.py               # Abstract rule interface
│       ├── push_time_rule.py     # Push time detection
│       ├── hacker_team_rule.py   # Team name detection
│       └── repo_lifetime_rule.py # Repository lifetime detection
├── services/
│   ├── anomaly_engine.py         # Core detection engine
│   └── repo_state_store.py       # Repository state tracking
├── notifiers/
│   ├── base.py                   # Abstract notifier interface
│   └── console_notifier.py       # Console output notifier
├── tests/
│   ├── test_anomaly_engine_demo.py    # Engine demo tests
│   ├── test_webhook_integration.py    # Integration tests
│   └── run_integration_tests.py       # Test runner
├── requirements.txt
├── run.py
└── start.sh

License

MIT

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors