Modern Docker-based testing framework with API, UI, Performance & Database tests + Professional Web Dashboard
|
Just run Professional dashboard with real-time test streaming All tests run in isolated containers |
Interactive HTML reports with historical trends & SQLite storage API • UI (Playwright) • Performance (k6) • Database Stop all processes & exit cleanly |
Only 2 things needed on your machine:
| Requirement | Version | Installation |
|---|---|---|
| 🐳 Docker Desktop | Latest | Download Here |
| 🐍 Python | 3.11+ | Download Here |
That's it! Everything else (Java, Node.js, Playwright, Allure) is auto-installed in Docker.
Verify installation:
docker --version && python --version# 1️⃣ Clone the repository
git clone https://github.com/Timi953/unified-testing-framework.git
cd unified-testing-framework
# 2️⃣ Launch the framework
python start.py
# 3️⃣ Open browser to http://127.0.0.1:5000graph LR
A[python start.py] --> B[Create venv]
B --> C[Install dependencies]
C --> D[Build Docker images]
D --> E[Start Web UI]
E --> F[🎉 Ready to test!]
- ⚙️ Creates isolated virtual environment
- 📦 Installs Python dependencies
- 🐳 Builds Docker images (first time only, ~5 min)
- 🌐 Launches Web UI at http://127.0.0.1:5000
- 🎉 Opens browser automatically
| Button | What It Does |
|---|---|
| 🟢 Run All Tests | Executes API + UI + Performance + Database tests |
| 🔵 API Tests | Tests REST API endpoints |
| 🟣 UI Tests | Browser automation with Playwright |
| 🟡 Performance Tests | k6 load/stress/spike testing |
| 🟠 Database Tests | Tests database operations (CRUD, validation, performance) |
| 📊 Show Results | Opens custom interactive dashboard with test history |
| ⏹️ Stop | Terminates running tests |
| 🧹 Cleanup | Kills background processes |
| ❌ Stop All & Exit | Complete shutdown + closes browser |
Watch your tests execute in real-time with live streaming output!
[INFO] Starting ALL tests...
============================= test session starts ==============================
platform linux -- Python 3.11.14, pytest-8.4.2
collected 7 items
tests/integration/api/test_simple_api.py::test_get_users PASSED [ 14%]
tests/integration/api/test_simple_api.py::test_create_user PASSED [ 28%]
tests/performance/test_api_performance.py::test_response_time PASSED [ 42%]
tests/ui/test_simple_demo.py::test_navigate[chromium] PASSED [ 57%]
tests/ui/test_simple_demo.py::test_page_elements[chromium] PASSED [ 71%]
tests/ui/test_youtube_validation.py::test_search[chromium] PASSED [ 85%]
tests/ui/test_youtube_validation.py::test_metadata[chromium] PASSED [100%]
========================= 7 passed in 18.23s ================================
[SUCCESS] Tests completed! View the report below. ✓
Clean interface with real-time test execution & streaming console output
Custom dashboard with historical trends, test statistics & Allure integration
unified-testing-framework/
├── 🚀 start.py # One-command launcher
├── 🌐 web_ui/ # Flask dashboard + SQLite database + report parser
├── 🐳 docker/ # Docker configs
├── 🧪 tests/
│ ├── integration/
│ │ ├── api/ # API tests
│ │ └── database/ # Database tests (SQLite mock + query performance)
│ ├── ui/simple/ # UI tests (Playwright)
│ └── performance/k6/ # k6 performance tests (load/stress/spike)
├── 📊 reports/
│ ├── raw/ # Test results (JSON/XML)
│ ├── archive/ # Timestamped reports
│ └── latest/ # Current report + history
├── 📖 docs/ # Documentation
│ ├── DATABASE_INTEGRATION_GUIDE.md # DB setup guide
│ ├── CUSTOM_UI_README.md # Custom UI features
│ └── QUICK_START_CUSTOM_UI.md # Custom UI quick start
└── 📦 venv/ # Auto-created virtualenv
🔧 Writing API Tests
import pytest
import allure
@allure.epic("API Testing")
@allure.feature("User Management")
@pytest.mark.api
class TestMyAPI:
@allure.title("Test GET endpoint")
@allure.description("Verify that the API returns user list correctly")
@allure.severity(allure.severity_level.CRITICAL)
def test_get_endpoint(self, api_client, api_base_url):
with allure.step("Send GET request to /users endpoint"):
response = api_client.get(f"{api_base_url}/users")
with allure.step("Verify response status code"):
assert response.status_code == 200
with allure.step("Verify response contains users"):
users = response.json()
assert len(users) > 0
assert 'id' in users[0]
assert 'username' in users[0]
@allure.title("Test POST endpoint - Create user")
def test_create_user(self, api_client, api_base_url):
new_user = {
"username": "testuser",
"email": "test@example.com",
"first_name": "Test",
"last_name": "User"
}
with allure.step("Send POST request to create user"):
response = api_client.post(f"{api_base_url}/users", json=new_user)
with allure.step("Verify user creation was successful"):
assert response.status_code == 201
created_user = response.json()
assert created_user['username'] == new_user['username']
assert 'id' in created_user
@allure.title("Test error handling - Invalid data")
def test_invalid_data(self, api_client, api_base_url):
with allure.step("Send request with invalid data"):
response = api_client.post(f"{api_base_url}/users", json={})
with allure.step("Verify API returns error"):
assert response.status_code in [400, 422]Available fixtures:
api_client- Pre-configuredrequests.Sessionapi_base_url- Mock server URL (Docker network)sample_user_data- Sample test data
Best Practices:
- Use
@allure.step()for detailed test steps - Add
@allure.severity()to prioritize tests - Use
@allure.feature()and@allure.epic()for organization - Test both success and error scenarios
- Validate response structure, not just status codes
Maintenance:
- API tests run automatically when mock-server container starts
- To update mock data: Edit
src/services/mock_server.py - To add new endpoints: Update mock server and add corresponding tests
- Run specific test:
pytest tests/integration/api/test_my_api.py -v
🎭 Writing UI Tests
import pytest
import allure
from playwright.sync_api import Page, expect
@allure.epic("UI Testing")
@allure.feature("Web Navigation")
@pytest.mark.ui
class TestMyUI:
@allure.title("Test page navigation")
@allure.description("Verify page loads correctly and title is displayed")
@allure.severity(allure.severity_level.CRITICAL)
def test_navigation(self, page: Page):
with allure.step("Navigate to homepage"):
page.goto("https://example.com")
expect(page).to_have_title("Example Domain")
with allure.step("Verify page elements are visible"):
expect(page.locator("h1")).to_be_visible()
with allure.step("Capture screenshot"):
allure.attach(
page.screenshot(),
name="Page Loaded",
attachment_type=allure.attachment_type.PNG
)
@allure.title("Test form interaction")
def test_form_interaction(self, page: Page):
with allure.step("Navigate to form page"):
page.goto("https://example.com/form")
with allure.step("Fill out form fields"):
page.fill("#username", "testuser")
page.fill("#email", "test@example.com")
with allure.step("Submit form"):
page.click("button[type='submit']")
with allure.step("Verify success message"):
expect(page.locator(".success-message")).to_be_visible()
allure.attach(
page.screenshot(),
name="Form Submitted",
attachment_type=allure.attachment_type.PNG
)
@allure.title("Test responsive design - Mobile")
def test_mobile_view(self, page: Page):
with allure.step("Set mobile viewport"):
page.set_viewport_size({"width": 375, "height": 667})
with allure.step("Navigate to page"):
page.goto("https://example.com")
with allure.step("Verify mobile menu is visible"):
expect(page.locator(".mobile-menu")).to_be_visible()Playwright runs in headless Chromium inside Docker!
Available Fixtures:
page- Playwright Page object (auto-configured)browser- Browser instancecontext- Browser context with custom settings
Best Practices:
- Use
expect()assertions for auto-waiting - Take screenshots at key steps for debugging
- Test on different viewport sizes for responsiveness
- Use explicit locators (
#id,.class,data-testid) - Add
allure.step()for clear test flow - Test error states and edge cases
Common Playwright Actions:
# Navigation
page.goto("https://example.com")
page.go_back()
page.reload()
# Interactions
page.click("button")
page.fill("input[name='email']", "test@test.com")
page.select_option("select", "option-value")
page.check("input[type='checkbox']")
# Assertions
expect(page.locator("h1")).to_have_text("Welcome")
expect(page).to_have_url("https://example.com/success")
expect(page.locator("button")).to_be_enabled()
# Waits
page.wait_for_selector(".loading", state="hidden")
page.wait_for_load_state("networkidle")Maintenance:
- UI tests run in Docker with Playwright pre-installed
- Update Playwright: Edit
docker/test-runner.dockerfile - Add custom fixtures: Edit
tests/conftest.py - Run specific test:
pytest tests/ui/simple/test_my_ui.py::test_navigation -v - Debug with headed mode: Modify
pytest.inito remove--headless
⚡ Writing Performance Tests (k6)
From Web UI:
- Click the "Performance Tests" button in the dashboard
- Tests run automatically in Docker (includes all 3 test types)
- View k6 metrics in the console output
- Quick demo: ~30 seconds total (load: 10s + stress: 10s + spike: 10s)
- Click "View Performance Report" to see the k6 summary
From Command Line:
# Run all k6 tests via Web UI
python start.py # Click "Performance Tests" button
# Or run individual k6 tests directly
cd tests/performance/k6
# Load test (~10 seconds)
npm run test:load
# Stress test (~10 seconds)
npm run test:stress
# Spike test (~10 seconds)
npm run test:spike
# Run all tests sequentially (~30 seconds)
npm run test:allimport http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate } from 'k6/metrics';
// Custom metrics
const errorRate = new Rate('errors');
// Test configuration
export const options = {
stages: [
{ duration: '30s', target: 10 }, // Ramp up to 10 users
{ duration: '1m', target: 10 }, // Stay at 10 users
{ duration: '30s', target: 0 }, // Ramp down to 0
],
thresholds: {
http_req_duration: ['p(95)<500'], // 95% of requests under 500ms
errors: ['rate<0.1'], // Error rate under 10%
},
};
// Test scenario
export default function () {
// Test homepage load
const homeRes = http.get('https://example.com');
check(homeRes, {
'homepage status is 200': (r) => r.status === 200,
'homepage loads in <500ms': (r) => r.timings.duration < 500,
});
errorRate.add(homeRes.status !== 200);
sleep(1);
// Test API endpoint
const apiRes = http.get('https://api.example.com/users');
check(apiRes, {
'API status is 200': (r) => r.status === 200,
'API returns JSON': (r) => r.headers['Content-Type'].includes('application/json'),
'API response time <300ms': (r) => r.timings.duration < 300,
});
errorRate.add(apiRes.status !== 200);
sleep(2);
// Test POST request
const payload = JSON.stringify({
username: 'testuser',
email: 'test@example.com',
});
const postRes = http.post('https://api.example.com/users', payload, {
headers: { 'Content-Type': 'application/json' },
});
check(postRes, {
'POST status is 201': (r) => r.status === 201,
'User created successfully': (r) => {
const body = JSON.parse(r.body);
return body.username === 'testuser';
},
});
sleep(1);
}
// Setup function (runs once at start)
export function setup() {
console.log('Starting performance test...');
return { timestamp: Date.now() };
}
// Teardown function (runs once at end)
export function teardown(data) {
console.log(`Test completed. Started at: ${new Date(data.timestamp)}`);
}{
"scripts": {
"test:custom": "k6 run custom_test.js --out json=../../../reports/k6_custom.json"
}
}Available k6 Tests (Quick Demo Versions):
load_test.js- Gradual ramp-up to 3 users (10 seconds)stress_test.js- Stress testing up to 10 users (10 seconds)spike_test.js- Sudden traffic spikes to 10 users (10 seconds)
k6 Best Practices:
- Define clear thresholds for pass/fail criteria
- Use custom metrics to track specific KPIs
- Test realistic user scenarios, not just endpoints
- Add think time with
sleep()between requests - Group related requests for better reporting
- Test both read and write operations
- Simulate different user types/behaviors
k6 Common Patterns:
// Authentication flow
const loginRes = http.post('https://api.example.com/login', {
username: 'user',
password: 'pass',
});
const token = loginRes.json('token');
// Authenticated requests
const headers = { Authorization: `Bearer ${token}` };
http.get('https://api.example.com/profile', { headers });
// Batch requests
const responses = http.batch([
['GET', 'https://example.com/api/users'],
['GET', 'https://example.com/api/products'],
['GET', 'https://example.com/api/orders'],
]);
// Custom metrics
import { Trend, Counter } from 'k6/metrics';
const loginDuration = new Trend('login_duration');
const failedLogins = new Counter('failed_logins');
loginDuration.add(loginRes.timings.duration);
if (loginRes.status !== 200) {
failedLogins.add(1);
}Maintenance:
- Performance tests run in Docker with k6 pre-installed
- Update k6 version: Edit
docker/test-runner.dockerfile - Adjust test duration: Modify
stagesin test files - Change thresholds: Update
thresholdsobject - Add new test: Create
.jsfile and add npm script - View detailed metrics: Check
reports/k6_*.jsonfiles
k6 Thresholds Reference:
http_req_duration: Request duration (p95, p99, avg, max)http_req_failed: Failed request ratehttp_reqs: Total requests per seconditerations: Number of scenario iterationsvus: Number of virtual users
Note: For database query performance, use pytest (see Database Tests section).
🗄️ Writing Database Tests
import pytest
import allure
import requests
import time
@allure.epic("Database Testing")
@allure.feature("User Management")
@pytest.mark.database
class TestMyDatabase:
@allure.title("Test user CRUD operations")
@allure.description("Verify complete CRUD workflow for user management")
@allure.severity(allure.severity_level.CRITICAL)
def test_user_crud(self, mock_server_url):
# Create
with allure.step("Create new user"):
new_user = {
"username": "john_doe",
"email": "john@example.com",
"first_name": "John",
"last_name": "Doe"
}
response = requests.post(
f"{mock_server_url}/api/users",
json=new_user
)
assert response.status_code == 201
user = response.json()
user_id = user['id']
assert user['username'] == new_user['username']
# Read
with allure.step("Fetch created user"):
response = requests.get(f"{mock_server_url}/api/users/{user_id}")
assert response.status_code == 200
fetched_user = response.json()
assert fetched_user['username'] == 'john_doe'
assert fetched_user['email'] == 'john@example.com'
# Update
with allure.step("Update user information"):
update_data = {
"first_name": "Jonathan",
"last_name": "Doe-Smith"
}
response = requests.put(
f"{mock_server_url}/api/users/{user_id}",
json=update_data
)
assert response.status_code == 200
with allure.step("Verify update was successful"):
response = requests.get(f"{mock_server_url}/api/users/{user_id}")
updated_user = response.json()
assert updated_user['first_name'] == "Jonathan"
assert updated_user['last_name'] == "Doe-Smith"
# Delete
with allure.step("Delete user"):
response = requests.delete(f"{mock_server_url}/api/users/{user_id}")
assert response.status_code == 200
with allure.step("Verify user was deleted"):
response = requests.get(f"{mock_server_url}/api/users/{user_id}")
assert response.status_code == 404
@allure.title("Test data validation")
def test_data_validation(self, mock_server_url):
with allure.step("Attempt to create user with invalid email"):
invalid_user = {
"username": "testuser",
"email": "not-an-email"
}
response = requests.post(
f"{mock_server_url}/api/users",
json=invalid_user
)
assert response.status_code in [400, 422]
with allure.step("Attempt to create user with duplicate username"):
response = requests.post(
f"{mock_server_url}/api/users",
json={"username": "admin", "email": "new@example.com"}
)
# Should either reject or handle gracefully
assert response.status_code in [200, 201, 409]
@allure.title("Test query performance")
def test_query_performance(self, mock_server_url):
with allure.step("Fetch all users and measure response time"):
start_time = time.time()
response = requests.get(f"{mock_server_url}/api/users")
elapsed_time = time.time() - start_time
assert response.status_code == 200
assert elapsed_time < 0.5 # Should respond in under 500ms
allure.attach(
f"Query time: {elapsed_time:.3f}s",
name="Performance Metrics",
attachment_type=allure.attachment_type.TEXT
)
with allure.step("Test pagination performance"):
start_time = time.time()
response = requests.get(
f"{mock_server_url}/api/users",
params={"page": 1, "limit": 10}
)
elapsed_time = time.time() - start_time
assert response.status_code == 200
assert elapsed_time < 0.3 # Paginated query should be faster
@allure.title("Test concurrent operations")
def test_concurrent_operations(self, mock_server_url):
import concurrent.futures
def create_user(index):
return requests.post(
f"{mock_server_url}/api/users",
json={
"username": f"user_{index}",
"email": f"user{index}@example.com"
}
)
with allure.step("Create 10 users concurrently"):
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
futures = [executor.submit(create_user, i) for i in range(10)]
responses = [f.result() for f in concurrent.futures.as_completed(futures)]
with allure.step("Verify all users were created successfully"):
successful = sum(1 for r in responses if r.status_code == 201)
assert successful >= 8 # Allow some failures in concurrent scenarioWhat's tested:
- CRUD operations (Create, Read, Update, Delete)
- Data validation & constraints
- Query performance benchmarks
- Concurrent operations
- Database statistics
Mock Database:
- SQLite in-memory (zero setup!)
- 15 sample users + 20 products
- Tests use HTTP API (no direct SQL)
- Automatically starts Docker containers (mock-server on port 8080)
How it works:
- Click "Database Tests" button → Docker containers start automatically
- Tests connect to mock-server with SQLite database
- Tests execute (CRUD, validation, performance)
- Results appear in Allure report
- Containers shut down automatically
Best Practices:
- Test complete CRUD workflows, not just individual operations
- Validate data constraints and error handling
- Measure query performance with benchmarks
- Test edge cases (empty strings, null values, duplicates)
- Use
allure.attach()to log performance metrics - Clean up test data after each test (or use transactions)
Common Database Test Patterns:
# Testing relationships
def test_user_orders(mock_server_url):
user = create_test_user(mock_server_url)
order = create_test_order(mock_server_url, user['id'])
# Verify relationship
orders = get_user_orders(mock_server_url, user['id'])
assert len(orders) == 1
assert orders[0]['id'] == order['id']
# Testing transactions
def test_transaction_rollback(mock_server_url):
# This would require direct DB access or special endpoints
# to test transaction handling
pass
# Testing data integrity
def test_cascade_delete(mock_server_url):
user = create_test_user(mock_server_url)
create_test_order(mock_server_url, user['id'])
# Delete user
delete_user(mock_server_url, user['id'])
# Verify orders were also deleted (cascade)
orders = get_user_orders(mock_server_url, user['id'])
assert len(orders) == 0Maintenance:
- Database tests run in Docker with mock SQLite server
- Update mock data: Edit
src/services/mock_server.py - Add new endpoints: Update mock server routes
- Run specific test:
pytest tests/integration/database/test_my_database.py -v - Performance benchmarks: Adjust thresholds based on your requirements
- For real database: See Database Integration Guide
Performance Testing Tips:
- Use
time.time()for basic performance measurements - Set realistic thresholds (e.g., <500ms for queries)
- Test with realistic data volumes
- Measure both individual queries and batch operations
- Track performance trends over time using Allure history
Want to test your real database? 📖 See Database Integration Guide
🔌 Connecting Your Personal Database
The framework includes a mock SQLite database that works out-of-the-box with zero setup. To test your real database (PostgreSQL, MySQL, MongoDB, etc.), follow these steps:
Point tests to your existing API that connects to your database:
# tests/conftest.py
@pytest.fixture(scope='session')
def mock_server_url():
# Change this to your real API URL
return "https://your-api.example.com"Benefits:
- No new dependencies required
- Tests realistic production workflows
- No database credentials in test code
- Existing database tests work immediately
For advanced database testing with direct SQL queries:
1. Install database driver:
# PostgreSQL
pip install psycopg2-binary
# MySQL
pip install pymysql
# MongoDB
pip install pymongo2. Create database configuration:
# config/database.py
import os
DATABASE_CONFIG = {
'host': os.getenv('DB_HOST', 'localhost'),
'port': int(os.getenv('DB_PORT', '5432')),
'database': os.getenv('DB_NAME', 'test_db'),
'user': os.getenv('DB_USER', 'postgres'),
'password': os.getenv('DB_PASSWORD', 'your_password')
}3. Create .env file (never commit this!):
DB_HOST=localhost
DB_PORT=5432
DB_NAME=test_database
DB_USER=test_user
DB_PASSWORD=your_secure_password4. Add .env to .gitignore:
echo ".env" >> .gitignore
echo "*.env" >> .gitignore5. Create database fixtures:
# tests/conftest.py
import psycopg2
from config.database import DATABASE_CONFIG
@pytest.fixture(scope='session')
def db_connection():
conn = psycopg2.connect(**DATABASE_CONFIG)
yield conn
conn.close()
@pytest.fixture
def db_cursor(db_connection):
cursor = db_connection.cursor()
yield cursor
db_connection.rollback() # Cleanup after each test
cursor.close()6. Write direct SQL tests:
def test_direct_query(db_cursor):
db_cursor.execute("SELECT COUNT(*) FROM users")
count = db_cursor.fetchone()[0]
assert count > 0Add a database container for isolated testing:
# docker/docker-compose.yml
services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_DB: test_database
POSTGRES_USER: test_user
POSTGRES_PASSWORD: test_password
ports:
- "5432:5432"
volumes:
- ./init-scripts:/docker-entrypoint-initdb.d📖 For detailed instructions, examples, and troubleshooting, see: Database Integration Guide
Includes:
- Step-by-step PostgreSQL/MySQL/MongoDB setup
- Connection pooling for performance
- Test data management strategies
- Transaction handling & cleanup
- Common issues & solutions
📊 Test History & Trends
- First run: Creates baseline in
reports/latest/history/ - Subsequent runs: Preserves previous history automatically
- Allure: Displays trends over multiple runs
After running tests multiple times, check the Allure report:
- 📈 Duration Trend
- ✅ Success Rate Trend
- 🔄 Retry Trend
- 📊 Categories Trend
# Windows
rmdir /s /q reports\latest
# Linux/Mac
rm -rf reports/latest🐳 Docker Commands
# Rebuild test-runner
docker-compose -f docker/docker-compose.yml build test-runner
# Rebuild mock-server
docker-compose -f docker/docker-compose.yml build mock-server
# Rebuild everything
docker-compose -f docker/docker-compose.yml build --no-cache# Stop all containers
docker-compose -f docker/docker-compose.yml down
# Remove images & volumes
docker-compose -f docker/docker-compose.yml down --rmi all --volumesPort 5000 already in use
The dashboard auto-tries ports 5000-5009. If all are busy:
# Windows
netstat -ano | findstr :5000
taskkill /F /PID <PID>
# Linux/Mac
lsof -ti:5000 | xargs kill -9Tests won't start
- ✅ Check Docker is running:
docker ps - 🧹 Click "Cleanup" button in dashboard
- 🔄 Restart: Press
Ctrl+C, thenpython start.py
Virtual environment issues
Delete venv and restart:
# Windows
rmdir /s /q venv && python start.py
# Linux/Mac
rm -rf venv && python start.pyComplete reset
Nuclear option - start from scratch:
# 1. Remove venv
rm -rf venv # or: rmdir /s /q venv (Windows)
# 2. Remove Docker
docker-compose -f docker/docker-compose.yml down --rmi all --volumes
# 3. Restart
python start.py
|
|
GitHub Actions Example
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Run tests
run: |
python start.py &
sleep 10
pkill -f start.py
source venv/bin/activate
python run_tests.py
- name: Upload Allure report
uses: actions/upload-artifact@v3
if: always()
with:
name: allure-report
path: reports/archive/python start.pyMade with ❤️ by Timi953