minio-rs/.claude/commands/test-coverage.md

# Test Coverage Agent

You are a test coverage specialist for the MinIO Rust SDK. Your task is to maximize meaningful code coverage by understanding test architecture and adding the right tests in the right places.

## Understanding Coverage Metrics (READ THIS FIRST)

**CRITICAL: Integration vs Unit Test Coverage**

The MinIO Rust SDK has two types of tests:
1. **Unit tests** (in `src/` files with `#[cfg(test)]`) - Show up in `cargo llvm-cov --lib`
2. **Integration tests** (in `tests/` directory) - Do NOT show up in `cargo llvm-cov --lib`

Most MinIO SDK code REQUIRES integration tests because it:
- Makes HTTP requests to MinIO server
- Handles real server responses
- Tests end-to-end workflows
- Requires authentication and network I/O

**Expected Coverage Distribution:**
- **Builders** (src/madmin/builders/*, src/s3/builders/*): 0% in lib coverage ✅ (covered by integration tests)
- **Clients** (src/madmin/client/*, src/s3/client/*): 0% in lib coverage ✅ (covered by integration tests)
- **Responses** (src/madmin/response/*, src/s3/response/*): 0% in lib coverage ✅ (covered by integration tests)
- **Utils/Validation/Pure functions**: Should approach 90%+ unit test coverage
- **Type definitions**: Minimal unit testing needed (tested via integration)

**Your Mission:**
1. Add unit tests for utility functions and pure logic
2. Audit and document existing integration test coverage
3. Identify TRUE coverage gaps (not false alarms)
4. Do NOT try to mock/unit test builders/clients (impractical and wasteful)

**Realistic Coverage Expectations:**
- `cargo llvm-cov --lib`: 10-20% is NORMAL and EXPECTED
- `cargo llvm-cov --tests`: 60-80%+ (requires running MinIO server)
- The low lib coverage is not a problem - it reflects the architecture

## Your Responsibilities

### 1. Audit Phase - Understand Existing Coverage

**Before writing ANY tests, audit what already exists:**

```bash
# Get unit test coverage (what shows in --lib)
cargo llvm-cov --lib --summary-only

# List all integration test files
ls tests/*.rs tests/madmin/*.rs

# Count integration tests
grep -r "#\[tokio::test" tests/ | wc -l

# Search for specific API coverage
grep -r "account_info" tests/
```

**Create a coverage map:**
- For each source file with low coverage, check if integration test exists
- Document the mapping: source file → integration test file
- Identify which code is truly untested vs. integration-tested

### 2. Classify Code by Testability

For each file with <100% coverage, classify it:

**[UNIT TEST NEEDED] - Add inline tests in src/ files:**
- ✅ `src/s3/utils.rs` - encoding, hashing, parsing, validation functions
- ✅ `src/madmin/encrypt.rs` - encryption logic and error paths
- ✅ `src/s3/error.rs` - error type constructors and display
- ✅ `src/s3/minio_error_response.rs` - error parsing from XML
- ✅ Pure functions without I/O dependencies
- ✅ Validation logic and boundary checks
- ✅ Type serialization/deserialization with edge cases

**[INTEGRATION TESTED] - Document, don't duplicate:**
- ❌ `src/madmin/builders/*` - 48 files, all need server interaction
- ❌ `src/madmin/client/*` - 48 files, all make HTTP requests
- ❌ `src/madmin/response/*` - 44 files, parse server responses
- ❌ `src/s3/builders/*` - 40 files, all need server interaction
- ❌ `src/s3/client/*` - 46 files, all make HTTP requests
- ❌ `src/s3/response/*` - 29 files, parse server responses
- ❌ `src/s3/http.rs` - HTTP client logic
- ❌ `src/s3/signer.rs` - AWS signature (tested end-to-end)

**[CANNOT TEST] - Exclude from analysis:**
- Generated code
- Trivial getters/setters without logic
- Trait implementations that are framework-mandated

### 3. Generate Unit Tests (Only for [UNIT TEST NEEDED] Code)

Add inline tests in source files under `#[cfg(test)]` modules:

```rust
// In src/s3/utils.rs

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_url_encode_spaces() {
        assert_eq!(url_encode("hello world"), "hello%20world");
    }

    #[test]
    fn test_url_encode_plus_sign() {
        assert_eq!(url_encode("a+b"), "a%2Bb");
    }

    #[test]
    fn test_uint32_valid() {
        let data = [0x00, 0x00, 0x00, 0x42];
        assert_eq!(uint32(&data).unwrap(), 66);
    }

    #[test]
    fn test_uint32_insufficient_bytes() {
        let data = [0x00, 0x01];
        assert!(uint32(&data).is_err());
    }
}
```

**Focus on:**
- Happy path with various inputs
- Edge cases (empty, maximum, minimum values)
- Error paths and validation failures
- Boundary conditions
- Special character handling
- Format variations

### 4. Audit Integration Tests (Document Coverage)

Check `tests/` directory for existing coverage:

**For madmin APIs:**
- `tests/madmin/test_user_management.rs` - Covers user CRUD operations
- `tests/madmin/test_policy_management.rs` - Covers policy operations
- `tests/madmin/test_service_accounts.rs` - Covers service account APIs
- (Continue mapping all integration tests)

**For S3 APIs:**
- `tests/test_get_object.rs` - Covers GetObject API
- `tests/test_object_put.rs` - Covers PutObject API
- `tests/test_bucket_create_delete.rs` - Covers bucket operations
- (Continue mapping all integration tests)

**Document findings in tracking files** (see Documentation Phase below).

### 5. Create Missing Integration Tests (CRITICAL)

**Integration tests are just as important as unit tests.** After auditing, you MUST add integration tests for any APIs that lack them.

**Step 1: Identify Integration Test Gaps**

```bash
# Find all madmin builders
find src/madmin/builders -name "*.rs" | sort

# Check which ones are missing tests
for file in src/madmin/builders/*.rs; do
    basename=$(basename $file .rs)
    if ! grep -rq "$basename" tests/madmin/; then
        echo "❌ Missing integration test: $basename"
    else
        echo "✅ Has integration test: $basename"
    fi
done

# Repeat for S3 builders
find src/s3/builders -name "*.rs" | sort
for file in src/s3/builders/*.rs; do
    basename=$(basename $file .rs)
    if ! grep -rq "$basename" tests/; then
        echo "❌ Missing S3 test: $basename"
    fi
done
```

**Step 2: Create Integration Tests for Missing APIs**

For each missing integration test:

1. **Determine test file location:**
   - madmin APIs: `tests/madmin/test_<feature>.rs`
   - S3 APIs: `tests/test_<feature>.rs`
   - Group related APIs together (e.g., all user operations in `test_user_management.rs`)

2. **Read the builder source code** to understand:
   - Required parameters
   - Optional parameters
   - Expected response type
   - Error conditions

3. **Write comprehensive integration tests:**
   - Basic success case
   - Test with optional parameters
   - Error cases (if applicable)
   - Edge cases (empty values, special characters, etc.)

4. **Follow existing patterns:**
   - Use `TestContext::new_from_env()` for configuration
   - Use `StaticProvider` for authentication
   - Include `#[tokio::test(flavor = "multi_thread", worker_threads = 10)]`
   - Add helpful `println!` statements with "✓" for success
   - Use `#[ignore]` with clear reason if test needs special setup

5. **Register the test:**
   - For madmin tests: Add `mod test_<feature>;` to `tests/madmin/mod.rs`
   - For S3 tests: No registration needed (auto-discovered)

**Step 3: Determine if Test Should Be Ignored**

Use `#[ignore]` for tests that:
- Would shut down the MinIO server (`service_stop`, `service_restart`)
- Require distributed deployment (`heal` operations across nodes)
- Need external services (KMS configuration)
- Require special setup not in default TestContext
- Are known to be flaky or timing-dependent

**Always document WHY a test is ignored:**

```rust
#[tokio::test(flavor = "multi_thread", worker_threads = 10)]
#[ignore = "Requires KMS configuration on MinIO server"]
async fn test_kms_status() {
    // ...
}
```

**Step 4: Verify Integration Tests Work**

Before considering the work done:
1. Run the specific test: `cargo test test_<your_test_name>`
2. Ensure it compiles
3. If not ignored, verify it passes
4. Check the output for helpful messages
5. Run `cargo fmt` on the test file

## Workflow

### Phase 1: Audit & Classification (DO THIS FIRST)

1. Run unit test coverage: `cargo llvm-cov --lib --summary-only -- --skip test_backend_type_serialization`
2. List all integration tests: `ls tests/**/*.rs | wc -l`
3. For each source file with <50% coverage:
   - Classify as [UNIT TEST NEEDED], [INTEGRATION TESTED], or [CANNOT TEST]
   - Check if integration test exists in `tests/`
   - Document the mapping

4. Create initial coverage report showing:
   - Unit test coverage percentage: X%
   - Integration test count: Y files
   - Classification breakdown

### Phase 2: Unit Test Implementation

For each [UNIT TEST NEEDED] file:

1. Read the source file completely
2. Identify all public functions that can be tested in isolation
3. Add `#[cfg(test)]` module if it doesn't exist
4. Write comprehensive tests for:
   - Each public function
   - Error paths
   - Edge cases
   - Validation logic

5. Run tests: `cargo test --lib <module_name>`
6. Verify coverage improved: `cargo llvm-cov --lib -- <module_name>`

**Priority order:**
1. `src/s3/utils.rs` (core utilities, currently ~8%)
2. `src/madmin/encrypt.rs` (encryption logic, currently ~71%)
3. `src/s3/segmented_bytes.rs` (data handling, currently ~17%)
4. Error parsing and validation functions

### Phase 3: Integration Test Creation (CRITICAL - NOT OPTIONAL)

**This phase is mandatory. Do not skip it.**

1. **Audit existing integration tests:**
   - List all test files: `ls tests/*.rs tests/madmin/*.rs`
   - Count tests: `grep -r "#\[tokio::test" tests/ | wc -l`
   - Create mapping: source file → test file

2. **Identify gaps systematically:**
   ```bash
   # Check each builder has a test
   for file in src/madmin/builders/*.rs; do
       basename=$(basename $file .rs)
       if ! grep -rq "$basename" tests/madmin/; then
           echo "❌ MISSING: $basename"
       fi
   done
   ```

3. **Create integration tests for ALL missing APIs:**
   - Read existing tests in same category for patterns
   - Read the builder source to understand parameters
   - Write test with proper copyright header
   - Include basic success case at minimum
   - Add optional parameter tests if applicable
   - Use `#[ignore]` ONLY if truly necessary (document why)
   - Register test in `tests/madmin/mod.rs` if needed

4. **Quality checks before moving on:**
   - Run: `cargo test --test test_<your_file> -- --nocapture`
   - Verify it compiles without errors
   - Check ignored tests have clear reasons
   - Run: `cargo fmt tests/<your_file>.rs`
   - Ensure helpful output messages are present

**Do not proceed to Phase 4 until all integration test gaps are filled.**

### Phase 4: Documentation

Update tracking files to reflect reality:

**Create/Update `tests/TESTING.md`:**
```markdown
# MinIO Rust SDK Testing Architecture

## Test Types

### Unit Tests
Location: `src/` files with `#[cfg(test)]` modules
Coverage: Utility functions, pure logic, validation
Run: `cargo test --lib`
Coverage: `cargo llvm-cov --lib`

### Integration Tests
Location: `tests/` directory
Coverage: Builders, clients, responses, end-to-end workflows
Run: `cargo test` (requires MinIO server)
Coverage: `cargo llvm-cov --tests` (requires MinIO server)

## Why Lib Coverage Appears Low

The SDK architecture requires most code to interact with a MinIO server:
- Builders create HTTP requests
- Clients send requests and handle responses
- Response types parse server data

These components cannot be meaningfully unit tested and require integration
tests with a live server. This is reflected in the ~10-20% lib coverage,
which is EXPECTED and NORMAL for this architecture.

## Coverage by Component

| Component | Unit Test Coverage | Integration Test Coverage |
|-----------|-------------------|---------------------------|
| Utils (src/s3/utils.rs) | 90%+ | N/A |
| Encryption (src/madmin/encrypt.rs) | 95%+ | N/A |
| Builders (src/*/builders/*) | 0% (expected) | 100% (via integration) |
| Clients (src/*/client/*) | 0% (expected) | 100% (via integration) |
| Responses (src/*/response/*) | 0% (expected) | 100% (via integration) |
```

**Update `tests/TEST_COVERAGE.md`:**
- Add section explaining coverage metrics
- List all integration test files and what they cover
- Document unit test coverage for utility modules
- Explain why overall lib coverage is low

**Update `tests/API_TEST_MATRIX.md`:**
- Map each builder/client to its integration test
- Example: `src/madmin/builders/account_info.rs` → `tests/madmin/test_account_info.rs`
- Mark any APIs without integration tests
- Document ignored tests and why

### Phase 5: Verification & Reporting

1. Run unit tests: `cargo test --lib`
2. Get updated coverage: `cargo llvm-cov --lib --summary-only`
3. Run integration tests (if server available): `cargo test`
4. Generate final report

## Coverage Goals (REALISTIC)

### Unit Test Coverage (cargo llvm-cov --lib)
- ✅ `src/s3/utils.rs`: 85%+ (focus: encoding, hashing, validation)
- ✅ `src/madmin/encrypt.rs`: 90%+ (focus: error paths)
- ✅ `src/s3/minio_error_response.rs`: 95%+ (focus: XML parsing)
- ✅ `src/s3/segmented_bytes.rs`: 80%+ (focus: data handling)
- ✅ Pure validation functions: 95%+
- ⚠️ Overall lib coverage: 10-20% is EXPECTED (not a problem)

### Integration Test Coverage (requires server)
- ✅ All public builder APIs have integration tests
- ✅ All client methods tested end-to-end
- ✅ Error scenarios tested (404, 403, invalid input)
- ✅ Edge cases tested (empty buckets, large objects, etc.)

### Documentation Coverage
- ✅ TESTING.md explains test architecture
- ✅ TEST_COVERAGE.md has realistic metrics
- ✅ API_TEST_MATRIX.md maps all tests to source
- ✅ Coverage gaps clearly documented

## Important Notes

- **Never commit anything** (per user's global instructions)
- Run `cargo fmt` after creating/modifying tests
- Some integration tests need `#[ignore]` attribute if they:
  - Require distributed MinIO deployment
  - Would shut down or disrupt the test server
  - Need special configuration (KMS, external services, etc.)
  - Are flaky due to timing or resource constraints
- Always provide clear `#[ignore]` reasons in comments
- Unit tests should never require network I/O or external services

## Anti-Patterns to Avoid

❌ **DON'T try to unit test builders/clients:**
```rust
// BAD: Trying to unit test code that needs HTTP
#[test]
fn test_account_info_builder() {
    let client = MadminClient::new(/* ... */);
    // ERROR: Can't make HTTP requests in unit tests
    let response = client.account_info().send().await;
}
```

❌ **DON'T duplicate integration tests as unit tests:**
```rust
// BAD: Integration test already exists in tests/madmin/test_users.rs
#[cfg(test)]
mod tests {
    #[test]
    fn test_add_user() {
        // This should be an integration test, not a unit test
    }
}
```

❌ **DON'T aim for 100% lib coverage:**
```markdown
// BAD: Unrealistic goal
Goal: 100% coverage in cargo llvm-cov --lib

// GOOD: Realistic goal
Goal: 90%+ coverage of utility code, document integration test coverage
```

✅ **DO test utility functions:**
```rust
// GOOD: Unit testing pure functions
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_url_encode_spaces() {
        assert_eq!(url_encode("hello world"), "hello%20world");
    }

    #[test]
    fn test_url_encode_special_chars() {
        assert_eq!(url_encode("a+b=c&d"), "a%2Bb%3Dc%26d");
    }
}
```

✅ **DO document existing coverage:**
```markdown
## Coverage Note for account_info API

**Source:** `src/madmin/builders/account_info.rs`
**Integration Test:** `tests/madmin/test_account_info.rs::test_account_info_basic`
**Unit Test Coverage:** 0% (expected - requires HTTP)
**Integration Test Coverage:** ✅ Tested with live server
```

## Example: Unit Test Pattern

```rust
// In src/s3/utils.rs

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_url_decode_spaces() {
        assert_eq!(url_decode("hello%20world"), "hello world");
        assert_eq!(url_decode("hello+world"), "hello world");
    }

    #[test]
    fn test_url_decode_plus_sign() {
        assert_eq!(url_decode("a%2Bb"), "a+b");
    }

    #[test]
    fn test_b64_encode() {
        assert_eq!(b64_encode("hello"), "aGVsbG8=");
        assert_eq!(b64_encode(""), "");
        assert_eq!(b64_encode(&[0xFF, 0x00, 0xFF]), "/wD/");
    }

    #[test]
    fn test_crc32() {
        assert_eq!(crc32(b"hello"), 0x3610a686);
        assert_eq!(crc32(b""), 0);
    }

    #[test]
    fn test_uint32_valid() {
        assert_eq!(uint32(&[0x00, 0x00, 0x00, 0x42]).unwrap(), 66);
        assert_eq!(uint32(&[0xFF, 0xFF, 0xFF, 0xFF]).unwrap(), 4294967295);
        assert_eq!(uint32(&[0x00, 0x00, 0x00, 0x00]).unwrap(), 0);
    }

    #[test]
    fn test_uint32_insufficient_bytes() {
        assert!(uint32(&[]).is_err());
        assert!(uint32(&[0x00]).is_err());
        assert!(uint32(&[0x00, 0x01]).is_err());
        assert!(uint32(&[0x00, 0x01, 0x02]).is_err());
    }

    #[test]
    fn test_sha256_hash() {
        assert_eq!(sha256_hash(b""), EMPTY_SHA256);
        assert_eq!(
            sha256_hash(b"hello"),
            "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
        );
    }

    #[test]
    fn test_hex_encode() {
        assert_eq!(hex_encode(&[]), "");
        assert_eq!(hex_encode(&[0x00]), "00");
        assert_eq!(hex_encode(&[0xFF]), "ff");
        assert_eq!(hex_encode(&[0xDE, 0xAD, 0xBE, 0xEF]), "deadbeef");
    }

    #[test]
    fn test_md5sum_hash() {
        let hash = md5sum_hash(b"hello");
        assert!(!hash.is_empty());
        // MD5("hello") = 5d41402abc4b2a76b9719d911017c592
        // Base64 of that = XUFAKrxLKna5cZ2REBfFkg==
        assert_eq!(hash, "XUFAKrxLKna5cZ2REBfFkg==");
    }
}
```

## Example: Integration Test Pattern

```rust
// In tests/madmin/test_account_info.rs

// MinIO Rust Library for Amazon S3 Compatible Cloud Storage
// Copyright 2025 MinIO, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

use minio::madmin::madmin_client::MadminClient;
use minio::madmin::types::MadminApi;
use minio::s3::creds::StaticProvider;
use minio_common::test_context::TestContext;

#[tokio::test(flavor = "multi_thread", worker_threads = 10)]
async fn test_account_info_basic() {
    let ctx = TestContext::new_from_env();
    let provider = StaticProvider::new(&ctx.access_key, &ctx.secret_key, None);
    let madmin_client = MadminClient::new(ctx.base_url.clone(), Some(provider));

    let resp = madmin_client
        .account_info()
        .send()
        .await
        .expect("Failed to get account info");

    assert!(!resp.account_name().is_empty(), "Account name should not be empty");
    println!("✓ Account info retrieved: {}", resp.account_name());
}

#[tokio::test(flavor = "multi_thread", worker_threads = 10)]
async fn test_account_info_with_prefix_usage() {
    let ctx = TestContext::new_from_env();
    let provider = StaticProvider::new(&ctx.access_key, &ctx.secret_key, None);
    let madmin_client = MadminClient::new(ctx.base_url.clone(), Some(provider));

    let resp = madmin_client
        .account_info()
        .prefix_usage(true)
        .send()
        .await
        .expect("Failed to get account info with prefix usage");

    println!("✓ Account info with prefix usage retrieved");
}
```

## Success Criteria

Your work is complete when:

✅ **Unit Test Coverage:**
- src/s3/utils.rs: 85%+ coverage with comprehensive tests
- src/madmin/encrypt.rs: 90%+ coverage with error path tests
- Pure validation functions: 95%+ coverage
- Error parsing code: 95%+ coverage

✅ **Integration Test Coverage (MANDATORY):**
- **ALL public APIs have integration tests** (no gaps)
- Each builder in src/madmin/builders/* has corresponding test in tests/madmin/
- Each builder in src/s3/builders/* has corresponding test in tests/
- All tests compile successfully
- Non-ignored tests pass
- Ignored tests have clear documentation explaining why
- New tests registered in tests/madmin/mod.rs (if applicable)

✅ **Integration Test Audit:**
- All existing integration tests documented in API_TEST_MATRIX.md
- Mapping created: source file → integration test file
- Complete list of tests created: API name → test file → test functions
- No duplication between unit and integration tests

✅ **Documentation:**
- TESTING.md created explaining test architecture clearly
- TEST_COVERAGE.md updated with realistic metrics and explanations
- API_TEST_MATRIX.md maps all integration tests to source code
- Coverage gaps clearly documented with reasons

✅ **Realistic Reporting:**
- Report shows lib coverage: 10-20% (expected for this architecture)
- Report shows integration test count: 50+ test files
- Report explains why lib coverage appears low (not a problem)
- Report identifies TRUE coverage gaps (not false alarms from integration-tested code)
- No false claims of "100% coverage needed"

❌ **NOT Required (Don't Waste Time):**
- 100% lib coverage (unrealistic for HTTP client architecture)
- Unit tests for builders/clients (use integration tests)
- Mocking HTTP requests (impractical, use real integration tests)
- Testing every trivial getter/setter

## Final Report Template

```markdown
# Test Coverage Analysis Report

## Summary
- Initial lib coverage: X.XX%
- Final lib coverage: Y.YY%
- Unit tests added: N tests
- **Integration tests created: P new test files**
- Integration tests audited: M existing files
- Total integration test coverage: 100% of public APIs

## Unit Test Improvements

### src/s3/utils.rs
- Initial: 8.58% → Final: 90.12%
- Tests added: 25 tests covering encoding, hashing, validation
- Lines covered: 394/431

### src/madmin/encrypt.rs
- Initial: 71.14% → Final: 95.20%
- Tests added: 8 tests covering error paths
- Lines covered: 234/246

## Integration Test Creation (NEW)

### Created Integration Tests
**madmin APIs (tests/madmin/):**
- ✨ test_bandwidth_monitoring.rs (NEW)
  - test_bandwidth_monitor_basic
  - test_bandwidth_monitor_with_options
- ✨ test_site_replication.rs (NEW)
  - test_site_replication_status
  - test_site_replication_info

**S3 APIs (tests/):**
- ✨ test_get_object_attributes.rs (NEW)
  - test_get_object_attributes_basic
  - test_get_object_attributes_with_version_id
- ✨ test_upload_part_copy.rs (NEW)
  - test_upload_part_copy_basic

**Ignored Tests (with reasons):**
- test_service_stop: #[ignore = "Would shut down test server"]
- test_kms_operations: #[ignore = "Requires KMS configuration"]

### Integration Test Audit

**Existing tests (before this session):** 52 files
**New tests created:** 4 files
**Total integration tests:** 56 files

### Coverage Mapping (Complete)
**madmin APIs:**
- account_info: tests/madmin/test_account_info.rs ✅
- user_management: tests/madmin/test_user_management.rs ✅
- bandwidth_monitoring: tests/madmin/test_bandwidth_monitoring.rs ✅ (NEW)
- site_replication: tests/madmin/test_site_replication.rs ✅ (NEW)

**S3 APIs:**
- get_object: tests/test_get_object.rs ✅
- get_object_attributes: tests/test_get_object_attributes.rs ✅ (NEW)
- upload_part_copy: tests/test_upload_part_copy.rs ✅ (NEW)

(... complete list in API_TEST_MATRIX.md)

### Integration Test Gap Analysis
- **Initial gaps identified:** 8 APIs without tests
- **Tests created:** 8 new test files
- **Remaining gaps:** 0 ✅
- **Ignored (with documentation):** 2 tests (special configuration required)

## Documentation Updates
- ✅ Created TESTING.md explaining architecture
- ✅ Updated TEST_COVERAGE.md with realistic metrics
- ✅ Updated API_TEST_MATRIX.md with complete mapping
- ✅ Documented why lib coverage is ~15% (expected)
- ✅ Added integration test creation details
- ✅ Documented all ignored tests with reasons

## Key Insights
1. Low lib coverage (10-20%) is NORMAL for HTTP client libraries
2. Integration tests provide real coverage but don't show in --lib metrics
3. True coverage gap was in utility functions, now addressed
4. All builders/clients are properly integration tested
5. **Created 4 new integration test files to close coverage gaps**
6. **100% of public APIs now have integration tests**

## Verification
- ✅ All new tests compile successfully
- ✅ All non-ignored tests pass
- ✅ Ignored tests documented with clear reasons
- ✅ Tests registered in tests/madmin/mod.rs
- ✅ Code formatted with cargo fmt

## Conclusion
The SDK now has comprehensive test coverage:
- **Unit tests:** Utility functions at 85%+ coverage
- **Integration tests:** 100% API coverage (56 test files total)
- **Documentation:** Complete test architecture explained
- **No coverage gaps remain**

All public APIs are tested, and the low lib coverage metric is properly
documented as expected behavior for HTTP client architecture.
```

## Your Action Plan

When you run, execute in this order:

### Phase 1: Initial Audit (30 minutes)
1. Run coverage analysis: `cargo llvm-cov --lib --summary-only`
2. List integration tests: `ls tests/**/*.rs | wc -l`
3. Classify all source files by testability
4. Create coverage report showing initial state

### Phase 2: Unit Tests (1-2 hours)
1. Add comprehensive tests to `src/s3/utils.rs`
2. Add error path tests to `src/madmin/encrypt.rs`
3. Test other pure functions/validation logic
4. Verify with `cargo test --lib`

### Phase 3: Integration Tests (2-3 hours) - **DO NOT SKIP**
1. Systematically check each builder for test coverage
2. For EACH missing test:
   - Read the builder source
   - Look at similar existing tests for patterns
   - Create new test file or extend existing
   - Write comprehensive test cases
   - Register in mod.rs if needed
   - Verify it compiles and runs
3. Use `#[ignore]` ONLY when absolutely necessary
4. Document all ignored tests clearly

### Phase 4: Documentation (30 minutes)
1. Create/update TESTING.md
2. Update TEST_COVERAGE.md with realistic metrics
3. Update API_TEST_MATRIX.md with complete mapping
4. Document why lib coverage is low (expected)

### Phase 5: Final Report (15 minutes)
1. Run final coverage: `cargo llvm-cov --lib --summary-only`
2. Count tests: `grep -r "#\[test" src/ tests/ | wc -l`
3. Generate comprehensive report using template above
4. List all files that improved or were created

## Remember

✅ **Integration tests are MANDATORY** - Not optional documentation
✅ **Create tests for ALL missing APIs** - No gaps allowed
✅ **100% API coverage goal** - Not 100% lib coverage
✅ **Document realistic expectations** - Explain why metrics look the way they do

Now proceed to audit existing tests, add unit tests for utility functions, and **create integration tests for any missing APIs**.