How to Debug Fiber Deadlocks and Memory Issues in libfiber

Use acl_fiber_ndead() and acl_fiber_memstat() to monitor the dead-fiber cache and memory usage, instrument fiber_swap in c/src/fiber.c and __lock in c/src/sync/fiber_lock.c to detect scheduler stalls and lock contention, and enable FIBER_STACK_GUARD to catch stack overruns.

Debugging concurrency issues in iqiyi/libfiber requires understanding its cooperative-multithreading scheduler. Because libfiber implements user-level context switching, deadlocks and memory leaks manifest differently than in traditional pthread programs, often involving the dead-fiber cache, shared-stack corruption, or stranded lock waiting lists. This guide covers the specific source locations and diagnostic techniques needed to resolve these issues.

Verify the Scheduler State in c/src/fiber.c

Libfiber only switches fibers when acl_fiber_schedule() or acl_fiber_switch() is called. The global __scheduled flag indicates whether the scheduler is active, and the ready queue (__thread_fiber->ready) drives execution.

In c/src/fiber.c, the fiber_swap function (lines 71-73) moves fibers to the dead queue when status == FIBER_STATUS_EXITING. If fibers never return to the ready queue, the scheduler appears frozen.

Symptoms of a stalled scheduler include acl_fiber_ndead() growing while acl_fiber_number() remains constant. Add this diagnostic snippet after acl_fiber_schedule() or in a periodic timer:

printf("scheduled=%d ready=%zu dead=%zu total=%u\n",
       __scheduled,
       ring_size(&__thread_fiber->ready),
       acl_fiber_ndead(),
       acl_fiber_number());

This reveals whether the ready ring is depleting while fibers accumulate in the dead cache. For I/O-related stalls, inspect c/src/fiber_io.c to verify that event loops are correctly waking fibers stuck in FIBER_WAIT_IO.

Detect Lock-Based Deadlocks in c/src/sync/fiber_lock.c

ACL_FIBER_LOCK is a non-recursive mutex implemented in c/src/sync/fiber_lock.c. The __lock function (lines 35-63) acquires the lock by setting curr->wstatus |= FIBER_WAIT_LOCK and placing the fiber on lk->waiting. The unlock path in acl_fiber_lock_unlock (lines 101-124) wakes the first waiter using the FIRST_FIBER macro.

A deadlock occurs when the owner fiber never reaches the unlock path, leaving the waiting list permanently populated. Reader-writer lock deadlocks follow similar patterns in c/src/sync/fiber_rwlock.c.

Instrument the lock state with:

printf("Lock %p owner=%p waiting=%zu\n",
       (void*)lk,
       (void*)lk->owner,
       ring_size(&lk->waiting));

Run the program under Valgrind with --track-origins=yes and enable FIBER_STACK_GUARD (uncomment line 9 in c/src/fiber.c) to detect stack overwrites that corrupt the lock's ring structures. Channel-based deadlocks in c/src/sync/channel.c can be diagnosed similarly by inspecting channel queue lengths.

Diagnose Memory Leaks in Fiber-Local Storage

Fiber-local storage uses acl_fiber_set_specific and acl_fiber_get_specific, implemented in c/src/fiber.c. Each FIBER_LOCAL structure stores a user-supplied free_fn destructor in curr->locals (lines 90-115).

If a fiber exits without invoking these destructors—either because the fiber never reaches fiber_start's cleanup loop or because free_fn is NULL—the memory remains allocated in the dead-fiber cache. The custom allocator in c/src/common/memory.c tracks these allocations via mem_stat.

Monitor nlocal inside fiber_start to catch this, using acl_fiber_id from c/src/common/atomic.c to identify specific fibers:

printf("Fiber %u locals=%d\n", acl_fiber_id(fiber), fiber->nlocal);

Persistent non-zero nlocal values for dead fibers indicate missing destructor calls. Check acl_fiber_memstat() reports to confirm steady allocation growth after fiber termination.

Identify Shared-Stack Corruption

When SHARE_STACK is defined, fibers share a single buffer (__thread_fiber->stack_buff) sized by acl_fiber_set_shared_stack_size (lines 86-93 in c/src/fiber.c). The fiber_share_stack_* helpers (lines 94-108) manage this buffer.

Corruption appears as crashes inside fiber_real_swap. Enable FIBER_STACK_GUARD to trigger OS faults on stack overruns, then verify the guard-region size defined around line 10 in c/src/fiber.c.

Leverage Built-in Diagnostic Functions

Libfiber provides several introspection functions to debug fiber deadlocks and memory issues:

• acl_fiber_ndead(): Returns the number of fibers in the dead-queue cache. If this exceeds MAX_CACHE without decreasing, the cleanup mechanism in thread_free is failing.
• acl_fiber_memstat(): Reports global memory usage tracked by the internal allocator in c/src/common/memory.c.
• acl_fiber_check_timer(max): Starts a watchdog that periodically calls fiber_kick(max) to free up to max dead fibers (see check_timer at line 75 in c/src/fiber.c).
• fiber_kick(max): Forces immediate freeing of cached dead fibers, useful for preventing unbounded cache growth after bursts of fiber creation.

Implement the watchdog in your main function:

int main(void) {
    /* Free up to 50 dead fibers every second */
    acl_fiber_check_timer(50);
    
    for (int i = 0; i < 1000; ++i) {
        acl_fiber_create(worker, NULL, 64000);
    }
    
    acl_fiber_schedule();
    return 0;
}

Summary

• Scheduler stalls: Monitor __scheduled, ring_size(&__thread_fiber->ready), and acl_fiber_ndead() in c/src/fiber.c to detect fibers stuck outside the ready queue.
• Lock deadlocks: Inspect lk->waiting ring size in c/src/sync/fiber_lock.c and verify acl_fiber_lock_unlock is reached; check c/src/sync/fiber_rwlock.c for reader-writer contention.
• Memory leaks: Ensure acl_fiber_set_specific destructors execute in fiber_start and watch acl_fiber_memstat() for growth via c/src/common/memory.c.
• Stack corruption: Enable FIBER_STACK_GUARD in c/src/fiber.c when using SHARE_STACK mode to catch overflows in fiber_real_swap.
• Cache management: Use acl_fiber_check_timer() to prevent MAX_CACHE overflows in long-running applications.

Frequently Asked Questions

How do I know if the libfiber scheduler is stuck?

Check if acl_fiber_ndead() increases while acl_fiber_number() stays constant and ring_size(&__thread_fiber->ready) remains at zero. This indicates fibers are exiting but new fibers aren't being scheduled, usually because acl_fiber_schedule() was never called or fiber_swap is not being invoked due to missing I/O events in c/src/fiber_io.c.

What causes memory leaks in fiber-local storage?

Leaks occur when acl_fiber_set_specific registers data with a free_fn destructor, but the fiber exits without calling it. This happens if the fiber terminates abnormally before reaching the cleanup loop in fiber_start (lines 90-115 in c/src/fiber.c), leaving the allocation in the dead-fiber cache tracked by c/src/common/memory.c.

How can I detect shared-stack corruption?

Enable FIBER_STACK_GUARD (line 9 in c/src/fiber.c) to create a guard page around __thread_fiber->stack_buff. If a fiber overflows its stack slice during context switching in fiber_real_swap, the OS will trigger a segmentation fault immediately rather than corrupting adjacent fiber data in the shared buffer.

Why does the dead-fiber cache keep growing?

The cache grows when fibers die faster than thread_free or fiber_kick can reclaim them. By default, dead fibers are cached up to MAX_CACHE to avoid malloc/free overhead. Use acl_fiber_check_timer(50) to schedule periodic cleanup, or call fiber_kick(max) manually after bursts of fiber creation to force immediate reclamation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}