Graceful Shutdown for Order and Position Management

1. Why Graceful Shutdown Matters

An abrupt termination of our trading framework (e.g., a crash or a forced quit) can lead to serious problems:

• Orphaned Orders: Pending limit orders sent to the broker might remain active, potentially getting filled at an undesirable time when the system is no longer monitoring them.
• Unmonitored Positions: Open positions are left without their managing logic (e.g., stop-loss, profit-take), exposing you to unlimited risk or missed opportunities.
• State Inconsistency: The system might not save its final state, leading to a discrepancy between its internal records and your actual broker account upon restart.
• Resource Leaks: Connections to the broker API, log files, and other system resources might not be closed properly.

A graceful shutdown procedure prevents these issues by ensuring a controlled, orderly teardown of the application.

2. Capturing Shutdown Signals

Our application needs to listen for commands telling it to shut down. These typically come in two forms:

1. SIGINT (Signal Interrupt): Sent when you press Ctrl+C in the terminal.
2. SIGTERM (Signal Terminate): A generic signal used to request termination, often sent by process managers or deployment tools (like Docker or Kubernetes). It's the standard "please shut down cleanly" signal.

Python's built-in signal module allows us to "catch" these OS-level signals and trigger a custom function instead of letting the program terminate immediately.

3. A Centralized, Prioritized Shutdown Manager

Instead of scattering if shutdown_requested: checks throughout our code, we will adopt a professional pattern: a centralized shutdown manager. This class will be responsible for catching signals and executing a series of predefined cleanup tasks in a specific order.

The following article provides an excellent template for a production-grade shutdown handler. While its implementation is based on threading, the concepts of a central shutdown event and prioritized handlers are directly applicable to our asyncio-based framework.

Adapting the Concept for asyncio

The class in the article is designed for multi-threaded applications. Since our framework uses asyncio, we'll create our own version that leverages asyncio.Event and awaits asynchronous cleanup functions. This avoids mixing threading and asyncio complexities.

Here is a simplified, asyncio-native version of the shutdown manager that we will integrate into our Framework class.

# In a new file, e.g., 'framework/shutdown.py'
import asyncio
import signal
import logging
from typing import Callable, List, Coroutine, Tuple

logger = logging.getLogger(__name__)

class AsyncGracefulShutdown:
    """Manages graceful shutdown for an asyncio application."""
    def __init__(self):
        self.shutdown_event = asyncio.Event()
        # Handlers are stored as (priority, function)
        self.handlers: List[Tuple[int, Callable[[], Coroutine]]] = []
        self._setup_signals()

    def _signal_handler(self, signum, frame):
        """Sets the event to trigger shutdown."""
        logger.info(f"Received signal {signal.Signals(signum).name}, initiating graceful shutdown...")
        self.shutdown_event.set()

    def _setup_signals(self):
        """Register signal handlers for SIGINT and SIGTERM."""
        try:
            # This works on Unix-like systems. For Windows, SIGTERM is not available.
            signal.signal(signal.SIGTERM, self._signal_handler)
            signal.signal(signal.SIGINT, self._signal_handler)
        except AttributeError:
            # Handle Windows compatibility where SIGTERM might not be defined
            logger.warning("SIGTERM not available on this OS. Only SIGINT (Ctrl+C) is handled.")
            signal.signal(signal.SIGINT, self._signal_handler)

    def register_handler(self, name: str, handler: Callable[[], Coroutine], priority: int):
        """Registers an async handler function with a given priority."""
        logger.info(f"Registering shutdown handler: {name} (Priority: {priority})")
        self.handlers.append((priority, name, handler))

    async def wait_for_shutdown(self):
        """Coroutine that blocks until the shutdown event is set."""
        await self.shutdown_event.wait()

    async def execute_handlers(self):
        """Executes all registered handlers in descending order of priority."""
        logger.info("Executing shutdown handlers...")
        # Sort by priority, highest first
        self.handlers.sort(key=lambda x: x[0], reverse=True)
        
        for priority, name, handler_coro in self.handlers:
            try:
                logger.info(f"Executing handler '{name}' (Priority: {priority})...")
                await handler_coro()
                logger.info(f"Handler '{name}' completed successfully.")
            except Exception as e:
                logger.error(f"Error in shutdown handler '{name}': {e}", exc_info=True)

4. Defining Our Shutdown Sequence

The order of shutdown operations is critical. You must manage active orders before disconnecting from the broker that manages them. We'll use priorities to enforce this sequence.

1. Priority 30: Stop New Activity. The first step is to prevent the system from initiating any new trades. This means turning off the signal generation logic.
2. Priority 20: Manage Pending Orders. Next, we must address any orders that have been sent to the broker but are not yet filled. A safe default strategy is to cancel all pending orders. This prevents them from being filled after our system has shut down.
3. Priority 10: Manage Open Positions. This is a key decision. Should the system automatically close all open positions upon shutdown?
   • Forced Liquidation: This provides a "clean slate" but can lock in losses or forfeit potential gains.
   • Leave Open: This is often the preferred approach for a continuous trading system. The positions remain with the broker, and the system can re-establish monitoring when it restarts.
     We will implement the "leave open" strategy as the default but design it so it could be configured to liquidate if desired. The primary shutdown task will be to log the state of all open positions.
4. Priority 0: Disconnect & Clean Up. The final step is to cleanly disconnect from the broker API, flush any remaining log messages, and release all other resources.

This pattern of orchestrating component shutdowns is also highlighted in the ForexTradingSystem example, which calls stop() on its various agents in an orderly fashion. Our prioritized approach adds a layer of robustness to this concept.

5. Integrating Graceful Shutdown into the Framework

Now, let's wire our AsyncGracefulShutdown manager into the main Framework class.

Step 1: Initialize the Shutdown Manager

In your Framework class, instantiate the manager and create a method to register the handlers from your different components.

# In framework.py
from .shutdown import AsyncGracefulShutdown

class Framework:
    def __init__(self, config_path: str):
        # ... other initializations ...
        self.shutdown_manager = AsyncGracefulShutdown()
        self._register_shutdown_handlers()

    def _register_shutdown_handlers(self):
        # The priority values enforce our desired shutdown order.
        self.shutdown_manager.register_handler(
            name="SignalGenerator",
            handler=self.signal_generator.stop, # Assume SignalGenerator has an async stop() method
            priority=30
        )
        self.shutdown_manager.register_handler(
            name="ExecutionHandler-CancelOrders",
            handler=self.execution_handler.cancel_all_pending_orders,
            priority=20
        )
        self.shutdown_manager.register_handler(
            name="Portfolio-LogPositions",
            handler=self.portfolio.log_final_positions, # A new method to log open positions
            priority=10
        )
        self.shutdown_manager.register_handler(
            name="Broker-Disconnect",
            handler=self.connection_manager.disconnect,
            priority=0
        )

Step 2: Update the Main Event Loop

Modify Framework.run() to wait for the shutdown signal instead of running in an infinite loop. We also need to handle the .stop_request file from our CLI. We'll do this with a separate, lightweight task that checks for the file and sets the shutdown event.

# In Framework class (framework.py)

    async def _check_stop_file_task(self):
        """Polls for the stop file and triggers shutdown if found."""
        stop_file = Path(".stop_request")
        while not self.shutdown_manager.shutdown_event.is_set():
            if stop_file.exists():
                logger.info("Stop file detected. Initiating shutdown.")
                self.shutdown_manager.shutdown_event.set()
                stop_file.unlink(missing_ok=True) # Clean up the file
                break
            await asyncio.sleep(1) # Check once per second

    async def run(self):
        """Main entry point to run the framework."""
        logger.info("Starting framework...")
        
        # Start all components (data subscriptions, etc.)
        await self.connection_manager.connect()
        # ... start other components ...

        # Start the background task to check for the stop file
        stop_file_checker = asyncio.create_task(self._check_stop_file_task())

        logger.info("Framework is running. Waiting for shutdown signal (Ctrl+C or stop command)...")
        # This will block until a signal is received or the stop file is found
        await self.shutdown_manager.wait_for_shutdown()

        # Cancel the file checker task as it's no longer needed
        stop_file_checker.cancel()
        
        # Execute the graceful shutdown sequence
        await self.shutdown()

    async def shutdown(self):
        """Executes the prioritized shutdown handlers."""
        await self.shutdown_manager.execute_handlers()
        logger.info("Framework shutdown complete.")

With this structure, whether you hit Ctrl+C or run your tradesys stop command, the same orderly, graceful shutdown sequence is triggered.

Conclusion and Course Wrap-up

In this final lesson, you have implemented the capstone feature of a truly robust automated trading system. By handling OS signals and executing a prioritized shutdown sequence, you've ensured your framework can exit cleanly, managing its orders and positions to prevent costly errors.

Key Takeaways from this Lesson:

• Graceful shutdown is essential for any serious automated system to prevent orphaned orders, unmonitored positions, and resource leaks.
• Python's signal module is the standard way to catch system signals like SIGINT (Ctrl+C) and SIGTERM.
• A centralized, asyncio-native shutdown manager provides a clean, robust pattern for orchestrating a complex shutdown.
• The order of shutdown operations is critical. We use a priority system to ensure we manage orders and positions before disconnecting from the services that control them.

You have now built a complete, end-to-end framework for a credit spread trading system in Python—from broker connection and data handling to strategy logic and a professional CLI. You've designed a modular, event-driven system capable of running in both paper and live trading modes, with robust control and shutdown mechanisms.

Thank you for your dedication throughout this course. The skills you've developed are the foundation for building even more sophisticated quantitative trading systems. The journey of a quantitative trader is one of continuous learning, and I hope this course has equipped you well for your next steps. Happy trading