# TODO(Danya): this is more general than simulation, the output format is common for all types of Systems (e.g., prod system).
Streaming simulation output
Example of the output of the simulation
- The output is truncated to include only 2 files from each subdirectory
> i docker_bash
> cd /shared_data/ecs/preprod/system_reconciliation/C5b/paper_trading/20240115_131000.20240116_130500/system_log_dir.scheduled
> find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"
.
|-process_forecasts
| |-portfolio
| | |-executed_trades_notional
| | | |-20240115_124500.20240115_124531.csv
| | | |-20240116_073000.20240116_073050.csv
| | |-executed_trades_shares
| | | |-20240115_124500.20240115_124531.csv
| | | |-20240115_182000.20240115_182033.csv
| | |-holdings_shares
| | | |-20240115_124500.20240115_124531.csv
| | | |-20240115_182000.20240115_182033.csv
| | |-statistics
| | | |-20240115_124500.20240115_124531.csv
| | | |-20240115_182000.20240115_182033.csv
| | |-holdings_notional
| | | |-20240115_124500.20240115_124531.csv
| | | |-20240115_182000.20240115_182033.csv
| |-target_positions
| | |-20240115_124500.20240115_124531.csv
| | |-20240115_182000.20240115_182033.csv
| |-orders
| | |-20240115_124500.20240115_124531.csv
| | |-20240115_182000.20240115_182033.csv
|-system_config.output.values_as_strings.pkl
|-dag
| |-node_io
| | |-node_io.prof
| | | |-predict.9.process_forecasts.before_execution.20240115_190000.txt
| | | |-predict.5.compress_rets.after_execution.20240116_062500.txt
| | |-node_io.data
| | | |-predict.4.adjust_rets.df_out.20240115_235500.20240115_235526.parquet
| | | |-predict.2.compute_vol.df_out.20240116_040000.20240116_040027.parquet
|-system_config.input.txt
|-system_config.output.txt
|-system_config.input.values_as_strings.pkl
Config
For explanation of SystemConfig structure and SystemConfig examples, refer
to
docs/kaizenflow/ck.system_config.explanation.md
Process_forecasts
Portfolio
# TODO(Danya): the section should refer to a Portfolio doc, since the files just represent Portfolio state.
executed_trades_notional
- Total notional value of executed trades (in dollars), indexed by order ID and timestamp
- Positive value for buys, negative value for sells
Example:
> csvlook portfolio/executed_trades_notional/20240115_081000.20240115_081029.csv
| a | 6051632686 | 8717633868 | 2540896331 | 1528092593 | 8968126878 | 1467591036 | 5115052901 | 3065029174 | 1891737434 | 3401245610 | 1464553467 | 1966583502 | 1030828978 | 2601760471 | 2683705052 | 9872743573 | 2484635488 | 2099673105 | 4516629366 | 2237530510 | 2425308589 | 1776791608 | 2384892553 | 5118394986 |
| -------------------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
| 2024-01-15 08:15:28.918502-05:00 | 63.774 | -72.493… | -55.294… | 52.737… | -231.655… | -469.389… | 61.316… | 98.954… | -66.558… | -44.272… | 289.258… | -51.685… | 76.234… | -65.109… | -83.608… | -46.724… | -255.200… | 78.367… | 116.388… | 94.789… | 55.854… | -96.39 | 64.439… | 414.729… |
executed_trades_shares
- Total number of shares in executed trades, indexed by timestamp and order ID
- Positive value for buys, negative value for sells
Example:
> csvlook portfolio/executed_trades_shares/20240115_081500.20240115_081529.csv
| a | 6051632686 | 8717633868 | 2540896331 | 1528092593 | 8968126878 | 1467591036 | 5115052901 | 3065029174 | 1891737434 | 3401245610 | 1464553467 | 1966583502 | 1030828978 | 2601760471 | 2683705052 | 9872743573 | 2484635488 | 2099673105 | 4516629366 | 2237530510 | 2425308589 | 1776791608 | 2384892553 | 5118394986 |
| -------------------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
| 2024-01-15 08:15:28.918502-05:00 | 45 | -2 | -7 | 139 | -0.73 | -0.011 | 87 | 1,217 | -8.8 | -15.8 | 0.114 | -130 | 247 | -4.12 | -97 | -14 | -1,514 | 17 | 233 | 1 | 93 | -15.3 | 26.8 | 717.4 |
holdings_notional
- Notional of held shares at the beginning of the bar in dollars, based on number of shares and price
- Positive for long positions, negative for short positions
Example:
> csvlook portfolio/holdings_notional/20240115_081000.20240115_081029.csv
| a | 1030828978 | 1464553467 | 1467591036 | 1528092593 | 1776791608 | 1891737434 | 1966583502 | 2099673105 | 2237530510 | 2384892553 | 2425308589 | 2484635488 | 2540896331 | 2601760471 | 2683705052 | 3065029174 | 3401245610 | 4516629366 | 5115052901 | 5118394986 | 6051632686 | 8717633868 | 8968126878 | 9872743573 |
| -------------------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
| 2024-01-15 08:15:28.918502-05:00 | 76.234… | 289.258… | -469.389… | 52.737… | -96.39 | -66.558… | -51.685… | 78.367… | 94.789… | 64.439… | 55.854… | -255.200… | -55.294… | -65.109… | -83.608… | 98.954… | -44.272… | 116.388… | 61.316… | 414.729… | 63.774 | -72.493… | -231.655… | -46.724… |
holdings_shares
- Number of held shares at the beginning of the bar
- Positive for long positions, negative for short positions
Example:
> csvlook portfolio/holdings_shares/20240115_081500.20240115_081529.csv
| a | 6051632686 | 8717633868 | 2540896331 | 1528092593 | 8968126878 | 1467591036 | 5115052901 | 3065029174 | 1891737434 | 3401245610 | 1464553467 | 1966583502 | 1030828978 | 2601760471 | 2683705052 | 9872743573 | 2484635488 | 2099673105 | 4516629366 | 2237530510 | 2425308589 | 1776791608 | 2384892553 | 5118394986 |
| -------------------------------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |
| 2024-01-15 08:15:28.918502-05:00 | 45 | -2 | -7 | 139 | -0.73 | -0.011 | 87 | 1,217 | -8.8 | -15.8 | 0.114 | -130 | 247 | -4.12 | -97 | -14 | -1,514 | 17 | 233 | 1 | 93 | -15.3 | 26.8 | 717.4 |
statistics
- Various statistics calculated based on simulation execution inside the bar
Example:
> csvlook portfolio/statistics/20240115_081500.20240115_081529.csv
| a | pnl | gross_volume | net_volume | gmv | nmv | cash | net_wealth | leverage |
| -------------------------------- | --- | ------------ | ---------- | ---------- | -------- | -------- | ---------- | -------- |
| 2024-01-15 08:15:28.918502-05:00 | 0 | 3,005.216… | -71.539… | 3,005.216… | -71.539… | 771.539… | 700.000… | 4.293… |
Orders
- Examples of orders submitted in the simulation
Example:
> head -3 orders/20240115_124500.20240115_124531.csv
Order: order_id=0 creation_timestamp=2024-01-15 08:10:29.273867-05:00 asset_id=6051632686 type_=price@twap start_timestamp=2024-01-15 08:10:29.273867-05:00 end_timestamp=2024-01-15 08:15:00-05:00 curr_num_shares=0.0 diff_num_shares=45.0 tz=America/New_York extra_params={}
Order: order_id=1 creation_timestamp=2024-01-15 08:10:29.273867-05:00 asset_id=8717633868 type_=price@twap start_timestamp=2024-01-15 08:10:29.273867-05:00 end_timestamp=2024-01-15 08:15:00-05:00 curr_num_shares=0.0 diff_num_shares=-2.0 tz=America/New_York extra_params={}
Order: order_id=2 creation_timestamp=2024-01-15 08:10:29.273867-05:00 asset_id=2540896331 type_=price@twap start_timestamp=2024-01-15 08:10:29.273867-05:00 end_timestamp=2024-01-15 08:15:00-05:00 curr_num_shares=0.0 diff_num_shares=-7.0 tz=America/New_York extra_params={}
Target_positions
- Orders that are planned to execute at the beginning of the bar, projected price and other stats
- Calculated in
oms/order_processing/target_positions_and_order_generator.py
> csvlook target_positions/20240115_081500.20240115_081529.csv
| asset_id | holdings_shares | price | holdings_notional | wall_clock_timestamp | prediction | volatility | spread | target_holdings_notional | target_trades_notional | target_trades_shares | target_holdings_shares | target_holdings_shares.before_apply_cc_limits | target_holdings_notional.before_apply_cc_limits | target_trades_shares.before_apply_cc_limits | target_trades_notional.before_apply_cc_limits |
| ------------- | --------------- | ----------- | ----------------- | -------------------------------- | ---------- | ---------- | ------ | ------------------------ | ---------------------- | -------------------- | ---------------------- | --------------------------------------------- | ----------------------------------------------- | ------------------------------------------- | --------------------------------------------- |
| 6,051,632,686 | 45.000 | 1.417… | 63.774… | 2024-01-15 08:15:28.918502-05:00 | -0.881… | 0.001… | False | -358.552… | -422.326… | -298.000 | -253.000… | -253.000… | -358.552… | -298.000 | -422.326… |
| 8,717,633,868 | -2.000 | 36.247… | -72.493… | 2024-01-15 08:15:28.918502-05:00 | 0.909… | 0.002… | False | 72.493… | 144.986… | 4.000 | 2.000… | 2.000… | 72.493… | 4.000 | 144.986… |
| 2,540,896,331 | -7.000 | 7.899… | -55.294… | 2024-01-15 08:15:28.918502-05:00 | 0.003… | 0.002… | False | 118.488… | 173.782… | 22.000 | 15.000… | 15.000… | 118.488… | 22.000 | 173.782… |
| 1,528,092,593 | 139.000 | 0.379… | 52.737… | 2024-01-15 08:15:28.918502-05:00 | -0.278… | 0.002… | False | -63.739… | -116.476… | -307.000 | -168.000… | -168.000… | -63.739… | -307.000 | -116.476… |
| 8,968,126,878 | -0.730 | 317.336… | -231.655… | 2024-01-15 08:15:28.918502-05:00 | 0.723… | 0.003… | False | 31.734… | 263.389… | 0.830 | 0.100… | 0.100… | 31.734… | 0.830 | 263.389… |
| 1,467,591,036 | -0.011 | 42,671.700… | -469.389… | 2024-01-15 08:15:28.918502-05:00 | -0.588… | 0.001… | False | -298.702… | 170.687… | 0.004 | -0.007… | -0.007… | -298.702… | 0.004 | 170.687… |
| 5,115,052,901 | 87.000 | 0.705… | 61.316… | 2024-01-15 08:15:28.918502-05:00 | 0.431… | 0.004… | False | 20.439… | -40.877… | -58.000 | 29.000… | 29.000… | 20.439… | -58.000 | -40.877… |
| 3,065,029,174 | 1,217.000 | 0.081… | 98.954… | 2024-01-15 08:15:28.918502-05:00 | -0.257… | 0.001… | False | -164.409… | -263.363… | -3,239.000 | -2,022.000… | -2,022.000… | -164.409… | -3,239.000 | -263.363… |
| 1,891,737,434 | -8.800 | 7.563… | -66.558… | 2024-01-15 08:15:28.918502-05:00 | -0.145… | 0.002… | False | -86.979… | -20.421… | -2.700 | -11.500… | -11.500… | -86.979… | -2.700 | -20.421… |
| 3,401,245,610 | -15.800 | 2.802… | -44.272… | 2024-01-15 08:15:28.918502-05:00 | -0.548… | 0.002… | False | -63.325… | -19.054… | -6.800 | -22.600… | -22.600… | -63.325… | -6.800 | -19.054… |
| 1,464,553,467 | 0.114 | 2,537.348… | 289.258… | 2024-01-15 08:15:28.918502-05:00 | 0.075… | 0.001… | False | 332.393… | 43.135… | 0.017 | 0.131… | 0.131… | 332.393… | 0.017 | 43.135… |
| 1,966,583,502 | -130.000 | 0.398… | -51.685… | 2024-01-15 08:15:28.918502-05:00 | 0.523… | 0.002… | False | 77.926… | 129.611… | 326.000 | 196.000… | 196.000… | 77.926… | 326.000 | 129.611… |
| 1,030,828,978 | 247.000 | 0.309… | 76.234… | 2024-01-15 08:15:28.918502-05:00 | -0.305… | 0.003… | False | -49.691… | -125.925… | -408.000 | -161.000… | -161.000… | -49.691… | -408.000 | -125.925… |
| 2,601,760,471 | -4.120 | 15.803… | -65.109… | 2024-01-15 08:15:28.918502-05:00 | -0.540… | 0.004… | False | -22.757… | 42.353… | 2.680 | -1.440… | -1.440… | -22.757… | 2.680 | 42.353… |
| 2,683,705,052 | -97.000 | 0.862… | -83.608… | 2024-01-15 08:15:28.918502-05:00 | 0.063… | 0.002… | False | 72.403… | 156.011… | 181.000 | 84.000… | 84.000… | 72.403… | 181.000 | 156.011… |
| 9,872,743,573 | -14.000 | 3.337… | -46.724… | 2024-01-15 08:15:28.918502-05:00 | -0.538… | 0.002… | False | -66.748… | -20.024… | -6.000 | -20.000… | -20.000… | -66.748… | -6.000 | -20.024… |
| 2,484,635,488 | -1,514.000 | 0.169… | -255.200… | 2024-01-15 08:15:28.918502-05:00 | 0.351… | 0.004… | False | 23.093… | 278.293… | 1,651.000 | 137.000… | 137.000… | 23.093… | 1,651.000 | 278.293… |
| 2,099,673,105 | 17.000 | 4.610… | 78.367… | 2024-01-15 08:15:28.918502-05:00 | 0.268… | 0.002… | False | 115.245… | 36.878… | 8.000 | 25.000… | 25.000… | 115.245… | 8.000 | 36.878… |
| 4,516,629,366 | 233.000 | 0.500… | 116.388… | 2024-01-15 08:15:28.918502-05:00 | -0.113… | 0.002… | False | -110.394… | -226.782… | -454.000 | -221.000… | -221.000… | -110.394… | -454.000 | -226.782… |
| 2,237,530,510 | 1.000 | 94.789… | 94.789… | 2024-01-15 08:15:28.918502-05:00 | -0.054… | 0.002… | False | -94.789… | -189.578… | -2.000 | -1.000… | -1.000… | -94.789… | -2.000 | -189.578… |
| 2,425,308,589 | 93.000 | 0.601… | 55.854… | 2024-01-15 08:15:28.918502-05:00 | -0.000… | 0.002… | False | 99.096… | 43.242… | 72.000 | 165.000… | 165.000… | 99.096… | 72.000 | 43.242… |
| 1,776,791,608 | -15.300 | 6.300… | -96.390… | 2024-01-15 08:15:28.918502-05:00 | 0.508… | 0.001… | False | 152.460… | 248.850… | 39.500 | 24.200… | 24.200… | 152.460… | 39.500 | 248.850… |
| 2,384,892,553 | 26.800 | 2.404… | 64.439… | 2024-01-15 08:15:28.918502-05:00 | 1.019… | 0.003… | False | 51.215… | -13.224… | -5.500 | 21.300… | 21.300… | 51.215… | -5.500 | -13.224… |
| 5,118,394,986 | 717.400 | 0.578… | 414.729… | 2024-01-15 08:15:28.918502-05:00 | -0.645… | 0.001… | False | -429.644… | -844.373… | -1,460.600 | -743.200… | -743.200… | -429.644… | -1,460.600 | -844.373… |
DAG
# TODO(Grisha): Describe the DAG folder contents.