Hugging Face's logo

nhiro3303
/
rl_course_vizdoom_health_gathering_supreme

Reinforcement Learning
sample-factory
TensorBoard
deep-reinforcement-learning
Eval Results
Model card Files Metrics Community
nhiro3303 commited on
Commit
50f4465
·
1 Parent(s): e695a5f

Upload . with huggingface_hub

Browse files
Files changed (4) hide show
  1. .summary/0/events.out.tfevents.1677751105.ff5fdd36e73f +3 -0
  2. README.md +1 -1
  3. replay.mp4 +2 -2
  4. sf_log.txt +316 -662
.summary/0/events.out.tfevents.1677751105.ff5fdd36e73f ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:419f9564e6d1c6252237dc4dad867564b99248ff95993b6a209cc063f6c0a177
3
+ size 5140
README.md CHANGED
@@ -15,7 +15,7 @@ model-index:
15
  type: doom_health_gathering_supreme
16
  metrics:
17
  - type: mean_reward
18
- value: 3.94 +/- 0.66
19
  name: mean_reward
20
  verified: false
21
  ---
 
15
  type: doom_health_gathering_supreme
16
  metrics:
17
  - type: mean_reward
18
+ value: 4.03 +/- 0.29
19
  name: mean_reward
20
  verified: false
21
  ---
replay.mp4 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:88d5850b57a33e86bced6b3fd1294fceb3faec99c197b47d416c2779fa79c7ab
3
- size 5827554
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:37871f92635a2443e918e53cc66df096c4d3cd6e312271cb30662fb024504817
3
+ size 6197877
sf_log.txt CHANGED
@@ -1,39 +1,39 @@
1
- [2023-03-02 08:33:09,561][08277] Saving configuration to /home/gpu/train_dir/default_experiment/config.json...
2
- [2023-03-02 08:33:09,561][08277] Rollout worker 0 uses device cpu
3
- [2023-03-02 08:33:09,561][08277] Rollout worker 1 uses device cpu
4
- [2023-03-02 08:33:09,562][08277] Rollout worker 2 uses device cpu
5
- [2023-03-02 08:33:09,562][08277] Rollout worker 3 uses device cpu
6
- [2023-03-02 08:33:09,562][08277] Rollout worker 4 uses device cpu
7
- [2023-03-02 08:33:09,562][08277] Rollout worker 5 uses device cpu
8
- [2023-03-02 08:33:09,562][08277] Rollout worker 6 uses device cpu
9
- [2023-03-02 08:33:09,562][08277] Rollout worker 7 uses device cpu
10
- [2023-03-02 08:33:09,608][08277] Using GPUs [0] for process 0 (actually maps to GPUs [0])
11
- [2023-03-02 08:33:09,608][08277] InferenceWorker_p0-w0: min num requests: 2
12
- [2023-03-02 08:33:09,624][08277] Starting all processes...
13
- [2023-03-02 08:33:09,625][08277] Starting process learner_proc0
14
- [2023-03-02 08:33:10,271][08277] Starting all processes...
15
- [2023-03-02 08:33:10,275][08335] Using GPUs [0] for process 0 (actually maps to GPUs [0])
16
- [2023-03-02 08:33:10,275][08335] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0
17
- [2023-03-02 08:33:10,275][08277] Starting process inference_proc0-0
18
- [2023-03-02 08:33:10,275][08277] Starting process rollout_proc0
19
- [2023-03-02 08:33:10,276][08277] Starting process rollout_proc1
20
- [2023-03-02 08:33:10,276][08277] Starting process rollout_proc2
21
- [2023-03-02 08:33:10,279][08277] Starting process rollout_proc3
22
- [2023-03-02 08:33:10,279][08277] Starting process rollout_proc4
23
- [2023-03-02 08:33:10,280][08277] Starting process rollout_proc5
24
- [2023-03-02 08:33:10,280][08277] Starting process rollout_proc6
25
- [2023-03-02 08:33:10,280][08277] Starting process rollout_proc7
26
- [2023-03-02 08:33:10,310][08335] Num visible devices: 1
27
- [2023-03-02 08:33:10,353][08335] Starting seed is not provided
28
- [2023-03-02 08:33:10,353][08335] Using GPUs [0] for process 0 (actually maps to GPUs [0])
29
- [2023-03-02 08:33:10,353][08335] Initializing actor-critic model on device cuda:0
30
- [2023-03-02 08:33:10,354][08335] RunningMeanStd input shape: (3, 72, 128)
31
- [2023-03-02 08:33:10,355][08335] RunningMeanStd input shape: (1,)
32
- [2023-03-02 08:33:10,363][08335] ConvEncoder: input_channels=3
33
- [2023-03-02 08:33:10,681][08335] Conv encoder output size: 512
34
- [2023-03-02 08:33:10,681][08335] Policy head output size: 512
35
- [2023-03-02 08:33:10,732][08335] Created Actor Critic model with architecture:
36
- [2023-03-02 08:33:10,733][08335] ActorCriticSharedWeights(
37
  (obs_normalizer): ObservationNormalizer(
38
  (running_mean_std): RunningMeanStdDictInPlace(
39
  (running_mean_std): ModuleDict(
@@ -74,631 +74,285 @@
74
  (distribution_linear): Linear(in_features=512, out_features=5, bias=True)
75
  )
76
  )
77
- [2023-03-02 08:33:11,319][08369] Worker 6 uses CPU cores [12, 13]
78
- [2023-03-02 08:33:11,329][08365] Worker 1 uses CPU cores [2, 3]
79
- [2023-03-02 08:33:11,355][08366] Worker 3 uses CPU cores [6, 7]
80
- [2023-03-02 08:33:11,366][08363] Using GPUs [0] for process 0 (actually maps to GPUs [0])
81
- [2023-03-02 08:33:11,366][08363] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0
82
- [2023-03-02 08:33:11,378][08368] Worker 5 uses CPU cores [10, 11]
83
- [2023-03-02 08:33:11,379][08363] Num visible devices: 1
84
- [2023-03-02 08:33:11,410][08364] Worker 0 uses CPU cores [0, 1]
85
- [2023-03-02 08:33:11,420][08367] Worker 2 uses CPU cores [4, 5]
86
- [2023-03-02 08:33:11,425][08370] Worker 7 uses CPU cores [14, 15]
87
- [2023-03-02 08:33:11,426][08371] Worker 4 uses CPU cores [8, 9]
88
- [2023-03-02 08:33:14,220][08335] Using optimizer <class 'torch.optim.adam.Adam'>
89
- [2023-03-02 08:33:14,221][08335] No checkpoints found
90
- [2023-03-02 08:33:14,221][08335] Did not load from checkpoint, starting from scratch!
91
- [2023-03-02 08:33:14,221][08335] Initialized policy 0 weights for model version 0
92
- [2023-03-02 08:33:14,225][08335] LearnerWorker_p0 finished initialization!
93
- [2023-03-02 08:33:14,225][08335] Using GPUs [0] for process 0 (actually maps to GPUs [0])
94
- [2023-03-02 08:33:14,336][08363] Unhandled exception CUDA error: invalid resource handle
95
  CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
96
  For debugging consider passing CUDA_LAUNCH_BLOCKING=1. in evt loop inference_proc0-0_evt_loop
97
- [2023-03-02 08:33:17,898][08277] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
98
- [2023-03-02 08:33:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
99
- [2023-03-02 08:33:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
100
- [2023-03-02 08:33:29,604][08277] Heartbeat connected on Batcher_0
101
- [2023-03-02 08:33:29,606][08277] Heartbeat connected on LearnerWorker_p0
102
- [2023-03-02 08:33:29,611][08277] Heartbeat connected on RolloutWorker_w0
103
- [2023-03-02 08:33:29,612][08277] Heartbeat connected on RolloutWorker_w1
104
- [2023-03-02 08:33:29,614][08277] Heartbeat connected on RolloutWorker_w2
105
- [2023-03-02 08:33:29,616][08277] Heartbeat connected on RolloutWorker_w3
106
- [2023-03-02 08:33:29,618][08277] Heartbeat connected on RolloutWorker_w4
107
- [2023-03-02 08:33:29,620][08277] Heartbeat connected on RolloutWorker_w5
108
- [2023-03-02 08:33:29,622][08277] Heartbeat connected on RolloutWorker_w6
109
- [2023-03-02 08:33:29,624][08277] Heartbeat connected on RolloutWorker_w7
110
- [2023-03-02 08:33:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
111
- [2023-03-02 08:33:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
112
- [2023-03-02 08:33:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
113
- [2023-03-02 08:33:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
114
- [2023-03-02 08:33:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
115
- [2023-03-02 08:33:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
116
- [2023-03-02 08:34:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
117
- [2023-03-02 08:34:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
118
- [2023-03-02 08:34:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
119
- [2023-03-02 08:34:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
120
- [2023-03-02 08:34:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
121
- [2023-03-02 08:34:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
122
- [2023-03-02 08:34:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
123
- [2023-03-02 08:34:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
124
- [2023-03-02 08:34:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
125
- [2023-03-02 08:34:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
126
- [2023-03-02 08:34:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
127
- [2023-03-02 08:34:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
128
- [2023-03-02 08:35:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
129
- [2023-03-02 08:35:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
130
- [2023-03-02 08:35:07,899][08335] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
131
- [2023-03-02 08:35:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
132
- [2023-03-02 08:35:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
133
- [2023-03-02 08:35:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
134
- [2023-03-02 08:35:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
135
- [2023-03-02 08:35:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
136
- [2023-03-02 08:35:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
137
- [2023-03-02 08:35:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
138
- [2023-03-02 08:35:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
139
- [2023-03-02 08:35:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
140
- [2023-03-02 08:35:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
141
- [2023-03-02 08:36:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
142
- [2023-03-02 08:36:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
143
- [2023-03-02 08:36:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
144
- [2023-03-02 08:36:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
145
- [2023-03-02 08:36:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
146
- [2023-03-02 08:36:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
147
- [2023-03-02 08:36:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
148
- [2023-03-02 08:36:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
149
- [2023-03-02 08:36:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
150
- [2023-03-02 08:36:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
151
- [2023-03-02 08:36:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
152
- [2023-03-02 08:36:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
153
- [2023-03-02 08:37:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
154
- [2023-03-02 08:37:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
155
- [2023-03-02 08:37:07,899][08335] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
156
- [2023-03-02 08:37:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
157
- [2023-03-02 08:37:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
158
- [2023-03-02 08:37:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
159
- [2023-03-02 08:37:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
160
- [2023-03-02 08:37:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
161
- [2023-03-02 08:37:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
162
- [2023-03-02 08:37:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
163
- [2023-03-02 08:37:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
164
- [2023-03-02 08:37:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
165
- [2023-03-02 08:37:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
166
- [2023-03-02 08:38:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
167
- [2023-03-02 08:38:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
168
- [2023-03-02 08:38:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
169
- [2023-03-02 08:38:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
170
- [2023-03-02 08:38:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
171
- [2023-03-02 08:38:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
172
- [2023-03-02 08:38:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
173
- [2023-03-02 08:38:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
174
- [2023-03-02 08:38:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
175
- [2023-03-02 08:38:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
176
- [2023-03-02 08:38:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
177
- [2023-03-02 08:38:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
178
- [2023-03-02 08:39:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
179
- [2023-03-02 08:39:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
180
- [2023-03-02 08:39:07,899][08335] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
181
- [2023-03-02 08:39:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
182
- [2023-03-02 08:39:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
183
- [2023-03-02 08:39:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
184
- [2023-03-02 08:39:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
185
- [2023-03-02 08:39:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
186
- [2023-03-02 08:39:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
187
- [2023-03-02 08:39:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
188
- [2023-03-02 08:39:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
189
- [2023-03-02 08:39:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
190
- [2023-03-02 08:39:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
191
- [2023-03-02 08:40:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
192
- [2023-03-02 08:40:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
193
- [2023-03-02 08:40:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
194
- [2023-03-02 08:40:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
195
- [2023-03-02 08:40:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
196
- [2023-03-02 08:40:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
197
- [2023-03-02 08:40:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
198
- [2023-03-02 08:40:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
199
- [2023-03-02 08:40:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
200
- [2023-03-02 08:40:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
201
- [2023-03-02 08:40:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
202
- [2023-03-02 08:40:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
203
- [2023-03-02 08:41:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
204
- [2023-03-02 08:41:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
205
- [2023-03-02 08:41:07,899][08335] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
206
- [2023-03-02 08:41:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
207
- [2023-03-02 08:41:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
208
- [2023-03-02 08:41:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
209
- [2023-03-02 08:41:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
210
- [2023-03-02 08:41:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
211
- [2023-03-02 08:41:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
212
- [2023-03-02 08:41:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
213
- [2023-03-02 08:41:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
214
- [2023-03-02 08:41:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
215
- [2023-03-02 08:41:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
216
- [2023-03-02 08:42:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
217
- [2023-03-02 08:42:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
218
- [2023-03-02 08:42:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
219
- [2023-03-02 08:42:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
220
- [2023-03-02 08:42:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
221
- [2023-03-02 08:42:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
222
- [2023-03-02 08:42:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
223
- [2023-03-02 08:42:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
224
- [2023-03-02 08:42:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
225
- [2023-03-02 08:42:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
226
- [2023-03-02 08:42:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
227
- [2023-03-02 08:42:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
228
- [2023-03-02 08:43:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
229
- [2023-03-02 08:43:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
230
- [2023-03-02 08:43:07,899][08335] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
231
- [2023-03-02 08:43:07,900][08277] Components not started: InferenceWorker_p0-w0, wait_time=600.0 seconds
232
- [2023-03-02 08:43:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
233
- [2023-03-02 08:43:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
234
- [2023-03-02 08:43:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
235
- [2023-03-02 08:43:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
236
- [2023-03-02 08:43:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
237
- [2023-03-02 08:43:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
238
- [2023-03-02 08:43:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
239
- [2023-03-02 08:43:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
240
- [2023-03-02 08:43:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
241
- [2023-03-02 08:43:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
242
- [2023-03-02 08:44:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
243
- [2023-03-02 08:44:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
244
- [2023-03-02 08:44:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
245
- [2023-03-02 08:44:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
246
- [2023-03-02 08:44:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
247
- [2023-03-02 08:44:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
248
- [2023-03-02 08:44:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
249
- [2023-03-02 08:44:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
250
- [2023-03-02 08:44:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
251
- [2023-03-02 08:44:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
252
- [2023-03-02 08:44:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
253
- [2023-03-02 08:44:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
254
- [2023-03-02 08:45:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
255
- [2023-03-02 08:45:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
256
- [2023-03-02 08:45:07,899][08335] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
257
- [2023-03-02 08:45:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
258
- [2023-03-02 08:45:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
259
- [2023-03-02 08:45:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
260
- [2023-03-02 08:45:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
261
- [2023-03-02 08:45:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
262
- [2023-03-02 08:45:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
263
- [2023-03-02 08:45:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
264
- [2023-03-02 08:45:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
265
- [2023-03-02 08:45:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
266
- [2023-03-02 08:45:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
267
- [2023-03-02 08:46:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
268
- [2023-03-02 08:46:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
269
- [2023-03-02 08:46:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
270
- [2023-03-02 08:46:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
271
- [2023-03-02 08:46:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
272
- [2023-03-02 08:46:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
273
- [2023-03-02 08:46:32,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
274
- [2023-03-02 08:46:37,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
275
- [2023-03-02 08:46:42,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
276
- [2023-03-02 08:46:47,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
277
- [2023-03-02 08:46:52,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
278
- [2023-03-02 08:46:57,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
279
- [2023-03-02 08:47:02,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
280
- [2023-03-02 08:47:07,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
281
- [2023-03-02 08:47:07,899][08335] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
282
- [2023-03-02 08:47:12,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
283
- [2023-03-02 08:47:17,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
284
- [2023-03-02 08:47:22,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
285
- [2023-03-02 08:47:27,898][08277] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
286
- [2023-03-02 08:47:31,424][08277] Keyboard interrupt detected in the event loop EvtLoop [Runner_EvtLoop, process=main process 8277], exiting...
287
- [2023-03-02 08:47:31,424][08371] Stopping RolloutWorker_w4...
288
- [2023-03-02 08:47:31,424][08365] Stopping RolloutWorker_w1...
289
- [2023-03-02 08:47:31,424][08368] Stopping RolloutWorker_w5...
290
- [2023-03-02 08:47:31,424][08364] Stopping RolloutWorker_w0...
291
- [2023-03-02 08:47:31,424][08277] Runner profile tree view:
292
- main_loop: 861.8000
293
- [2023-03-02 08:47:31,424][08367] Stopping RolloutWorker_w2...
294
- [2023-03-02 08:47:31,425][08277] Collected {0: 0}, FPS: 0.0
295
- [2023-03-02 08:47:31,424][08369] Stopping RolloutWorker_w6...
296
- [2023-03-02 08:47:31,424][08366] Stopping RolloutWorker_w3...
297
- [2023-03-02 08:47:31,425][08365] Loop rollout_proc1_evt_loop terminating...
298
- [2023-03-02 08:47:31,425][08371] Loop rollout_proc4_evt_loop terminating...
299
- [2023-03-02 08:47:31,425][08368] Loop rollout_proc5_evt_loop terminating...
300
- [2023-03-02 08:47:31,425][08335] Stopping Batcher_0...
301
- [2023-03-02 08:47:31,425][08370] Stopping RolloutWorker_w7...
302
- [2023-03-02 08:47:31,425][08364] Loop rollout_proc0_evt_loop terminating...
303
- [2023-03-02 08:47:31,425][08367] Loop rollout_proc2_evt_loop terminating...
304
- [2023-03-02 08:47:31,425][08369] Loop rollout_proc6_evt_loop terminating...
305
- [2023-03-02 08:47:31,425][08366] Loop rollout_proc3_evt_loop terminating...
306
- [2023-03-02 08:47:31,425][08370] Loop rollout_proc7_evt_loop terminating...
307
- [2023-03-02 08:47:31,425][08335] Loop batcher_evt_loop terminating...
308
- [2023-03-02 08:47:31,426][08335] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
309
- [2023-03-02 08:47:31,446][08335] Stopping LearnerWorker_p0...
310
- [2023-03-02 08:47:31,447][08335] Loop learner_proc0_evt_loop terminating...
311
- [2023-03-02 08:47:31,510][08277] Loading existing experiment configuration from /home/gpu/train_dir/default_experiment/config.json
312
- [2023-03-02 08:47:31,510][08277] Overriding arg 'num_workers' with value 1 passed from command line
313
- [2023-03-02 08:47:31,511][08277] Adding new argument 'no_render'=True that is not in the saved config file!
314
- [2023-03-02 08:47:31,511][08277] Adding new argument 'save_video'=True that is not in the saved config file!
315
- [2023-03-02 08:47:31,511][08277] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file!
316
- [2023-03-02 08:47:31,511][08277] Adding new argument 'video_name'=None that is not in the saved config file!
317
- [2023-03-02 08:47:31,511][08277] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file!
318
- [2023-03-02 08:47:31,511][08277] Adding new argument 'max_num_episodes'=10 that is not in the saved config file!
319
- [2023-03-02 08:47:31,511][08277] Adding new argument 'push_to_hub'=False that is not in the saved config file!
320
- [2023-03-02 08:47:31,511][08277] Adding new argument 'hf_repository'=None that is not in the saved config file!
321
- [2023-03-02 08:47:31,511][08277] Adding new argument 'policy_index'=0 that is not in the saved config file!
322
- [2023-03-02 08:47:31,512][08277] Adding new argument 'eval_deterministic'=False that is not in the saved config file!
323
- [2023-03-02 08:47:31,512][08277] Adding new argument 'train_script'=None that is not in the saved config file!
324
- [2023-03-02 08:47:31,512][08277] Adding new argument 'enjoy_script'=None that is not in the saved config file!
325
- [2023-03-02 08:47:31,512][08277] Using frameskip 1 and render_action_repeat=4 for evaluation
326
- [2023-03-02 08:47:31,521][08277] Doom resolution: 160x120, resize resolution: (128, 72)
327
- [2023-03-02 08:47:31,521][08277] RunningMeanStd input shape: (3, 72, 128)
328
- [2023-03-02 08:47:31,522][08277] RunningMeanStd input shape: (1,)
329
- [2023-03-02 08:47:31,533][08277] ConvEncoder: input_channels=3
330
- [2023-03-02 08:47:31,775][08277] Conv encoder output size: 512
331
- [2023-03-02 08:47:31,776][08277] Policy head output size: 512
332
- [2023-03-02 08:47:33,379][08277] Loading state from checkpoint /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
333
- [2023-03-02 08:47:34,725][08277] Num frames 100...
334
- [2023-03-02 08:47:34,819][08277] Num frames 200...
335
- [2023-03-02 08:47:34,913][08277] Num frames 300...
336
- [2023-03-02 08:47:35,007][08277] Num frames 400...
337
- [2023-03-02 08:47:35,136][08277] Avg episode rewards: #0: 6.800, true rewards: #0: 4.800
338
- [2023-03-02 08:47:35,136][08277] Avg episode reward: 6.800, avg true_objective: 4.800
339
- [2023-03-02 08:47:35,155][08277] Num frames 500...
340
- [2023-03-02 08:47:35,248][08277] Num frames 600...
341
- [2023-03-02 08:47:35,342][08277] Num frames 700...
342
- [2023-03-02 08:47:35,438][08277] Num frames 800...
343
- [2023-03-02 08:47:35,551][08277] Avg episode rewards: #0: 5.320, true rewards: #0: 4.320
344
- [2023-03-02 08:47:35,552][08277] Avg episode reward: 5.320, avg true_objective: 4.320
345
- [2023-03-02 08:47:35,590][08277] Num frames 900...
346
- [2023-03-02 08:47:35,693][08277] Num frames 1000...
347
- [2023-03-02 08:47:35,790][08277] Num frames 1100...
348
- [2023-03-02 08:47:35,885][08277] Num frames 1200...
349
- [2023-03-02 08:47:35,984][08277] Avg episode rewards: #0: 4.827, true rewards: #0: 4.160
350
- [2023-03-02 08:47:35,984][08277] Avg episode reward: 4.827, avg true_objective: 4.160
351
- [2023-03-02 08:47:36,035][08277] Num frames 1300...
352
- [2023-03-02 08:47:36,131][08277] Num frames 1400...
353
- [2023-03-02 08:47:36,226][08277] Num frames 1500...
354
- [2023-03-02 08:47:36,321][08277] Num frames 1600...
355
- [2023-03-02 08:47:36,404][08277] Avg episode rewards: #0: 4.580, true rewards: #0: 4.080
356
- [2023-03-02 08:47:36,405][08277] Avg episode reward: 4.580, avg true_objective: 4.080
357
- [2023-03-02 08:47:36,469][08277] Num frames 1700...
358
- [2023-03-02 08:47:36,563][08277] Num frames 1800...
359
- [2023-03-02 08:47:36,657][08277] Num frames 1900...
360
- [2023-03-02 08:47:36,751][08277] Num frames 2000...
361
- [2023-03-02 08:47:36,821][08277] Avg episode rewards: #0: 4.432, true rewards: #0: 4.032
362
- [2023-03-02 08:47:36,821][08277] Avg episode reward: 4.432, avg true_objective: 4.032
363
- [2023-03-02 08:47:36,905][08277] Num frames 2100...
364
- [2023-03-02 08:47:36,999][08277] Num frames 2200...
365
- [2023-03-02 08:47:37,093][08277] Num frames 2300...
366
- [2023-03-02 08:47:37,187][08277] Num frames 2400...
367
- [2023-03-02 08:47:37,300][08277] Avg episode rewards: #0: 4.607, true rewards: #0: 4.107
368
- [2023-03-02 08:47:37,301][08277] Avg episode reward: 4.607, avg true_objective: 4.107
369
- [2023-03-02 08:47:37,335][08277] Num frames 2500...
370
- [2023-03-02 08:47:37,431][08277] Num frames 2600...
371
- [2023-03-02 08:47:37,525][08277] Num frames 2700...
372
- [2023-03-02 08:47:37,620][08277] Num frames 2800...
373
- [2023-03-02 08:47:37,718][08277] Avg episode rewards: #0: 4.497, true rewards: #0: 4.069
374
- [2023-03-02 08:47:37,718][08277] Avg episode reward: 4.497, avg true_objective: 4.069
375
- [2023-03-02 08:47:37,768][08277] Num frames 2900...
376
- [2023-03-02 08:47:37,862][08277] Num frames 3000...
377
- [2023-03-02 08:47:37,956][08277] Num frames 3100...
378
- [2023-03-02 08:47:38,051][08277] Num frames 3200...
379
- [2023-03-02 08:47:38,134][08277] Avg episode rewards: #0: 4.415, true rewards: #0: 4.040
380
- [2023-03-02 08:47:38,134][08277] Avg episode reward: 4.415, avg true_objective: 4.040
381
- [2023-03-02 08:47:38,199][08277] Num frames 3300...
382
- [2023-03-02 08:47:38,294][08277] Num frames 3400...
383
- [2023-03-02 08:47:38,395][08277] Num frames 3500...
384
- [2023-03-02 08:47:38,502][08277] Num frames 3600...
385
- [2023-03-02 08:47:38,571][08277] Avg episode rewards: #0: 4.351, true rewards: #0: 4.018
386
- [2023-03-02 08:47:38,571][08277] Avg episode reward: 4.351, avg true_objective: 4.018
387
- [2023-03-02 08:47:38,658][08277] Num frames 3700...
388
- [2023-03-02 08:47:38,760][08277] Num frames 3800...
389
- [2023-03-02 08:47:38,858][08277] Num frames 3900...
390
- [2023-03-02 08:47:38,962][08277] Num frames 4000...
391
- [2023-03-02 08:47:39,012][08277] Avg episode rewards: #0: 4.300, true rewards: #0: 4.000
392
- [2023-03-02 08:47:39,013][08277] Avg episode reward: 4.300, avg true_objective: 4.000
393
- [2023-03-02 08:47:42,935][08277] Replay video saved to /home/gpu/train_dir/default_experiment/replay.mp4!
394
- [2023-03-02 08:48:11,637][08718] Saving configuration to /home/gpu/train_dir/default_experiment/config.json...
395
- [2023-03-02 08:48:11,638][08718] Rollout worker 0 uses device cpu
396
- [2023-03-02 08:48:11,638][08718] Rollout worker 1 uses device cpu
397
- [2023-03-02 08:48:11,638][08718] Rollout worker 2 uses device cpu
398
- [2023-03-02 08:48:11,638][08718] Rollout worker 3 uses device cpu
399
- [2023-03-02 08:48:11,638][08718] Rollout worker 4 uses device cpu
400
- [2023-03-02 08:48:11,638][08718] Rollout worker 5 uses device cpu
401
- [2023-03-02 08:48:11,638][08718] Rollout worker 6 uses device cpu
402
- [2023-03-02 08:48:11,638][08718] Rollout worker 7 uses device cpu
403
- [2023-03-02 08:48:11,665][08718] Using GPUs [0] for process 0 (actually maps to GPUs [0])
404
- [2023-03-02 08:48:11,666][08718] InferenceWorker_p0-w0: min num requests: 2
405
- [2023-03-02 08:48:11,682][08718] Starting all processes...
406
- [2023-03-02 08:48:11,682][08718] Starting process learner_proc0
407
- [2023-03-02 08:48:12,341][08718] Starting all processes...
408
- [2023-03-02 08:48:12,344][08718] Starting process inference_proc0-0
409
- [2023-03-02 08:48:12,344][08776] Using GPUs [0] for process 0 (actually maps to GPUs [0])
410
- [2023-03-02 08:48:12,344][08718] Starting process rollout_proc0
411
- [2023-03-02 08:48:12,344][08776] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0
412
- [2023-03-02 08:48:12,344][08718] Starting process rollout_proc1
413
- [2023-03-02 08:48:12,344][08718] Starting process rollout_proc2
414
- [2023-03-02 08:48:12,344][08718] Starting process rollout_proc3
415
- [2023-03-02 08:48:12,346][08718] Starting process rollout_proc4
416
- [2023-03-02 08:48:12,346][08718] Starting process rollout_proc5
417
- [2023-03-02 08:48:12,347][08718] Starting process rollout_proc6
418
- [2023-03-02 08:48:12,348][08718] Starting process rollout_proc7
419
- [2023-03-02 08:48:12,379][08776] Num visible devices: 1
420
- [2023-03-02 08:48:12,406][08776] Starting seed is not provided
421
- [2023-03-02 08:48:12,407][08776] Using GPUs [0] for process 0 (actually maps to GPUs [0])
422
- [2023-03-02 08:48:12,407][08776] Initializing actor-critic model on device cuda:0
423
- [2023-03-02 08:48:12,407][08776] RunningMeanStd input shape: (3, 72, 128)
424
- [2023-03-02 08:48:12,408][08776] RunningMeanStd input shape: (1,)
425
- [2023-03-02 08:48:12,418][08776] ConvEncoder: input_channels=3
426
- [2023-03-02 08:48:12,624][08776] Conv encoder output size: 512
427
- [2023-03-02 08:48:12,625][08776] Policy head output size: 512
428
- [2023-03-02 08:48:12,637][08776] Created Actor Critic model with architecture:
429
- [2023-03-02 08:48:12,638][08776] ActorCriticSharedWeights(
430
- (obs_normalizer): ObservationNormalizer(
431
- (running_mean_std): RunningMeanStdDictInPlace(
432
- (running_mean_std): ModuleDict(
433
- (obs): RunningMeanStdInPlace()
434
- )
435
- )
436
- )
437
- (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace)
438
- (encoder): VizdoomEncoder(
439
- (basic_encoder): ConvEncoder(
440
- (enc): RecursiveScriptModule(
441
- original_name=ConvEncoderImpl
442
- (conv_head): RecursiveScriptModule(
443
- original_name=Sequential
444
- (0): RecursiveScriptModule(original_name=Conv2d)
445
- (1): RecursiveScriptModule(original_name=ELU)
446
- (2): RecursiveScriptModule(original_name=Conv2d)
447
- (3): RecursiveScriptModule(original_name=ELU)
448
- (4): RecursiveScriptModule(original_name=Conv2d)
449
- (5): RecursiveScriptModule(original_name=ELU)
450
- )
451
- (mlp_layers): RecursiveScriptModule(
452
- original_name=Sequential
453
- (0): RecursiveScriptModule(original_name=Linear)
454
- (1): RecursiveScriptModule(original_name=ELU)
455
- )
456
- )
457
- )
458
- )
459
- (core): ModelCoreRNN(
460
- (core): GRU(512, 512)
461
- )
462
- (decoder): MlpDecoder(
463
- (mlp): Identity()
464
- )
465
- (critic_linear): Linear(in_features=512, out_features=1, bias=True)
466
- (action_parameterization): ActionParameterizationDefault(
467
- (distribution_linear): Linear(in_features=512, out_features=5, bias=True)
468
- )
469
- )
470
- [2023-03-02 08:48:13,355][08812] Worker 7 uses CPU cores [14, 15]
471
- [2023-03-02 08:48:13,396][08810] Worker 5 uses CPU cores [10, 11]
472
- [2023-03-02 08:48:13,399][08811] Worker 6 uses CPU cores [12, 13]
473
- [2023-03-02 08:48:13,430][08805] Worker 0 uses CPU cores [0, 1]
474
- [2023-03-02 08:48:13,433][08808] Worker 3 uses CPU cores [6, 7]
475
- [2023-03-02 08:48:13,439][08807] Worker 2 uses CPU cores [4, 5]
476
- [2023-03-02 08:48:13,482][08809] Worker 4 uses CPU cores [8, 9]
477
- [2023-03-02 08:48:13,485][08804] Using GPUs [0] for process 0 (actually maps to GPUs [0])
478
- [2023-03-02 08:48:13,485][08804] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0
479
- [2023-03-02 08:48:13,486][08806] Worker 1 uses CPU cores [2, 3]
480
- [2023-03-02 08:48:13,500][08804] Num visible devices: 1
481
- [2023-03-02 08:48:14,689][08776] Using optimizer <class 'torch.optim.adam.Adam'>
482
- [2023-03-02 08:48:14,689][08776] Loading state from checkpoint /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
483
- [2023-03-02 08:48:14,696][08776] Loading model from checkpoint
484
- [2023-03-02 08:48:14,697][08776] Loaded experiment state at self.train_step=0, self.env_steps=0
485
- [2023-03-02 08:48:14,697][08776] Initialized policy 0 weights for model version 0
486
- [2023-03-02 08:48:14,700][08776] LearnerWorker_p0 finished initialization!
487
- [2023-03-02 08:48:14,700][08776] Using GPUs [0] for process 0 (actually maps to GPUs [0])
488
- [2023-03-02 08:48:14,800][08804] Unhandled exception CUDA error: invalid resource handle
489
- CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
490
- For debugging consider passing CUDA_LAUNCH_BLOCKING=1. in evt loop inference_proc0-0_evt_loop
491
- [2023-03-02 08:48:15,123][08718] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
492
- [2023-03-02 08:48:20,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
493
- [2023-03-02 08:48:25,122][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
494
- [2023-03-02 08:48:30,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
495
- [2023-03-02 08:48:31,661][08718] Heartbeat connected on Batcher_0
496
- [2023-03-02 08:48:31,663][08718] Heartbeat connected on LearnerWorker_p0
497
- [2023-03-02 08:48:31,668][08718] Heartbeat connected on RolloutWorker_w0
498
- [2023-03-02 08:48:31,670][08718] Heartbeat connected on RolloutWorker_w1
499
- [2023-03-02 08:48:31,672][08718] Heartbeat connected on RolloutWorker_w2
500
- [2023-03-02 08:48:31,674][08718] Heartbeat connected on RolloutWorker_w3
501
- [2023-03-02 08:48:31,676][08718] Heartbeat connected on RolloutWorker_w4
502
- [2023-03-02 08:48:31,678][08718] Heartbeat connected on RolloutWorker_w5
503
- [2023-03-02 08:48:31,679][08718] Heartbeat connected on RolloutWorker_w6
504
- [2023-03-02 08:48:31,681][08718] Heartbeat connected on RolloutWorker_w7
505
- [2023-03-02 08:48:35,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
506
- [2023-03-02 08:48:40,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
507
- [2023-03-02 08:48:45,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
508
- [2023-03-02 08:48:50,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
509
- [2023-03-02 08:48:55,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
510
- [2023-03-02 08:49:00,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
511
- [2023-03-02 08:49:05,122][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
512
- [2023-03-02 08:49:10,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
513
- [2023-03-02 08:49:15,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
514
- [2023-03-02 08:49:20,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
515
- [2023-03-02 08:49:25,123][08718] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
516
- [2023-03-02 08:49:27,382][08718] Keyboard interrupt detected in the event loop EvtLoop [Runner_EvtLoop, process=main process 8718], exiting...
517
- [2023-03-02 08:49:27,382][08718] Runner profile tree view:
518
- main_loop: 75.7006
519
- [2023-03-02 08:49:27,382][08809] Stopping RolloutWorker_w4...
520
- [2023-03-02 08:49:27,382][08810] Stopping RolloutWorker_w5...
521
- [2023-03-02 08:49:27,382][08718] Collected {0: 0}, FPS: 0.0
522
- [2023-03-02 08:49:27,382][08808] Stopping RolloutWorker_w3...
523
- [2023-03-02 08:49:27,382][08811] Stopping RolloutWorker_w6...
524
- [2023-03-02 08:49:27,382][08809] Loop rollout_proc4_evt_loop terminating...
525
- [2023-03-02 08:49:27,382][08810] Loop rollout_proc5_evt_loop terminating...
526
- [2023-03-02 08:49:27,382][08776] Stopping Batcher_0...
527
- [2023-03-02 08:49:27,382][08812] Stopping RolloutWorker_w7...
528
- [2023-03-02 08:49:27,383][08808] Loop rollout_proc3_evt_loop terminating...
529
- [2023-03-02 08:49:27,383][08811] Loop rollout_proc6_evt_loop terminating...
530
- [2023-03-02 08:49:27,383][08807] Stopping RolloutWorker_w2...
531
- [2023-03-02 08:49:27,383][08812] Loop rollout_proc7_evt_loop terminating...
532
- [2023-03-02 08:49:27,383][08806] Stopping RolloutWorker_w1...
533
- [2023-03-02 08:49:27,383][08776] Loop batcher_evt_loop terminating...
534
- [2023-03-02 08:49:27,383][08805] Stopping RolloutWorker_w0...
535
- [2023-03-02 08:49:27,383][08807] Loop rollout_proc2_evt_loop terminating...
536
- [2023-03-02 08:49:27,383][08806] Loop rollout_proc1_evt_loop terminating...
537
- [2023-03-02 08:49:27,383][08805] Loop rollout_proc0_evt_loop terminating...
538
- [2023-03-02 08:49:27,384][08776] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
539
- [2023-03-02 08:49:27,405][08776] Stopping LearnerWorker_p0...
540
- [2023-03-02 08:49:27,405][08776] Loop learner_proc0_evt_loop terminating...
541
- [2023-03-02 08:49:27,470][08718] Loading existing experiment configuration from /home/gpu/train_dir/default_experiment/config.json
542
- [2023-03-02 08:49:27,471][08718] Overriding arg 'num_workers' with value 1 passed from command line
543
- [2023-03-02 08:49:27,471][08718] Adding new argument 'no_render'=True that is not in the saved config file!
544
- [2023-03-02 08:49:27,471][08718] Adding new argument 'save_video'=True that is not in the saved config file!
545
- [2023-03-02 08:49:27,471][08718] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file!
546
- [2023-03-02 08:49:27,472][08718] Adding new argument 'video_name'=None that is not in the saved config file!
547
- [2023-03-02 08:49:27,472][08718] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file!
548
- [2023-03-02 08:49:27,472][08718] Adding new argument 'max_num_episodes'=10 that is not in the saved config file!
549
- [2023-03-02 08:49:27,472][08718] Adding new argument 'push_to_hub'=False that is not in the saved config file!
550
- [2023-03-02 08:49:27,472][08718] Adding new argument 'hf_repository'=None that is not in the saved config file!
551
- [2023-03-02 08:49:27,472][08718] Adding new argument 'policy_index'=0 that is not in the saved config file!
552
- [2023-03-02 08:49:27,472][08718] Adding new argument 'eval_deterministic'=False that is not in the saved config file!
553
- [2023-03-02 08:49:27,472][08718] Adding new argument 'train_script'=None that is not in the saved config file!
554
- [2023-03-02 08:49:27,473][08718] Adding new argument 'enjoy_script'=None that is not in the saved config file!
555
- [2023-03-02 08:49:27,473][08718] Using frameskip 1 and render_action_repeat=4 for evaluation
556
- [2023-03-02 08:49:27,484][08718] Doom resolution: 160x120, resize resolution: (128, 72)
557
- [2023-03-02 08:49:27,485][08718] RunningMeanStd input shape: (3, 72, 128)
558
- [2023-03-02 08:49:27,486][08718] RunningMeanStd input shape: (1,)
559
- [2023-03-02 08:49:27,496][08718] ConvEncoder: input_channels=3
560
- [2023-03-02 08:49:27,714][08718] Conv encoder output size: 512
561
- [2023-03-02 08:49:27,714][08718] Policy head output size: 512
562
- [2023-03-02 08:49:29,351][08718] Loading state from checkpoint /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
563
- [2023-03-02 08:49:30,044][08718] Num frames 100...
564
- [2023-03-02 08:49:30,139][08718] Num frames 200...
565
- [2023-03-02 08:49:30,233][08718] Num frames 300...
566
- [2023-03-02 08:49:30,327][08718] Num frames 400...
567
- [2023-03-02 08:49:30,426][08718] Avg episode rewards: #0: 5.480, true rewards: #0: 4.480
568
- [2023-03-02 08:49:30,426][08718] Avg episode reward: 5.480, avg true_objective: 4.480
569
- [2023-03-02 08:49:30,477][08718] Num frames 500...
570
- [2023-03-02 08:49:30,577][08718] Num frames 600...
571
- [2023-03-02 08:49:30,678][08718] Num frames 700...
572
- [2023-03-02 08:49:30,780][08718] Num frames 800...
573
- [2023-03-02 08:49:30,863][08718] Avg episode rewards: #0: 4.660, true rewards: #0: 4.160
574
- [2023-03-02 08:49:30,863][08718] Avg episode reward: 4.660, avg true_objective: 4.160
575
- [2023-03-02 08:49:30,929][08718] Num frames 900...
576
- [2023-03-02 08:49:31,024][08718] Num frames 1000...
577
- [2023-03-02 08:49:31,118][08718] Num frames 1100...
578
- [2023-03-02 08:49:31,213][08718] Num frames 1200...
579
- [2023-03-02 08:49:31,281][08718] Avg episode rewards: #0: 4.387, true rewards: #0: 4.053
580
- [2023-03-02 08:49:31,281][08718] Avg episode reward: 4.387, avg true_objective: 4.053
581
- [2023-03-02 08:49:31,361][08718] Num frames 1300...
582
- [2023-03-02 08:49:31,458][08718] Num frames 1400...
583
- [2023-03-02 08:49:31,554][08718] Num frames 1500...
584
- [2023-03-02 08:49:31,650][08718] Num frames 1600...
585
- [2023-03-02 08:49:31,744][08718] Num frames 1700...
586
- [2023-03-02 08:49:31,858][08718] Avg episode rewards: #0: 5.150, true rewards: #0: 4.400
587
- [2023-03-02 08:49:31,858][08718] Avg episode reward: 5.150, avg true_objective: 4.400
588
- [2023-03-02 08:49:31,898][08718] Num frames 1800...
589
- [2023-03-02 08:49:31,993][08718] Num frames 1900...
590
- [2023-03-02 08:49:32,089][08718] Num frames 2000...
591
- [2023-03-02 08:49:32,184][08718] Num frames 2100...
592
- [2023-03-02 08:49:32,308][08718] Avg episode rewards: #0: 5.152, true rewards: #0: 4.352
593
- [2023-03-02 08:49:32,308][08718] Avg episode reward: 5.152, avg true_objective: 4.352
594
- [2023-03-02 08:49:32,331][08718] Num frames 2200...
595
- [2023-03-02 08:49:32,427][08718] Num frames 2300...
596
- [2023-03-02 08:49:32,523][08718] Num frames 2400...
597
- [2023-03-02 08:49:32,616][08718] Num frames 2500...
598
- [2023-03-02 08:49:32,727][08718] Avg episode rewards: #0: 4.933, true rewards: #0: 4.267
599
- [2023-03-02 08:49:32,727][08718] Avg episode reward: 4.933, avg true_objective: 4.267
600
- [2023-03-02 08:49:32,765][08718] Num frames 2600...
601
- [2023-03-02 08:49:32,860][08718] Num frames 2700...
602
- [2023-03-02 08:49:32,955][08718] Num frames 2800...
603
- [2023-03-02 08:49:33,049][08718] Num frames 2900...
604
- [2023-03-02 08:49:33,144][08718] Avg episode rewards: #0: 4.777, true rewards: #0: 4.206
605
- [2023-03-02 08:49:33,145][08718] Avg episode reward: 4.777, avg true_objective: 4.206
606
- [2023-03-02 08:49:33,200][08718] Num frames 3000...
607
- [2023-03-02 08:49:33,295][08718] Num frames 3100...
608
- [2023-03-02 08:49:33,391][08718] Num frames 3200...
609
- [2023-03-02 08:49:33,442][08718] Avg episode rewards: #0: 4.500, true rewards: #0: 4.000
610
- [2023-03-02 08:49:33,442][08718] Avg episode reward: 4.500, avg true_objective: 4.000
611
- [2023-03-02 08:49:33,539][08718] Num frames 3300...
612
- [2023-03-02 08:49:33,635][08718] Num frames 3400...
613
- [2023-03-02 08:49:33,729][08718] Num frames 3500...
614
- [2023-03-02 08:49:33,824][08718] Num frames 3600...
615
- [2023-03-02 08:49:33,921][08718] Avg episode rewards: #0: 4.609, true rewards: #0: 4.053
616
- [2023-03-02 08:49:33,922][08718] Avg episode reward: 4.609, avg true_objective: 4.053
617
- [2023-03-02 08:49:33,972][08718] Num frames 3700...
618
- [2023-03-02 08:49:34,065][08718] Num frames 3800...
619
- [2023-03-02 08:49:34,160][08718] Num frames 3900...
620
- [2023-03-02 08:49:34,286][08718] Num frames 4000...
621
- [2023-03-02 08:49:34,375][08718] Avg episode rewards: #0: 4.532, true rewards: #0: 4.032
622
- [2023-03-02 08:49:34,376][08718] Avg episode reward: 4.532, avg true_objective: 4.032
623
- [2023-03-02 08:49:38,275][08718] Replay video saved to /home/gpu/train_dir/default_experiment/replay.mp4!
624
- [2023-03-02 08:50:05,350][08718] Loading existing experiment configuration from /home/gpu/train_dir/default_experiment/config.json
625
- [2023-03-02 08:50:05,350][08718] Overriding arg 'num_workers' with value 1 passed from command line
626
- [2023-03-02 08:50:05,350][08718] Adding new argument 'no_render'=True that is not in the saved config file!
627
- [2023-03-02 08:50:05,350][08718] Adding new argument 'save_video'=True that is not in the saved config file!
628
- [2023-03-02 08:50:05,350][08718] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file!
629
- [2023-03-02 08:50:05,350][08718] Adding new argument 'video_name'=None that is not in the saved config file!
630
- [2023-03-02 08:50:05,350][08718] Adding new argument 'max_num_frames'=100000 that is not in the saved config file!
631
- [2023-03-02 08:50:05,350][08718] Adding new argument 'max_num_episodes'=10 that is not in the saved config file!
632
- [2023-03-02 08:50:05,350][08718] Adding new argument 'push_to_hub'=True that is not in the saved config file!
633
- [2023-03-02 08:50:05,351][08718] Adding new argument 'hf_repository'='nhiro3303/rl_course_vizdoom_health_gathering_supreme' that is not in the saved config file!
634
- [2023-03-02 08:50:05,351][08718] Adding new argument 'policy_index'=0 that is not in the saved config file!
635
- [2023-03-02 08:50:05,351][08718] Adding new argument 'eval_deterministic'=False that is not in the saved config file!
636
- [2023-03-02 08:50:05,351][08718] Adding new argument 'train_script'=None that is not in the saved config file!
637
- [2023-03-02 08:50:05,351][08718] Adding new argument 'enjoy_script'=None that is not in the saved config file!
638
- [2023-03-02 08:50:05,351][08718] Using frameskip 1 and render_action_repeat=4 for evaluation
639
- [2023-03-02 08:50:05,354][08718] RunningMeanStd input shape: (3, 72, 128)
640
- [2023-03-02 08:50:05,354][08718] RunningMeanStd input shape: (1,)
641
- [2023-03-02 08:50:05,361][08718] ConvEncoder: input_channels=3
642
- [2023-03-02 08:50:05,386][08718] Conv encoder output size: 512
643
- [2023-03-02 08:50:05,387][08718] Policy head output size: 512
644
- [2023-03-02 08:50:05,412][08718] Loading state from checkpoint /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
645
- [2023-03-02 08:50:05,707][08718] Num frames 100...
646
- [2023-03-02 08:50:05,851][08718] Num frames 200...
647
- [2023-03-02 08:50:05,990][08718] Num frames 300...
648
- [2023-03-02 08:50:06,132][08718] Num frames 400...
649
- [2023-03-02 08:50:06,269][08718] Num frames 500...
650
- [2023-03-02 08:50:06,386][08718] Avg episode rewards: #0: 7.440, true rewards: #0: 5.440
651
- [2023-03-02 08:50:06,386][08718] Avg episode reward: 7.440, avg true_objective: 5.440
652
- [2023-03-02 08:50:06,471][08718] Num frames 600...
653
- [2023-03-02 08:50:06,616][08718] Num frames 700...
654
- [2023-03-02 08:50:06,758][08718] Num frames 800...
655
- [2023-03-02 08:50:06,895][08718] Num frames 900...
656
- [2023-03-02 08:50:06,983][08718] Avg episode rewards: #0: 5.640, true rewards: #0: 4.640
657
- [2023-03-02 08:50:06,984][08718] Avg episode reward: 5.640, avg true_objective: 4.640
658
- [2023-03-02 08:50:07,086][08718] Num frames 1000...
659
- [2023-03-02 08:50:07,222][08718] Num frames 1100...
660
- [2023-03-02 08:50:07,356][08718] Num frames 1200...
661
- [2023-03-02 08:50:07,500][08718] Num frames 1300...
662
- [2023-03-02 08:50:07,568][08718] Avg episode rewards: #0: 5.040, true rewards: #0: 4.373
663
- [2023-03-02 08:50:07,569][08718] Avg episode reward: 5.040, avg true_objective: 4.373
664
- [2023-03-02 08:50:07,692][08718] Num frames 1400...
665
- [2023-03-02 08:50:07,828][08718] Num frames 1500...
666
- [2023-03-02 08:50:07,966][08718] Num frames 1600...
667
- [2023-03-02 08:50:08,151][08718] Avg episode rewards: #0: 4.740, true rewards: #0: 4.240
668
- [2023-03-02 08:50:08,151][08718] Avg episode reward: 4.740, avg true_objective: 4.240
669
- [2023-03-02 08:50:08,156][08718] Num frames 1700...
670
- [2023-03-02 08:50:08,296][08718] Num frames 1800...
671
- [2023-03-02 08:50:08,431][08718] Num frames 1900...
672
- [2023-03-02 08:50:08,575][08718] Num frames 2000...
673
- [2023-03-02 08:50:08,738][08718] Avg episode rewards: #0: 4.560, true rewards: #0: 4.160
674
- [2023-03-02 08:50:08,738][08718] Avg episode reward: 4.560, avg true_objective: 4.160
675
- [2023-03-02 08:50:08,771][08718] Num frames 2100...
676
- [2023-03-02 08:50:08,912][08718] Num frames 2200...
677
- [2023-03-02 08:50:09,052][08718] Num frames 2300...
678
- [2023-03-02 08:50:09,170][08718] Avg episode rewards: #0: 4.238, true rewards: #0: 3.905
679
- [2023-03-02 08:50:09,170][08718] Avg episode reward: 4.238, avg true_objective: 3.905
680
- [2023-03-02 08:50:09,250][08718] Num frames 2400...
681
- [2023-03-02 08:50:09,384][08718] Num frames 2500...
682
- [2023-03-02 08:50:09,519][08718] Num frames 2600...
683
- [2023-03-02 08:50:09,654][08718] Num frames 2700...
684
- [2023-03-02 08:50:09,832][08718] Avg episode rewards: #0: 4.416, true rewards: #0: 3.987
685
- [2023-03-02 08:50:09,832][08718] Avg episode reward: 4.416, avg true_objective: 3.987
686
- [2023-03-02 08:50:09,843][08718] Num frames 2800...
687
- [2023-03-02 08:50:09,975][08718] Num frames 2900...
688
- [2023-03-02 08:50:10,105][08718] Num frames 3000...
689
- [2023-03-02 08:50:10,234][08718] Num frames 3100...
690
- [2023-03-02 08:50:10,390][08718] Avg episode rewards: #0: 4.344, true rewards: #0: 3.969
691
- [2023-03-02 08:50:10,390][08718] Avg episode reward: 4.344, avg true_objective: 3.969
692
- [2023-03-02 08:50:10,433][08718] Num frames 3200...
693
- [2023-03-02 08:50:10,575][08718] Num frames 3300...
694
- [2023-03-02 08:50:10,709][08718] Num frames 3400...
695
- [2023-03-02 08:50:10,843][08718] Num frames 3500...
696
- [2023-03-02 08:50:10,975][08718] Avg episode rewards: #0: 4.288, true rewards: #0: 3.954
697
- [2023-03-02 08:50:10,976][08718] Avg episode reward: 4.288, avg true_objective: 3.954
698
- [2023-03-02 08:50:11,036][08718] Num frames 3600...
699
- [2023-03-02 08:50:11,171][08718] Num frames 3700...
700
- [2023-03-02 08:50:11,309][08718] Num frames 3800...
701
- [2023-03-02 08:50:11,448][08718] Num frames 3900...
702
- [2023-03-02 08:50:11,567][08718] Avg episode rewards: #0: 4.243, true rewards: #0: 3.943
703
- [2023-03-02 08:50:11,567][08718] Avg episode reward: 4.243, avg true_objective: 3.943
704
- [2023-03-02 08:50:15,364][08718] Replay video saved to /home/gpu/train_dir/default_experiment/replay.mp4!
 
1
+ [2023-03-02 09:58:26,873][08626] Saving configuration to /home/gpu/train_dir/default_experiment/config.json...
2
+ [2023-03-02 09:58:26,873][08626] Rollout worker 0 uses device cpu
3
+ [2023-03-02 09:58:26,873][08626] Rollout worker 1 uses device cpu
4
+ [2023-03-02 09:58:26,874][08626] Rollout worker 2 uses device cpu
5
+ [2023-03-02 09:58:26,874][08626] Rollout worker 3 uses device cpu
6
+ [2023-03-02 09:58:26,874][08626] Rollout worker 4 uses device cpu
7
+ [2023-03-02 09:58:26,874][08626] Rollout worker 5 uses device cpu
8
+ [2023-03-02 09:58:26,874][08626] Rollout worker 6 uses device cpu
9
+ [2023-03-02 09:58:26,874][08626] Rollout worker 7 uses device cpu
10
+ [2023-03-02 09:58:26,902][08626] Using GPUs [0] for process 0 (actually maps to GPUs [0])
11
+ [2023-03-02 09:58:26,902][08626] InferenceWorker_p0-w0: min num requests: 2
12
+ [2023-03-02 09:58:26,918][08626] Starting all processes...
13
+ [2023-03-02 09:58:26,919][08626] Starting process learner_proc0
14
+ [2023-03-02 09:58:27,574][08626] Starting all processes...
15
+ [2023-03-02 09:58:27,577][08626] Starting process inference_proc0-0
16
+ [2023-03-02 09:58:27,577][08626] Starting process rollout_proc0
17
+ [2023-03-02 09:58:27,577][08684] Using GPUs [0] for process 0 (actually maps to GPUs [0])
18
+ [2023-03-02 09:58:27,578][08684] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0
19
+ [2023-03-02 09:58:27,578][08626] Starting process rollout_proc1
20
+ [2023-03-02 09:58:27,578][08626] Starting process rollout_proc2
21
+ [2023-03-02 09:58:27,578][08626] Starting process rollout_proc3
22
+ [2023-03-02 09:58:27,580][08626] Starting process rollout_proc4
23
+ [2023-03-02 09:58:27,582][08626] Starting process rollout_proc5
24
+ [2023-03-02 09:58:27,582][08626] Starting process rollout_proc6
25
+ [2023-03-02 09:58:27,583][08626] Starting process rollout_proc7
26
+ [2023-03-02 09:58:27,612][08684] Num visible devices: 1
27
+ [2023-03-02 09:58:27,637][08684] Starting seed is not provided
28
+ [2023-03-02 09:58:27,637][08684] Using GPUs [0] for process 0 (actually maps to GPUs [0])
29
+ [2023-03-02 09:58:27,638][08684] Initializing actor-critic model on device cuda:0
30
+ [2023-03-02 09:58:27,638][08684] RunningMeanStd input shape: (3, 72, 128)
31
+ [2023-03-02 09:58:27,639][08684] RunningMeanStd input shape: (1,)
32
+ [2023-03-02 09:58:27,651][08684] ConvEncoder: input_channels=3
33
+ [2023-03-02 09:58:27,870][08684] Conv encoder output size: 512
34
+ [2023-03-02 09:58:27,870][08684] Policy head output size: 512
35
+ [2023-03-02 09:58:27,881][08684] Created Actor Critic model with architecture:
36
+ [2023-03-02 09:58:27,881][08684] ActorCriticSharedWeights(
37
  (obs_normalizer): ObservationNormalizer(
38
  (running_mean_std): RunningMeanStdDictInPlace(
39
  (running_mean_std): ModuleDict(
 
74
  (distribution_linear): Linear(in_features=512, out_features=5, bias=True)
75
  )
76
  )
77
+ [2023-03-02 09:58:28,670][08715] Worker 2 uses CPU cores [4, 5]
78
+ [2023-03-02 09:58:28,692][08713] Worker 0 uses CPU cores [0, 1]
79
+ [2023-03-02 09:58:28,721][08717] Worker 4 uses CPU cores [8, 9]
80
+ [2023-03-02 09:58:28,722][08720] Worker 7 uses CPU cores [14, 15]
81
+ [2023-03-02 09:58:28,727][08719] Worker 6 uses CPU cores [12, 13]
82
+ [2023-03-02 09:58:28,750][08718] Worker 5 uses CPU cores [10, 11]
83
+ [2023-03-02 09:58:28,752][08716] Worker 3 uses CPU cores [6, 7]
84
+ [2023-03-02 09:58:28,759][08714] Worker 1 uses CPU cores [2, 3]
85
+ [2023-03-02 09:58:28,776][08712] Using GPUs [0] for process 0 (actually maps to GPUs [0])
86
+ [2023-03-02 09:58:28,777][08712] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0
87
+ [2023-03-02 09:58:28,819][08712] Num visible devices: 1
88
+ [2023-03-02 09:58:29,994][08684] Using optimizer <class 'torch.optim.adam.Adam'>
89
+ [2023-03-02 09:58:29,996][08684] No checkpoints found
90
+ [2023-03-02 09:58:29,996][08684] Did not load from checkpoint, starting from scratch!
91
+ [2023-03-02 09:58:29,996][08684] Initialized policy 0 weights for model version 0
92
+ [2023-03-02 09:58:29,999][08684] LearnerWorker_p0 finished initialization!
93
+ [2023-03-02 09:58:29,999][08684] Using GPUs [0] for process 0 (actually maps to GPUs [0])
94
+ [2023-03-02 09:58:30,101][08712] Unhandled exception CUDA error: invalid resource handle
95
  CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
96
  For debugging consider passing CUDA_LAUNCH_BLOCKING=1. in evt loop inference_proc0-0_evt_loop
97
+ [2023-03-02 09:58:30,375][08626] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
98
+ [2023-03-02 09:58:35,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
99
+ [2023-03-02 09:58:40,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
100
+ [2023-03-02 09:58:45,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
101
+ [2023-03-02 09:58:46,897][08626] Heartbeat connected on Batcher_0
102
+ [2023-03-02 09:58:46,899][08626] Heartbeat connected on LearnerWorker_p0
103
+ [2023-03-02 09:58:46,905][08626] Heartbeat connected on RolloutWorker_w0
104
+ [2023-03-02 09:58:46,906][08626] Heartbeat connected on RolloutWorker_w1
105
+ [2023-03-02 09:58:46,908][08626] Heartbeat connected on RolloutWorker_w2
106
+ [2023-03-02 09:58:46,910][08626] Heartbeat connected on RolloutWorker_w3
107
+ [2023-03-02 09:58:46,912][08626] Heartbeat connected on RolloutWorker_w4
108
+ [2023-03-02 09:58:46,914][08626] Heartbeat connected on RolloutWorker_w5
109
+ [2023-03-02 09:58:46,916][08626] Heartbeat connected on RolloutWorker_w6
110
+ [2023-03-02 09:58:46,918][08626] Heartbeat connected on RolloutWorker_w7
111
+ [2023-03-02 09:58:50,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
112
+ [2023-03-02 09:58:55,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
113
+ [2023-03-02 09:59:00,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
114
+ [2023-03-02 09:59:05,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
115
+ [2023-03-02 09:59:10,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
116
+ [2023-03-02 09:59:15,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
117
+ [2023-03-02 09:59:20,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
118
+ [2023-03-02 09:59:25,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
119
+ [2023-03-02 09:59:30,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
120
+ [2023-03-02 09:59:35,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
121
+ [2023-03-02 09:59:40,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
122
+ [2023-03-02 09:59:45,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
123
+ [2023-03-02 09:59:50,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
124
+ [2023-03-02 09:59:55,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
125
+ [2023-03-02 10:00:00,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
126
+ [2023-03-02 10:00:05,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
127
+ [2023-03-02 10:00:10,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
128
+ [2023-03-02 10:00:15,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
129
+ [2023-03-02 10:00:20,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
130
+ [2023-03-02 10:00:25,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
131
+ [2023-03-02 10:00:25,377][08684] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
132
+ [2023-03-02 10:00:30,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
133
+ [2023-03-02 10:00:35,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
134
+ [2023-03-02 10:00:40,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
135
+ [2023-03-02 10:00:45,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
136
+ [2023-03-02 10:00:50,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
137
+ [2023-03-02 10:00:55,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
138
+ [2023-03-02 10:01:00,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
139
+ [2023-03-02 10:01:05,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
140
+ [2023-03-02 10:01:10,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
141
+ [2023-03-02 10:01:15,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
142
+ [2023-03-02 10:01:20,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
143
+ [2023-03-02 10:01:25,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
144
+ [2023-03-02 10:01:30,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
145
+ [2023-03-02 10:01:35,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
146
+ [2023-03-02 10:01:40,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
147
+ [2023-03-02 10:01:45,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
148
+ [2023-03-02 10:01:50,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
149
+ [2023-03-02 10:01:55,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
150
+ [2023-03-02 10:02:00,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
151
+ [2023-03-02 10:02:05,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
152
+ [2023-03-02 10:02:10,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
153
+ [2023-03-02 10:02:15,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
154
+ [2023-03-02 10:02:20,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
155
+ [2023-03-02 10:02:25,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
156
+ [2023-03-02 10:02:25,376][08684] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
157
+ [2023-03-02 10:02:30,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
158
+ [2023-03-02 10:02:35,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
159
+ [2023-03-02 10:02:40,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
160
+ [2023-03-02 10:02:45,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
161
+ [2023-03-02 10:02:50,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
162
+ [2023-03-02 10:02:55,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
163
+ [2023-03-02 10:03:00,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
164
+ [2023-03-02 10:03:05,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
165
+ [2023-03-02 10:03:10,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
166
+ [2023-03-02 10:03:15,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
167
+ [2023-03-02 10:03:20,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
168
+ [2023-03-02 10:03:25,375][08626] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
169
+ [2023-03-02 10:03:27,918][08626] Keyboard interrupt detected in the event loop EvtLoop [Runner_EvtLoop, process=main process 8626], exiting...
170
+ [2023-03-02 10:03:27,919][08717] Stopping RolloutWorker_w4...
171
+ [2023-03-02 10:03:27,919][08714] Stopping RolloutWorker_w1...
172
+ [2023-03-02 10:03:27,919][08626] Runner profile tree view:
173
+ main_loop: 301.0005
174
+ [2023-03-02 10:03:27,919][08718] Stopping RolloutWorker_w5...
175
+ [2023-03-02 10:03:27,919][08715] Stopping RolloutWorker_w2...
176
+ [2023-03-02 10:03:27,919][08717] Loop rollout_proc4_evt_loop terminating...
177
+ [2023-03-02 10:03:27,919][08626] Collected {0: 0}, FPS: 0.0
178
+ [2023-03-02 10:03:27,919][08719] Stopping RolloutWorker_w6...
179
+ [2023-03-02 10:03:27,919][08713] Stopping RolloutWorker_w0...
180
+ [2023-03-02 10:03:27,919][08720] Stopping RolloutWorker_w7...
181
+ [2023-03-02 10:03:27,919][08714] Loop rollout_proc1_evt_loop terminating...
182
+ [2023-03-02 10:03:27,919][08716] Stopping RolloutWorker_w3...
183
+ [2023-03-02 10:03:27,919][08684] Stopping Batcher_0...
184
+ [2023-03-02 10:03:27,919][08718] Loop rollout_proc5_evt_loop terminating...
185
+ [2023-03-02 10:03:27,919][08713] Loop rollout_proc0_evt_loop terminating...
186
+ [2023-03-02 10:03:27,919][08715] Loop rollout_proc2_evt_loop terminating...
187
+ [2023-03-02 10:03:27,919][08719] Loop rollout_proc6_evt_loop terminating...
188
+ [2023-03-02 10:03:27,919][08720] Loop rollout_proc7_evt_loop terminating...
189
+ [2023-03-02 10:03:27,919][08684] Loop batcher_evt_loop terminating...
190
+ [2023-03-02 10:03:27,919][08716] Loop rollout_proc3_evt_loop terminating...
191
+ [2023-03-02 10:03:27,920][08684] Saving /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
192
+ [2023-03-02 10:03:27,945][08684] Stopping LearnerWorker_p0...
193
+ [2023-03-02 10:03:27,945][08684] Loop learner_proc0_evt_loop terminating...
194
+ [2023-03-02 10:03:28,008][08626] Loading existing experiment configuration from /home/gpu/train_dir/default_experiment/config.json
195
+ [2023-03-02 10:03:28,009][08626] Overriding arg 'num_workers' with value 1 passed from command line
196
+ [2023-03-02 10:03:28,009][08626] Adding new argument 'no_render'=True that is not in the saved config file!
197
+ [2023-03-02 10:03:28,009][08626] Adding new argument 'save_video'=True that is not in the saved config file!
198
+ [2023-03-02 10:03:28,009][08626] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file!
199
+ [2023-03-02 10:03:28,009][08626] Adding new argument 'video_name'=None that is not in the saved config file!
200
+ [2023-03-02 10:03:28,009][08626] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file!
201
+ [2023-03-02 10:03:28,009][08626] Adding new argument 'max_num_episodes'=10 that is not in the saved config file!
202
+ [2023-03-02 10:03:28,009][08626] Adding new argument 'push_to_hub'=False that is not in the saved config file!
203
+ [2023-03-02 10:03:28,010][08626] Adding new argument 'hf_repository'=None that is not in the saved config file!
204
+ [2023-03-02 10:03:28,010][08626] Adding new argument 'policy_index'=0 that is not in the saved config file!
205
+ [2023-03-02 10:03:28,010][08626] Adding new argument 'eval_deterministic'=False that is not in the saved config file!
206
+ [2023-03-02 10:03:28,010][08626] Adding new argument 'train_script'=None that is not in the saved config file!
207
+ [2023-03-02 10:03:28,010][08626] Adding new argument 'enjoy_script'=None that is not in the saved config file!
208
+ [2023-03-02 10:03:28,010][08626] Using frameskip 1 and render_action_repeat=4 for evaluation
209
+ [2023-03-02 10:03:28,020][08626] Doom resolution: 160x120, resize resolution: (128, 72)
210
+ [2023-03-02 10:03:28,021][08626] RunningMeanStd input shape: (3, 72, 128)
211
+ [2023-03-02 10:03:28,023][08626] RunningMeanStd input shape: (1,)
212
+ [2023-03-02 10:03:28,035][08626] ConvEncoder: input_channels=3
213
+ [2023-03-02 10:03:28,222][08626] Conv encoder output size: 512
214
+ [2023-03-02 10:03:28,223][08626] Policy head output size: 512
215
+ [2023-03-02 10:03:29,855][08626] Loading state from checkpoint /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
216
+ [2023-03-02 10:03:30,602][08626] Num frames 100...
217
+ [2023-03-02 10:03:30,696][08626] Num frames 200...
218
+ [2023-03-02 10:03:30,791][08626] Num frames 300...
219
+ [2023-03-02 10:03:30,922][08626] Avg episode rewards: #0: 3.840, true rewards: #0: 3.840
220
+ [2023-03-02 10:03:30,922][08626] Avg episode reward: 3.840, avg true_objective: 3.840
221
+ [2023-03-02 10:03:30,940][08626] Num frames 400...
222
+ [2023-03-02 10:03:31,034][08626] Num frames 500...
223
+ [2023-03-02 10:03:31,127][08626] Num frames 600...
224
+ [2023-03-02 10:03:31,222][08626] Num frames 700...
225
+ [2023-03-02 10:03:31,338][08626] Avg episode rewards: #0: 3.840, true rewards: #0: 3.840
226
+ [2023-03-02 10:03:31,338][08626] Avg episode reward: 3.840, avg true_objective: 3.840
227
+ [2023-03-02 10:03:31,370][08626] Num frames 800...
228
+ [2023-03-02 10:03:31,465][08626] Num frames 900...
229
+ [2023-03-02 10:03:31,560][08626] Num frames 1000...
230
+ [2023-03-02 10:03:31,654][08626] Num frames 1100...
231
+ [2023-03-02 10:03:31,748][08626] Num frames 1200...
232
+ [2023-03-02 10:03:31,816][08626] Avg episode rewards: #0: 4.387, true rewards: #0: 4.053
233
+ [2023-03-02 10:03:31,816][08626] Avg episode reward: 4.387, avg true_objective: 4.053
234
+ [2023-03-02 10:03:31,895][08626] Num frames 1300...
235
+ [2023-03-02 10:03:31,993][08626] Num frames 1400...
236
+ [2023-03-02 10:03:32,088][08626] Num frames 1500...
237
+ [2023-03-02 10:03:32,185][08626] Num frames 1600...
238
+ [2023-03-02 10:03:32,236][08626] Avg episode rewards: #0: 4.250, true rewards: #0: 4.000
239
+ [2023-03-02 10:03:32,236][08626] Avg episode reward: 4.250, avg true_objective: 4.000
240
+ [2023-03-02 10:03:32,332][08626] Num frames 1700...
241
+ [2023-03-02 10:03:32,427][08626] Num frames 1800...
242
+ [2023-03-02 10:03:32,520][08626] Num frames 1900...
243
+ [2023-03-02 10:03:32,614][08626] Num frames 2000...
244
+ [2023-03-02 10:03:32,711][08626] Avg episode rewards: #0: 4.496, true rewards: #0: 4.096
245
+ [2023-03-02 10:03:32,711][08626] Avg episode reward: 4.496, avg true_objective: 4.096
246
+ [2023-03-02 10:03:32,760][08626] Num frames 2100...
247
+ [2023-03-02 10:03:32,853][08626] Num frames 2200...
248
+ [2023-03-02 10:03:32,947][08626] Num frames 2300...
249
+ [2023-03-02 10:03:33,041][08626] Num frames 2400...
250
+ [2023-03-02 10:03:33,123][08626] Avg episode rewards: #0: 4.387, true rewards: #0: 4.053
251
+ [2023-03-02 10:03:33,124][08626] Avg episode reward: 4.387, avg true_objective: 4.053
252
+ [2023-03-02 10:03:33,188][08626] Num frames 2500...
253
+ [2023-03-02 10:03:33,281][08626] Num frames 2600...
254
+ [2023-03-02 10:03:33,377][08626] Num frames 2700...
255
+ [2023-03-02 10:03:33,471][08626] Num frames 2800...
256
+ [2023-03-02 10:03:33,538][08626] Avg episode rewards: #0: 4.309, true rewards: #0: 4.023
257
+ [2023-03-02 10:03:33,539][08626] Avg episode reward: 4.309, avg true_objective: 4.023
258
+ [2023-03-02 10:03:33,618][08626] Num frames 2900...
259
+ [2023-03-02 10:03:33,715][08626] Num frames 3000...
260
+ [2023-03-02 10:03:33,810][08626] Num frames 3100...
261
+ [2023-03-02 10:03:33,906][08626] Num frames 3200...
262
+ [2023-03-02 10:03:34,022][08626] Avg episode rewards: #0: 4.455, true rewards: #0: 4.080
263
+ [2023-03-02 10:03:34,022][08626] Avg episode reward: 4.455, avg true_objective: 4.080
264
+ [2023-03-02 10:03:34,061][08626] Num frames 3300...
265
+ [2023-03-02 10:03:34,168][08626] Num frames 3400...
266
+ [2023-03-02 10:03:34,271][08626] Num frames 3500...
267
+ [2023-03-02 10:03:34,366][08626] Num frames 3600...
268
+ [2023-03-02 10:03:34,465][08626] Avg episode rewards: #0: 4.387, true rewards: #0: 4.053
269
+ [2023-03-02 10:03:34,465][08626] Avg episode reward: 4.387, avg true_objective: 4.053
270
+ [2023-03-02 10:03:34,520][08626] Num frames 3700...
271
+ [2023-03-02 10:03:34,622][08626] Num frames 3800...
272
+ [2023-03-02 10:03:34,726][08626] Num frames 3900...
273
+ [2023-03-02 10:03:34,829][08626] Num frames 4000...
274
+ [2023-03-02 10:03:34,915][08626] Avg episode rewards: #0: 4.332, true rewards: #0: 4.032
275
+ [2023-03-02 10:03:34,915][08626] Avg episode reward: 4.332, avg true_objective: 4.032
276
+ [2023-03-02 10:03:38,820][08626] Replay video saved to /home/gpu/train_dir/default_experiment/replay.mp4!
277
+ [2023-03-02 10:46:22,903][08626] Loading existing experiment configuration from /home/gpu/train_dir/default_experiment/config.json
278
+ [2023-03-02 10:46:22,903][08626] Overriding arg 'num_workers' with value 1 passed from command line
279
+ [2023-03-02 10:46:22,903][08626] Adding new argument 'no_render'=True that is not in the saved config file!
280
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'save_video'=True that is not in the saved config file!
281
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file!
282
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'video_name'=None that is not in the saved config file!
283
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'max_num_frames'=100000 that is not in the saved config file!
284
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'max_num_episodes'=10 that is not in the saved config file!
285
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'push_to_hub'=True that is not in the saved config file!
286
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'hf_repository'='nhiro3303/rl_course_vizdoom_health_gathering_supreme' that is not in the saved config file!
287
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'policy_index'=0 that is not in the saved config file!
288
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'eval_deterministic'=False that is not in the saved config file!
289
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'train_script'=None that is not in the saved config file!
290
+ [2023-03-02 10:46:22,904][08626] Adding new argument 'enjoy_script'=None that is not in the saved config file!
291
+ [2023-03-02 10:46:22,905][08626] Using frameskip 1 and render_action_repeat=4 for evaluation
292
+ [2023-03-02 10:46:22,908][08626] RunningMeanStd input shape: (3, 72, 128)
293
+ [2023-03-02 10:46:22,908][08626] RunningMeanStd input shape: (1,)
294
+ [2023-03-02 10:46:22,914][08626] ConvEncoder: input_channels=3
295
+ [2023-03-02 10:46:22,938][08626] Conv encoder output size: 512
296
+ [2023-03-02 10:46:22,938][08626] Policy head output size: 512
297
+ [2023-03-02 10:46:22,958][08626] Loading state from checkpoint /home/gpu/train_dir/default_experiment/checkpoint_p0/checkpoint_000000000_0.pth...
298
+ [2023-03-02 10:46:23,267][08626] Num frames 100...
299
+ [2023-03-02 10:46:23,410][08626] Num frames 200...
300
+ [2023-03-02 10:46:23,551][08626] Num frames 300...
301
+ [2023-03-02 10:46:23,718][08626] Avg episode rewards: #0: 3.840, true rewards: #0: 3.840
302
+ [2023-03-02 10:46:23,719][08626] Avg episode reward: 3.840, avg true_objective: 3.840
303
+ [2023-03-02 10:46:23,745][08626] Num frames 400...
304
+ [2023-03-02 10:46:23,884][08626] Num frames 500...
305
+ [2023-03-02 10:46:24,021][08626] Num frames 600...
306
+ [2023-03-02 10:46:24,177][08626] Num frames 700...
307
+ [2023-03-02 10:46:24,330][08626] Avg episode rewards: #0: 3.840, true rewards: #0: 3.840
308
+ [2023-03-02 10:46:24,330][08626] Avg episode reward: 3.840, avg true_objective: 3.840
309
+ [2023-03-02 10:46:24,372][08626] Num frames 800...
310
+ [2023-03-02 10:46:24,513][08626] Num frames 900...
311
+ [2023-03-02 10:46:24,647][08626] Num frames 1000...
312
+ [2023-03-02 10:46:24,777][08626] Num frames 1100...
313
+ [2023-03-02 10:46:24,901][08626] Avg episode rewards: #0: 3.840, true rewards: #0: 3.840
314
+ [2023-03-02 10:46:24,901][08626] Avg episode reward: 3.840, avg true_objective: 3.840
315
+ [2023-03-02 10:46:24,973][08626] Num frames 1200...
316
+ [2023-03-02 10:46:25,108][08626] Num frames 1300...
317
+ [2023-03-02 10:46:25,248][08626] Num frames 1400...
318
+ [2023-03-02 10:46:25,393][08626] Num frames 1500...
319
+ [2023-03-02 10:46:25,525][08626] Num frames 1600...
320
+ [2023-03-02 10:46:25,575][08626] Avg episode rewards: #0: 4.250, true rewards: #0: 4.000
321
+ [2023-03-02 10:46:25,576][08626] Avg episode reward: 4.250, avg true_objective: 4.000
322
+ [2023-03-02 10:46:25,707][08626] Num frames 1700...
323
+ [2023-03-02 10:46:25,842][08626] Num frames 1800...
324
+ [2023-03-02 10:46:25,977][08626] Num frames 1900...
325
+ [2023-03-02 10:46:26,145][08626] Avg episode rewards: #0: 4.168, true rewards: #0: 3.968
326
+ [2023-03-02 10:46:26,145][08626] Avg episode reward: 4.168, avg true_objective: 3.968
327
+ [2023-03-02 10:46:26,166][08626] Num frames 2000...
328
+ [2023-03-02 10:46:26,299][08626] Num frames 2100...
329
+ [2023-03-02 10:46:26,432][08626] Num frames 2200...
330
+ [2023-03-02 10:46:26,557][08626] Num frames 2300...
331
+ [2023-03-02 10:46:26,708][08626] Avg episode rewards: #0: 4.113, true rewards: #0: 3.947
332
+ [2023-03-02 10:46:26,708][08626] Avg episode reward: 4.113, avg true_objective: 3.947
333
+ [2023-03-02 10:46:26,755][08626] Num frames 2400...
334
+ [2023-03-02 10:46:26,883][08626] Num frames 2500...
335
+ [2023-03-02 10:46:27,016][08626] Num frames 2600...
336
+ [2023-03-02 10:46:27,157][08626] Num frames 2700...
337
+ [2023-03-02 10:46:27,284][08626] Avg episode rewards: #0: 4.074, true rewards: #0: 3.931
338
+ [2023-03-02 10:46:27,284][08626] Avg episode reward: 4.074, avg true_objective: 3.931
339
+ [2023-03-02 10:46:27,354][08626] Num frames 2800...
340
+ [2023-03-02 10:46:27,487][08626] Num frames 2900...
341
+ [2023-03-02 10:46:27,620][08626] Num frames 3000...
342
+ [2023-03-02 10:46:27,761][08626] Num frames 3100...
343
+ [2023-03-02 10:46:27,862][08626] Avg episode rewards: #0: 4.045, true rewards: #0: 3.920
344
+ [2023-03-02 10:46:27,862][08626] Avg episode reward: 4.045, avg true_objective: 3.920
345
+ [2023-03-02 10:46:27,952][08626] Num frames 3200...
346
+ [2023-03-02 10:46:28,092][08626] Num frames 3300...
347
+ [2023-03-02 10:46:28,231][08626] Num frames 3400...
348
+ [2023-03-02 10:46:28,363][08626] Num frames 3500...
349
+ [2023-03-02 10:46:28,530][08626] Avg episode rewards: #0: 4.204, true rewards: #0: 3.982
350
+ [2023-03-02 10:46:28,531][08626] Avg episode reward: 4.204, avg true_objective: 3.982
351
+ [2023-03-02 10:46:28,558][08626] Num frames 3600...
352
+ [2023-03-02 10:46:28,700][08626] Num frames 3700...
353
+ [2023-03-02 10:46:28,836][08626] Num frames 3800...
354
+ [2023-03-02 10:46:28,968][08626] Num frames 3900...
355
+ [2023-03-02 10:46:29,103][08626] Num frames 4000...
356
+ [2023-03-02 10:46:29,203][08626] Avg episode rewards: #0: 4.332, true rewards: #0: 4.032
357
+ [2023-03-02 10:46:29,203][08626] Avg episode reward: 4.332, avg true_objective: 4.032
358
+ [2023-03-02 10:46:33,125][08626] Replay video saved to /home/gpu/train_dir/default_experiment/replay.mp4!