File size: 75,044 Bytes
c011401 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 |
[2025-01-21 05:58:18,764 I 20340 20340] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-21 05:58:18,765 I 20340 20340] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-21 05:58:18,770 I 20340 20340] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 9f6a3ba5c12bdaae5991e0c6ce9a10b1c06e51056583b7869135876e
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-21 05:58:18,773 I 20340 20340] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 58630.
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 23 total (4 active)
Queueing time: mean = 82.234 ms, max = 209.811 ms, min = 1.348 us, total = 1.891 s
Execution time: mean = 9.126 ms, total = 209.887 ms
Event stats:
GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 23.315 ms, total = 209.831 ms, Queueing time: mean = 162.795 ms, max = 209.465 ms, min = 1.348 us, total = 1.465 s
GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 5.874 us, total = 29.369 us, Queueing time: mean = 36.556 us, max = 39.816 us, min = 33.188 us, total = 182.779 us
PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 1.955 us, total = 7.820 us, Queueing time: mean = 104.882 ms, max = 209.811 ms, min = 209.718 ms, total = 419.529 ms
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 5.213 us, total = 10.426 us, Queueing time: mean = 3.260 ms, max = 6.396 ms, min = 124.659 us, total = 6.520 ms
GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 8.936 us, total = 8.936 us, Queueing time: mean = 2.067 us, max = 2.067 us, min = 2.067 us, total = 2.067 us
ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 4 total (1 active)
Queueing time: mean = 110.391 us, max = 389.975 us, min = 12.503 us, total = 441.565 us
Execution time: mean = 70.540 us, total = 282.161 us
Event stats:
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 137.157 us, total = 274.315 us, Queueing time: mean = 214.531 us, max = 389.975 us, min = 39.087 us, total = 429.062 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us
GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 4 total (1 active)
Queueing time: mean = 1.141 ms, max = 2.957 ms, min = 113.015 us, total = 4.564 ms
Execution time: mean = 53.532 us, total = 214.128 us
Event stats:
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 79.139 us, total = 158.278 us, Queueing time: mean = 1.535 ms, max = 2.957 ms, min = 113.015 us, total = 3.070 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 4 total (0 active)
Queueing time: mean = 303.213 us, max = 1.158 ms, min = 12.186 us, total = 1.213 ms
Execution time: mean = 47.383 us, total = 189.532 us
Event stats:
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 94.208 us, total = 188.416 us, Queueing time: mean = 593.180 us, max = 1.158 ms, min = 28.000 us, total = 1.186 ms
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us
[2025-01-21 05:58:21,131 I 20340 20340] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c
[2025-01-21 05:58:21,131 I 20340 20340] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c
[2025-01-21 05:58:21,131 I 20340 20340] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c registered, will try to reschedule all the infeasible placement groups.
[2025-01-21 05:58:21,136 I 20340 20422] (gcs_server) ray_syncer.cc:377: Get connection node_id=331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c
[2025-01-21 05:58:21,922 I 20340 20340] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 20269
[2025-01-21 05:58:21,922 I 20340 20340] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 20269
[2025-01-21 05:58:28,780 W 20340 20363] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:58636: Failed to connect to remote host: Connection refused; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster.
[2025-01-21 05:59:18,981 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 12
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 13
- cumulative published bytes: 12345
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 3
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 2
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 1
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:59:18,982 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 1117 total (4 active)
Queueing time: mean = 1.770 ms, max = 209.811 ms, min = 740.000 ns, total = 1.977 s
Execution time: mean = 436.269 us, total = 487.313 ms
Event stats:
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 21.988 us, total = 5.321 ms, Queueing time: mean = 35.197 us, max = 6.396 ms, min = 2.670 us, total = 8.518 ms
GcsInMemoryStore.Put - 143 total (0 active), Execution time: mean = 1.478 ms, total = 211.380 ms, Queueing time: mean = 10.282 ms, max = 209.465 ms, min = 1.325 us, total = 1.470 s
GcsInMemoryStore.Get - 72 total (0 active), Execution time: mean = 16.750 us, total = 1.206 ms, Queueing time: mean = 21.673 us, max = 158.434 us, min = 2.067 us, total = 1.560 ms
InternalKVGcsService.grpc_server.InternalKVGet - 71 total (0 active), Execution time: mean = 834.058 us, total = 59.218 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 71 total (0 active), Execution time: mean = 33.602 us, total = 2.386 ms, Queueing time: mean = 433.801 us, max = 8.337 ms, min = 11.662 us, total = 30.800 ms
RayletLoadPulled - 60 total (1 active), Execution time: mean = 241.451 us, total = 14.487 ms, Queueing time: mean = 63.665 us, max = 183.264 us, min = 16.365 us, total = 3.820 ms
NodeManagerService.grpc_client.GetResourceLoad - 57 total (0 active), Execution time: mean = 1.499 ms, total = 85.458 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 57 total (0 active), Execution time: mean = 73.211 us, total = 4.173 ms, Queueing time: mean = 101.909 us, max = 178.023 us, min = 19.180 us, total = 5.809 ms
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 39 total (0 active), Execution time: mean = 32.339 us, total = 1.261 ms, Queueing time: mean = 79.184 us, max = 203.571 us, min = 8.800 us, total = 3.088 ms
InternalKVGcsService.grpc_server.InternalKVPut - 39 total (0 active), Execution time: mean = 497.287 us, total = 19.394 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 35.401 us, total = 1.204 ms, Queueing time: mean = 590.521 us, max = 8.472 ms, min = 11.517 us, total = 20.078 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 938.702 us, total = 31.916 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 30.708 us, total = 736.999 us, Queueing time: mean = 107.406 us, max = 207.471 us, min = 15.170 us, total = 2.578 ms
NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 521.698 us, total = 12.521 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 26.838 us, total = 563.603 us, Queueing time: mean = 126.532 us, max = 257.619 us, min = 30.842 us, total = 2.657 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 491.931 us, total = 10.331 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 20 total (1 active), Execution time: mean = 9.595 us, total = 191.895 us, Queueing time: mean = 146.339 us, max = 1.634 ms, min = 30.256 us, total = 2.927 ms
HealthCheck - 18 total (0 active), Execution time: mean = 8.213 us, total = 147.828 us, Queueing time: mean = 110.846 us, max = 204.653 us, min = 22.112 us, total = 1.995 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 644.168 us, total = 7.730 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 81.180 us, total = 974.157 us, Queueing time: mean = 105.073 us, max = 201.135 us, min = 20.386 us, total = 1.261 ms
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.268 us, total = 49.609 us, Queueing time: mean = 32.229 us, max = 39.816 us, min = 10.594 us, total = 193.373 us
GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.140 ms, total = 6.837 ms, Queueing time: mean = 58.245 us, max = 119.431 us, min = 20.387 us, total = 349.473 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 199.609 us, total = 998.045 us, Queueing time: mean = 58.098 us, max = 128.737 us, min = 15.591 us, total = 290.490 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 685.197 us, total = 3.426 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 117.076 us, total = 468.306 us, Queueing time: mean = 104.964 ms, max = 209.811 ms, min = 27.036 us, total = 419.855 ms
GcsResourceManager::Update - 2 total (0 active), Execution time: mean = 82.721 us, total = 165.442 us, Queueing time: mean = 106.719 us, max = 117.209 us, min = 96.230 us, total = 213.439 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 17.533 us, total = 17.533 us, Queueing time: mean = 94.945 us, max = 94.945 us, min = 94.945 us, total = 94.945 us
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 479.251 us, total = 479.251 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 33.033 us, total = 33.033 us, Queueing time: mean = 105.695 us, max = 105.695 us, min = 105.695 us, total = 105.695 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 54.650 us, total = 54.650 us, Queueing time: mean = 15.901 us, max = 15.901 us, min = 15.901 us, total = 15.901 us
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.530 us, total = 9.530 us, Queueing time: mean = 5.988 us, max = 5.988 us, min = 5.988 us, total = 5.988 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 46.994 us, total = 46.994 us, Queueing time: mean = 94.708 us, max = 94.708 us, min = 94.708 us, total = 94.708 us
GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 492.808 us, total = 492.808 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 920.233 us, total = 920.233 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.717 us, total = 15.717 us, Queueing time: mean = 740.000 ns, max = 740.000 ns, min = 740.000 ns, total = 740.000 ns
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 130.738 us, total = 130.738 us, Queueing time: mean = 148.481 us, max = 148.481 us, min = 148.481 us, total = 148.481 us
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 23.201 us, total = 23.201 us, Queueing time: mean = 89.441 us, max = 89.441 us, min = 89.441 us, total = 89.441 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 395.533 us, total = 395.533 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 431.027 us, total = 431.027 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 975.673 us, total = 975.673 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 513.632 us, total = 513.632 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 19.693 us, total = 19.693 us, Queueing time: mean = 10.630 us, max = 10.630 us, min = 10.630 us, total = 10.630 us
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 209.342 us, total = 209.342 us, Queueing time: mean = 134.661 us, max = 134.661 us, min = 134.661 us, total = 134.661 us
[2025-01-21 05:59:18,982 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 2692 total (1 active)
Queueing time: mean = 49.443 us, max = 498.514 us, min = 2.501 us, total = 133.102 ms
Execution time: mean = 250.657 us, total = 674.768 ms
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1218 total (0 active), Execution time: mean = 24.867 us, total = 30.288 ms, Queueing time: mean = 106.284 us, max = 498.514 us, min = 6.180 us, total = 129.454 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 1218 total (0 active), Execution time: mean = 523.411 us, total = 637.515 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 28.218 us, total = 6.829 ms, Queueing time: mean = 11.157 us, max = 389.975 us, min = 2.501 us, total = 2.700 ms
GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 9.916 us, total = 128.907 us, Queueing time: mean = 71.929 us, max = 155.412 us, min = 40.036 us, total = 935.078 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us
[2025-01-21 05:59:18,982 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 394 total (25 active)
Queueing time: mean = 33.171 us, max = 2.957 ms, min = 3.295 us, total = 13.069 ms
Execution time: mean = 136.099 ms, total = 53.623 s
Event stats:
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 24.367 us, total = 5.897 ms, Queueing time: mean = 21.342 us, max = 2.957 ms, min = 3.295 us, total = 5.165 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 37 total (24 active), Execution time: mean = 1.448 s, total = 53.588 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 37 total (0 active), Execution time: mean = 28.725 us, total = 1.063 ms, Queueing time: mean = 89.387 us, max = 239.158 us, min = 15.207 us, total = 3.307 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 41.727 us, total = 1.043 ms, Queueing time: mean = 71.718 us, max = 203.138 us, min = 5.341 us, total = 1.793 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 712.078 us, total = 17.802 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 13 total (0 active), Execution time: mean = 112.098 us, total = 1.457 ms, Queueing time: mean = 100.760 us, max = 175.467 us, min = 19.350 us, total = 1.310 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 13 total (0 active), Execution time: mean = 590.116 us, total = 7.672 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 05:59:18,983 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 248 total (0 active)
Queueing time: mean = 14.569 us, max = 1.158 ms, min = 692.000 ns, total = 3.613 ms
Execution time: mean = 27.721 us, total = 6.875 ms
Event stats:
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 27.609 us, total = 6.681 ms, Queueing time: mean = 14.627 us, max = 1.158 ms, min = 5.068 us, total = 3.540 ms
- 2 total (0 active), Execution time: mean = 77.838 us, total = 155.677 us, Queueing time: mean = 22.479 us, max = 27.307 us, min = 17.651 us, total = 44.958 us
RaySyncer.BroadcastMessage - 2 total (0 active), Execution time: mean = 18.387 us, total = 36.773 us, Queueing time: mean = 851.500 ns, max = 1.011 us, min = 692.000 ns, total = 1.703 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us
[2025-01-21 06:00:18,983 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 24
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 26
- cumulative published bytes: 22194
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 3
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 2
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 1
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 06:00:18,984 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 1836 total (4 active)
Queueing time: mean = 1.089 ms, max = 209.811 ms, min = 740.000 ns, total = 1.999 s
Execution time: mean = 343.881 us, total = 631.365 ms
Event stats:
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 22.394 us, total = 10.794 ms, Queueing time: mean = 21.955 us, max = 6.396 ms, min = 2.670 us, total = 10.582 ms
GcsInMemoryStore.Put - 239 total (0 active), Execution time: mean = 886.425 us, total = 211.856 ms, Queueing time: mean = 6.168 ms, max = 209.465 ms, min = 1.325 us, total = 1.474 s
RayletLoadPulled - 120 total (1 active), Execution time: mean = 236.316 us, total = 28.358 ms, Queueing time: mean = 61.931 us, max = 183.264 us, min = 16.365 us, total = 7.432 ms
NodeManagerService.grpc_client.GetResourceLoad - 117 total (0 active), Execution time: mean = 1.403 ms, total = 164.165 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 117 total (0 active), Execution time: mean = 69.642 us, total = 8.148 ms, Queueing time: mean = 89.966 us, max = 178.023 us, min = 19.180 us, total = 10.526 ms
GcsInMemoryStore.Get - 84 total (0 active), Execution time: mean = 17.022 us, total = 1.430 ms, Queueing time: mean = 20.124 us, max = 158.434 us, min = 2.067 us, total = 1.690 ms
InternalKVGcsService.grpc_server.InternalKVGet - 83 total (0 active), Execution time: mean = 767.781 us, total = 63.726 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 83 total (0 active), Execution time: mean = 35.450 us, total = 2.942 ms, Queueing time: mean = 377.364 us, max = 8.337 ms, min = 11.662 us, total = 31.321 ms
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 63 total (0 active), Execution time: mean = 31.098 us, total = 1.959 ms, Queueing time: mean = 67.742 us, max = 203.571 us, min = 8.800 us, total = 4.268 ms
InternalKVGcsService.grpc_server.InternalKVPut - 63 total (0 active), Execution time: mean = 441.874 us, total = 27.838 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 48 total (0 active), Execution time: mean = 28.084 us, total = 1.348 ms, Queueing time: mean = 93.026 us, max = 207.471 us, min = 15.170 us, total = 4.465 ms
NodeInfoGcsService.grpc_server.CheckAlive - 48 total (0 active), Execution time: mean = 487.566 us, total = 23.403 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 40 total (1 active), Execution time: mean = 9.161 us, total = 366.445 us, Queueing time: mean = 105.860 us, max = 1.634 ms, min = 26.287 us, total = 4.234 ms
HealthCheck - 38 total (0 active), Execution time: mean = 7.210 us, total = 273.970 us, Queueing time: mean = 96.160 us, max = 204.653 us, min = 20.297 us, total = 3.654 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 938.702 us, total = 31.916 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 35.401 us, total = 1.204 ms, Queueing time: mean = 590.521 us, max = 8.472 ms, min = 11.517 us, total = 20.078 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 24 total (0 active), Execution time: mean = 76.622 us, total = 1.839 ms, Queueing time: mean = 87.892 us, max = 201.135 us, min = 20.386 us, total = 2.109 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 24 total (0 active), Execution time: mean = 551.027 us, total = 13.225 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 26.838 us, total = 563.603 us, Queueing time: mean = 126.532 us, max = 257.619 us, min = 30.842 us, total = 2.657 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 491.931 us, total = 10.331 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GCSServer.deadline_timer.debug_state_dump - 12 total (1 active), Execution time: mean = 1.208 ms, total = 14.500 ms, Queueing time: mean = 44.701 us, max = 119.431 us, min = 16.811 us, total = 536.407 us
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.268 us, total = 49.609 us, Queueing time: mean = 32.229 us, max = 39.816 us, min = 10.594 us, total = 193.373 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 199.609 us, total = 998.045 us, Queueing time: mean = 58.098 us, max = 128.737 us, min = 15.591 us, total = 290.490 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 685.197 us, total = 3.426 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 117.076 us, total = 468.306 us, Queueing time: mean = 104.964 ms, max = 209.811 ms, min = 27.036 us, total = 419.855 ms
GcsResourceManager::Update - 2 total (0 active), Execution time: mean = 82.721 us, total = 165.442 us, Queueing time: mean = 106.719 us, max = 117.209 us, min = 96.230 us, total = 213.439 us
GCSServer.deadline_timer.debug_state_event_stats_print - 2 total (1 active, 1 running), Execution time: mean = 652.030 us, total = 1.304 ms, Queueing time: mean = 38.792 us, max = 77.585 us, min = 77.585 us, total = 77.585 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 17.533 us, total = 17.533 us, Queueing time: mean = 94.945 us, max = 94.945 us, min = 94.945 us, total = 94.945 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 513.632 us, total = 513.632 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 33.033 us, total = 33.033 us, Queueing time: mean = 105.695 us, max = 105.695 us, min = 105.695 us, total = 105.695 us
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.717 us, total = 15.717 us, Queueing time: mean = 740.000 ns, max = 740.000 ns, min = 740.000 ns, total = 740.000 ns
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 54.650 us, total = 54.650 us, Queueing time: mean = 15.901 us, max = 15.901 us, min = 15.901 us, total = 15.901 us
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.530 us, total = 9.530 us, Queueing time: mean = 5.988 us, max = 5.988 us, min = 5.988 us, total = 5.988 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 130.738 us, total = 130.738 us, Queueing time: mean = 148.481 us, max = 148.481 us, min = 148.481 us, total = 148.481 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 46.994 us, total = 46.994 us, Queueing time: mean = 94.708 us, max = 94.708 us, min = 94.708 us, total = 94.708 us
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 492.808 us, total = 492.808 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 920.233 us, total = 920.233 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 479.251 us, total = 479.251 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 209.342 us, total = 209.342 us, Queueing time: mean = 134.661 us, max = 134.661 us, min = 134.661 us, total = 134.661 us
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 23.201 us, total = 23.201 us, Queueing time: mean = 89.441 us, max = 89.441 us, min = 89.441 us, total = 89.441 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 395.533 us, total = 395.533 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 19.693 us, total = 19.693 us, Queueing time: mean = 10.630 us, max = 10.630 us, min = 10.630 us, total = 10.630 us
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 431.027 us, total = 431.027 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 975.673 us, total = 975.673 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 06:00:18,984 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 5464 total (1 active)
Queueing time: mean = 42.465 us, max = 606.639 us, min = 2.131 us, total = 232.028 ms
Execution time: mean = 219.054 us, total = 1.197 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 2478 total (0 active), Execution time: mean = 21.459 us, total = 53.175 ms, Queueing time: mean = 90.955 us, max = 498.514 us, min = 5.172 us, total = 225.386 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 2478 total (0 active), Execution time: mean = 456.554 us, total = 1.131 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 25.227 us, total = 12.159 ms, Queueing time: mean = 9.330 us, max = 389.975 us, min = 2.131 us, total = 4.497 ms
GcsTaskManager.GcJobSummary - 25 total (1 active), Execution time: mean = 9.176 us, total = 229.403 us, Queueing time: mean = 85.311 us, max = 606.639 us, min = 32.578 us, total = 2.133 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us
[2025-01-21 06:00:18,984 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 686 total (25 active)
Queueing time: mean = 23.604 us, max = 2.957 ms, min = 3.295 us, total = 16.192 ms
Execution time: mean = 165.942 ms, total = 113.836 s
Event stats:
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 23.776 us, total = 11.460 ms, Queueing time: mean = 14.910 us, max = 2.957 ms, min = 3.295 us, total = 7.186 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 50 total (24 active), Execution time: mean = 2.276 s, total = 113.789 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 50 total (0 active), Execution time: mean = 29.893 us, total = 1.495 ms, Queueing time: mean = 77.151 us, max = 239.158 us, min = 13.433 us, total = 3.858 ms
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 26 total (0 active), Execution time: mean = 94.267 us, total = 2.451 ms, Queueing time: mean = 71.579 us, max = 175.467 us, min = 17.549 us, total = 1.861 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 26 total (0 active), Execution time: mean = 485.686 us, total = 12.628 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 41.727 us, total = 1.043 ms, Queueing time: mean = 71.718 us, max = 203.138 us, min = 5.341 us, total = 1.793 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 712.078 us, total = 17.802 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 06:00:18,984 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 488 total (0 active)
Queueing time: mean = 11.901 us, max = 1.158 ms, min = 692.000 ns, total = 5.808 ms
Execution time: mean = 26.169 us, total = 12.771 ms
Event stats:
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 26.093 us, total = 12.577 ms, Queueing time: mean = 11.898 us, max = 1.158 ms, min = 4.729 us, total = 5.735 ms
- 2 total (0 active), Execution time: mean = 77.838 us, total = 155.677 us, Queueing time: mean = 22.479 us, max = 27.307 us, min = 17.651 us, total = 44.958 us
RaySyncer.BroadcastMessage - 2 total (0 active), Execution time: mean = 18.387 us, total = 36.773 us, Queueing time: mean = 851.500 ns, max = 1.011 us, min = 692.000 ns, total = 1.703 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us
[2025-01-21 06:01:18,985 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 36
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 39
- cumulative published bytes: 32043
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 3
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 2
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 1
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 06:01:18,985 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 2555 total (4 active)
Queueing time: mean = 794.373 us, max = 209.811 ms, min = 740.000 ns, total = 2.030 s
Execution time: mean = 317.087 us, total = 810.158 ms
Event stats:
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 23.405 us, total = 16.898 ms, Queueing time: mean = 17.921 us, max = 6.396 ms, min = 2.670 us, total = 12.939 ms
GcsInMemoryStore.Put - 335 total (0 active), Execution time: mean = 634.154 us, total = 212.442 ms, Queueing time: mean = 4.413 ms, max = 209.465 ms, min = 1.325 us, total = 1.478 s
RayletLoadPulled - 180 total (1 active), Execution time: mean = 245.093 us, total = 44.117 ms, Queueing time: mean = 68.265 us, max = 183.264 us, min = 16.365 us, total = 12.288 ms
NodeManagerService.grpc_client.GetResourceLoad - 177 total (0 active), Execution time: mean = 1.462 ms, total = 258.807 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 177 total (0 active), Execution time: mean = 72.124 us, total = 12.766 ms, Queueing time: mean = 99.180 us, max = 183.295 us, min = 19.180 us, total = 17.555 ms
GcsInMemoryStore.Get - 96 total (0 active), Execution time: mean = 18.975 us, total = 1.822 ms, Queueing time: mean = 19.112 us, max = 158.434 us, min = 2.067 us, total = 1.835 ms
InternalKVGcsService.grpc_server.InternalKVGet - 95 total (0 active), Execution time: mean = 746.310 us, total = 70.899 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 95 total (0 active), Execution time: mean = 37.327 us, total = 3.546 ms, Queueing time: mean = 341.891 us, max = 8.337 ms, min = 11.662 us, total = 32.480 ms
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 87 total (0 active), Execution time: mean = 34.112 us, total = 2.968 ms, Queueing time: mean = 76.665 us, max = 203.571 us, min = 8.800 us, total = 6.670 ms
InternalKVGcsService.grpc_server.InternalKVPut - 87 total (0 active), Execution time: mean = 473.275 us, total = 41.175 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 72 total (0 active), Execution time: mean = 30.253 us, total = 2.178 ms, Queueing time: mean = 101.638 us, max = 207.471 us, min = 15.170 us, total = 7.318 ms
NodeInfoGcsService.grpc_server.CheckAlive - 72 total (0 active), Execution time: mean = 521.294 us, total = 37.533 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 60 total (1 active), Execution time: mean = 9.533 us, total = 571.975 us, Queueing time: mean = 97.336 us, max = 1.634 ms, min = 26.287 us, total = 5.840 ms
HealthCheck - 58 total (0 active), Execution time: mean = 7.674 us, total = 445.084 us, Queueing time: mean = 101.178 us, max = 204.653 us, min = 20.297 us, total = 5.868 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 36 total (0 active), Execution time: mean = 78.946 us, total = 2.842 ms, Queueing time: mean = 98.087 us, max = 201.135 us, min = 20.386 us, total = 3.531 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 36 total (0 active), Execution time: mean = 581.580 us, total = 20.937 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 938.702 us, total = 31.916 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 35.401 us, total = 1.204 ms, Queueing time: mean = 590.521 us, max = 8.472 ms, min = 11.517 us, total = 20.078 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 26.838 us, total = 563.603 us, Queueing time: mean = 126.532 us, max = 257.619 us, min = 30.842 us, total = 2.657 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 491.931 us, total = 10.331 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GCSServer.deadline_timer.debug_state_dump - 18 total (1 active), Execution time: mean = 1.291 ms, total = 23.234 ms, Queueing time: mean = 52.790 us, max = 119.431 us, min = 16.811 us, total = 950.218 us
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.268 us, total = 49.609 us, Queueing time: mean = 32.229 us, max = 39.816 us, min = 10.594 us, total = 193.373 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 199.609 us, total = 998.045 us, Queueing time: mean = 58.098 us, max = 128.737 us, min = 15.591 us, total = 290.490 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 685.197 us, total = 3.426 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 117.076 us, total = 468.306 us, Queueing time: mean = 104.964 ms, max = 209.811 ms, min = 27.036 us, total = 419.855 ms
GCSServer.deadline_timer.debug_state_event_stats_print - 3 total (1 active, 1 running), Execution time: mean = 1.029 ms, total = 3.087 ms, Queueing time: mean = 49.251 us, max = 77.585 us, min = 70.167 us, total = 147.752 us
GcsResourceManager::Update - 2 total (0 active), Execution time: mean = 82.721 us, total = 165.442 us, Queueing time: mean = 106.719 us, max = 117.209 us, min = 96.230 us, total = 213.439 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 17.533 us, total = 17.533 us, Queueing time: mean = 94.945 us, max = 94.945 us, min = 94.945 us, total = 94.945 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 513.632 us, total = 513.632 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 33.033 us, total = 33.033 us, Queueing time: mean = 105.695 us, max = 105.695 us, min = 105.695 us, total = 105.695 us
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.717 us, total = 15.717 us, Queueing time: mean = 740.000 ns, max = 740.000 ns, min = 740.000 ns, total = 740.000 ns
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 54.650 us, total = 54.650 us, Queueing time: mean = 15.901 us, max = 15.901 us, min = 15.901 us, total = 15.901 us
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.530 us, total = 9.530 us, Queueing time: mean = 5.988 us, max = 5.988 us, min = 5.988 us, total = 5.988 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 130.738 us, total = 130.738 us, Queueing time: mean = 148.481 us, max = 148.481 us, min = 148.481 us, total = 148.481 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 46.994 us, total = 46.994 us, Queueing time: mean = 94.708 us, max = 94.708 us, min = 94.708 us, total = 94.708 us
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 492.808 us, total = 492.808 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 920.233 us, total = 920.233 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 479.251 us, total = 479.251 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 209.342 us, total = 209.342 us, Queueing time: mean = 134.661 us, max = 134.661 us, min = 134.661 us, total = 134.661 us
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 23.201 us, total = 23.201 us, Queueing time: mean = 89.441 us, max = 89.441 us, min = 89.441 us, total = 89.441 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 395.533 us, total = 395.533 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 19.693 us, total = 19.693 us, Queueing time: mean = 10.630 us, max = 10.630 us, min = 10.630 us, total = 10.630 us
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 431.027 us, total = 431.027 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 975.673 us, total = 975.673 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 06:01:18,986 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 8236 total (1 active)
Queueing time: mean = 44.822 us, max = 1.349 ms, min = 2.131 us, total = 369.154 ms
Execution time: mean = 230.525 us, total = 1.899 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 3738 total (0 active), Execution time: mean = 22.212 us, total = 83.029 ms, Queueing time: mean = 96.123 us, max = 1.349 ms, min = 5.172 us, total = 359.306 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 3738 total (0 active), Execution time: mean = 480.757 us, total = 1.797 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.119 us, total = 18.136 ms, Queueing time: mean = 9.202 us, max = 389.975 us, min = 2.131 us, total = 6.644 ms
GcsTaskManager.GcJobSummary - 37 total (1 active), Execution time: mean = 9.707 us, total = 359.160 us, Queueing time: mean = 86.250 us, max = 606.639 us, min = 32.578 us, total = 3.191 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us
[2025-01-21 06:01:18,986 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 978 total (25 active)
Queueing time: mean = 21.875 us, max = 2.957 ms, min = 3.295 us, total = 21.394 ms
Execution time: mean = 177.887 ms, total = 173.973 s
Event stats:
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.132 us, total = 18.145 ms, Queueing time: mean = 13.500 us, max = 2.957 ms, min = 3.295 us, total = 9.747 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 63 total (24 active), Execution time: mean = 2.760 s, total = 173.909 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 63 total (0 active), Execution time: mean = 34.135 us, total = 2.150 ms, Queueing time: mean = 81.435 us, max = 239.158 us, min = 13.433 us, total = 5.130 ms
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 39 total (0 active), Execution time: mean = 104.532 us, total = 4.077 ms, Queueing time: mean = 82.781 us, max = 175.467 us, min = 17.549 us, total = 3.228 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 39 total (0 active), Execution time: mean = 540.745 us, total = 21.089 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 41.727 us, total = 1.043 ms, Queueing time: mean = 71.718 us, max = 203.138 us, min = 5.341 us, total = 1.793 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 712.078 us, total = 17.802 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 06:01:18,986 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 728 total (0 active)
Queueing time: mean = 11.416 us, max = 1.158 ms, min = 692.000 ns, total = 8.311 ms
Execution time: mean = 26.389 us, total = 19.211 ms
Event stats:
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 26.340 us, total = 19.018 ms, Queueing time: mean = 11.409 us, max = 1.158 ms, min = 4.729 us, total = 8.238 ms
- 2 total (0 active), Execution time: mean = 77.838 us, total = 155.677 us, Queueing time: mean = 22.479 us, max = 27.307 us, min = 17.651 us, total = 44.958 us
RaySyncer.BroadcastMessage - 2 total (0 active), Execution time: mean = 18.387 us, total = 36.773 us, Queueing time: mean = 851.500 ns, max = 1.011 us, min = 692.000 ns, total = 1.703 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us
[2025-01-21 06:02:18,986 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 48
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 51
- cumulative published bytes: 41859
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 3
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 2
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 1
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 06:02:18,987 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 3274 total (4 active)
Queueing time: mean = 626.158 us, max = 209.811 ms, min = 740.000 ns, total = 2.050 s
Execution time: mean = 288.581 us, total = 944.813 ms
Event stats:
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 22.636 us, total = 21.776 ms, Queueing time: mean = 15.383 us, max = 6.396 ms, min = 1.699 us, total = 14.798 ms
GcsInMemoryStore.Put - 431 total (0 active), Execution time: mean = 493.935 us, total = 212.886 ms, Queueing time: mean = 3.441 ms, max = 209.465 ms, min = 1.325 us, total = 1.483 s
RayletLoadPulled - 240 total (1 active), Execution time: mean = 236.422 us, total = 56.741 ms, Queueing time: mean = 64.538 us, max = 183.264 us, min = 10.641 us, total = 15.489 ms
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 237 total (0 active), Execution time: mean = 70.073 us, total = 16.607 ms, Queueing time: mean = 88.671 us, max = 183.295 us, min = 15.696 us, total = 21.015 ms
NodeManagerService.grpc_client.GetResourceLoad - 237 total (0 active), Execution time: mean = 1.386 ms, total = 328.371 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 111 total (0 active), Execution time: mean = 32.947 us, total = 3.657 ms, Queueing time: mean = 72.641 us, max = 203.571 us, min = 8.800 us, total = 8.063 ms
InternalKVGcsService.grpc_server.InternalKVPut - 111 total (0 active), Execution time: mean = 455.889 us, total = 50.604 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Get - 108 total (0 active), Execution time: mean = 18.864 us, total = 2.037 ms, Queueing time: mean = 18.132 us, max = 158.434 us, min = 2.067 us, total = 1.958 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 107 total (0 active), Execution time: mean = 38.621 us, total = 4.132 ms, Queueing time: mean = 310.524 us, max = 8.337 ms, min = 11.662 us, total = 33.226 ms
InternalKVGcsService.grpc_server.InternalKVGet - 107 total (0 active), Execution time: mean = 711.148 us, total = 76.093 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 96 total (0 active), Execution time: mean = 28.675 us, total = 2.753 ms, Queueing time: mean = 92.781 us, max = 207.471 us, min = 15.170 us, total = 8.907 ms
NodeInfoGcsService.grpc_server.CheckAlive - 96 total (0 active), Execution time: mean = 494.699 us, total = 47.491 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 80 total (1 active), Execution time: mean = 9.078 us, total = 726.257 us, Queueing time: mean = 84.793 us, max = 1.634 ms, min = 13.790 us, total = 6.783 ms
HealthCheck - 78 total (0 active), Execution time: mean = 7.426 us, total = 579.227 us, Queueing time: mean = 90.262 us, max = 204.653 us, min = 16.570 us, total = 7.040 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 48 total (0 active), Execution time: mean = 75.505 us, total = 3.624 ms, Queueing time: mean = 92.063 us, max = 201.135 us, min = 19.197 us, total = 4.419 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 48 total (0 active), Execution time: mean = 550.398 us, total = 26.419 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 35.401 us, total = 1.204 ms, Queueing time: mean = 590.521 us, max = 8.472 ms, min = 11.517 us, total = 20.078 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 938.702 us, total = 31.916 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GCSServer.deadline_timer.debug_state_dump - 24 total (1 active), Execution time: mean = 1.325 ms, total = 31.793 ms, Queueing time: mean = 51.273 us, max = 119.431 us, min = 16.811 us, total = 1.231 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 26.838 us, total = 563.603 us, Queueing time: mean = 126.532 us, max = 257.619 us, min = 30.842 us, total = 2.657 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 491.931 us, total = 10.331 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.268 us, total = 49.609 us, Queueing time: mean = 32.229 us, max = 39.816 us, min = 10.594 us, total = 193.373 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 199.609 us, total = 998.045 us, Queueing time: mean = 58.098 us, max = 128.737 us, min = 15.591 us, total = 290.490 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 685.197 us, total = 3.426 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 117.076 us, total = 468.306 us, Queueing time: mean = 104.964 ms, max = 209.811 ms, min = 27.036 us, total = 419.855 ms
GCSServer.deadline_timer.debug_state_event_stats_print - 4 total (1 active, 1 running), Execution time: mean = 1.158 ms, total = 4.632 ms, Queueing time: mean = 53.479 us, max = 77.585 us, min = 66.164 us, total = 213.916 us
GcsResourceManager::Update - 2 total (0 active), Execution time: mean = 82.721 us, total = 165.442 us, Queueing time: mean = 106.719 us, max = 117.209 us, min = 96.230 us, total = 213.439 us
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.530 us, total = 9.530 us, Queueing time: mean = 5.988 us, max = 5.988 us, min = 5.988 us, total = 5.988 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 54.650 us, total = 54.650 us, Queueing time: mean = 15.901 us, max = 15.901 us, min = 15.901 us, total = 15.901 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 46.994 us, total = 46.994 us, Queueing time: mean = 94.708 us, max = 94.708 us, min = 94.708 us, total = 94.708 us
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 33.033 us, total = 33.033 us, Queueing time: mean = 105.695 us, max = 105.695 us, min = 105.695 us, total = 105.695 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 17.533 us, total = 17.533 us, Queueing time: mean = 94.945 us, max = 94.945 us, min = 94.945 us, total = 94.945 us
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 492.808 us, total = 492.808 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 920.233 us, total = 920.233 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 23.201 us, total = 23.201 us, Queueing time: mean = 89.441 us, max = 89.441 us, min = 89.441 us, total = 89.441 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 395.533 us, total = 395.533 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 431.027 us, total = 431.027 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 975.673 us, total = 975.673 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 513.632 us, total = 513.632 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 19.693 us, total = 19.693 us, Queueing time: mean = 10.630 us, max = 10.630 us, min = 10.630 us, total = 10.630 us
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 209.342 us, total = 209.342 us, Queueing time: mean = 134.661 us, max = 134.661 us, min = 134.661 us, total = 134.661 us
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 479.251 us, total = 479.251 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 130.738 us, total = 130.738 us, Queueing time: mean = 148.481 us, max = 148.481 us, min = 148.481 us, total = 148.481 us
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.717 us, total = 15.717 us, Queueing time: mean = 740.000 ns, max = 740.000 ns, min = 740.000 ns, total = 740.000 ns
[2025-01-21 06:02:18,987 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 11006 total (1 active)
Queueing time: mean = 41.435 us, max = 1.349 ms, min = 2.131 us, total = 456.032 ms
Execution time: mean = 218.563 us, total = 2.406 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 4997 total (0 active), Execution time: mean = 21.462 us, total = 107.245 ms, Queueing time: mean = 88.807 us, max = 1.349 ms, min = 3.632 us, total = 443.771 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 4997 total (0 active), Execution time: mean = 455.157 us, total = 2.274 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 24.292 us, total = 23.369 ms, Queueing time: mean = 8.789 us, max = 389.975 us, min = 2.131 us, total = 8.455 ms
GcsTaskManager.GcJobSummary - 49 total (1 active), Execution time: mean = 9.453 us, total = 463.209 us, Queueing time: mean = 77.432 us, max = 606.639 us, min = 29.279 us, total = 3.794 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us
[2025-01-21 06:02:18,987 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 1266 total (25 active)
Queueing time: mean = 20.072 us, max = 2.957 ms, min = 2.399 us, total = 25.411 ms
Execution time: mean = 184.984 ms, total = 234.190 s
Event stats:
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 24.642 us, total = 23.706 ms, Queueing time: mean = 12.306 us, max = 2.957 ms, min = 2.399 us, total = 11.838 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 75 total (24 active), Execution time: mean = 3.121 s, total = 234.112 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 75 total (0 active), Execution time: mean = 35.100 us, total = 2.632 ms, Queueing time: mean = 82.030 us, max = 239.158 us, min = 13.433 us, total = 6.152 ms
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 51 total (0 active), Execution time: mean = 107.728 us, total = 5.494 ms, Queueing time: mean = 81.036 us, max = 175.467 us, min = 17.549 us, total = 4.133 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 51 total (0 active), Execution time: mean = 545.307 us, total = 27.811 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 41.727 us, total = 1.043 ms, Queueing time: mean = 71.718 us, max = 203.138 us, min = 5.341 us, total = 1.793 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 712.078 us, total = 17.802 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 06:02:18,987 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 968 total (0 active)
Queueing time: mean = 11.091 us, max = 1.158 ms, min = 692.000 ns, total = 10.736 ms
Execution time: mean = 26.391 us, total = 25.546 ms
Event stats:
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 26.354 us, total = 25.353 ms, Queueing time: mean = 11.084 us, max = 1.158 ms, min = 4.425 us, total = 10.662 ms
- 2 total (0 active), Execution time: mean = 77.838 us, total = 155.677 us, Queueing time: mean = 22.479 us, max = 27.307 us, min = 17.651 us, total = 44.958 us
RaySyncer.BroadcastMessage - 2 total (0 active), Execution time: mean = 18.387 us, total = 36.773 us, Queueing time: mean = 851.500 ns, max = 1.011 us, min = 692.000 ns, total = 1.703 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us
|