File size: 175,425 Bytes
c011401 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 |
[2025-01-21 05:47:33,311 I 18551 18551] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-21 05:47:33,311 I 18551 18551] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-21 05:47:33,318 I 18551 18551] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-21 05:47:33,325 I 18551 18551] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-21 05:47:33,326 I 18551 18551] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: cf4503ed329bc2b5612e9c435582c6da51e2e7d5bac7b8183e0cfd01
[2025-01-21 05:47:33,327 I 18551 18551] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-21 05:47:33,330 I 18551 18551] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 56605.
[2025-01-21 05:47:33,581 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:47:33,581 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 25 total (5 active)
Queueing time: mean = 91.127 ms, max = 251.846 ms, min = 2.247 us, total = 2.278 s
Execution time: mean = 10.194 ms, total = 254.838 ms
Event stats:
GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 27.986 ms, total = 251.870 ms, Queueing time: mean = 195.243 ms, max = 251.330 ms, min = 2.247 us, total = 1.757 s
GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 9.914 us, total = 49.568 us, Queueing time: mean = 74.010 us, max = 93.369 us, min = 64.515 us, total = 370.049 us
PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 5.246 us, total = 20.983 us, Queueing time: mean = 125.891 ms, max = 251.846 ms, min = 251.719 ms, total = 503.565 ms
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 12.339 us, total = 24.678 us, Queueing time: mean = 7.565 ms, max = 14.800 ms, min = 328.606 us, total = 15.129 ms
RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 2.857 ms, total = 2.857 ms, Queueing time: mean = 1.926 ms, max = 1.926 ms, min = 1.926 ms, total = 1.926 ms
ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 15.587 us, total = 15.587 us, Queueing time: mean = 3.595 us, max = 3.595 us, min = 3.595 us, total = 3.595 us
[2025-01-21 05:47:33,582 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 1.549 ms, max = 7.604 ms, min = 9.340 us, total = 7.746 ms
Execution time: mean = 52.263 us, total = 261.314 us
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 83.150 us, total = 249.449 us, Queueing time: mean = 2.551 ms, max = 7.604 ms, min = 9.340 us, total = 7.652 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 05:47:33,582 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 2.042 ms, max = 8.828 ms, min = 9.247 us, total = 10.210 ms
Execution time: mean = 34.846 us, total = 174.231 us
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 43.227 us, total = 129.680 us, Queueing time: mean = 2.974 ms, max = 8.828 ms, min = 9.247 us, total = 8.921 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 05:47:33,582 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 5 total (0 active)
Queueing time: mean = 229.554 us, max = 779.592 us, min = 11.557 us, total = 1.148 ms
Execution time: mean = 259.601 us, total = 1.298 ms
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 432.013 us, total = 1.296 ms, Queueing time: mean = 319.245 us, max = 779.592 us, min = 11.557 us, total = 957.736 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:47:36,738 I 18551 18551] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:47:36,738 I 18551 18551] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:47:36,738 I 18551 18551] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d registered, will try to reschedule all the infeasible placement groups.
[2025-01-21 05:47:36,744 I 18551 18638] (gcs_server) ray_syncer.cc:377: Get connection node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:47:37,747 I 18551 18551] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 18344
[2025-01-21 05:47:37,747 I 18551 18551] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 18344
[2025-01-21 05:47:43,332 W 18551 18574] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:64843: Failed to connect to remote host: Connection refused; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster.
[2025-01-21 05:48:33,582 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 12
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 13
- cumulative published bytes: 12684
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 5
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 2
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 1
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:48:33,582 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 1113 total (4 active)
Queueing time: mean = 2.096 ms, max = 251.846 ms, min = 853.000 ns, total = 2.333 s
Execution time: mean = 529.727 us, total = 589.586 ms
Event stats:
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 22.461 us, total = 5.436 ms, Queueing time: mean = 71.164 us, max = 14.800 ms, min = 2.721 us, total = 17.222 ms
GcsInMemoryStore.Put - 143 total (0 active), Execution time: mean = 1.776 ms, total = 253.981 ms, Queueing time: mean = 12.318 ms, max = 251.330 ms, min = 2.247 us, total = 1.762 s
GcsInMemoryStore.Get - 72 total (0 active), Execution time: mean = 23.769 us, total = 1.711 ms, Queueing time: mean = 20.905 us, max = 337.749 us, min = 3.595 us, total = 1.505 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 71 total (0 active), Execution time: mean = 46.636 us, total = 3.311 ms, Queueing time: mean = 103.644 us, max = 240.717 us, min = 15.205 us, total = 7.359 ms
InternalKVGcsService.grpc_server.InternalKVGet - 71 total (0 active), Execution time: mean = 677.549 us, total = 48.106 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
RayletLoadPulled - 60 total (1 active), Execution time: mean = 242.463 us, total = 14.548 ms, Queueing time: mean = 73.646 us, max = 176.027 us, min = 16.795 us, total = 4.419 ms
NodeManagerService.grpc_client.GetResourceLoad - 56 total (0 active), Execution time: mean = 2.205 ms, total = 123.453 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 56 total (0 active), Execution time: mean = 78.094 us, total = 4.373 ms, Queueing time: mean = 150.019 us, max = 2.124 ms, min = 16.199 us, total = 8.401 ms
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 39 total (0 active), Execution time: mean = 40.728 us, total = 1.588 ms, Queueing time: mean = 112.935 us, max = 254.636 us, min = 31.997 us, total = 4.404 ms
InternalKVGcsService.grpc_server.InternalKVPut - 39 total (0 active), Execution time: mean = 626.221 us, total = 24.423 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 35.048 us, total = 841.162 us, Queueing time: mean = 108.178 us, max = 186.281 us, min = 30.232 us, total = 2.596 ms
NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 597.630 us, total = 14.343 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 21 total (1 active), Execution time: mean = 12.154 us, total = 255.234 us, Queueing time: mean = 78.771 us, max = 151.718 us, min = 40.254 us, total = 1.654 ms
HealthCheck - 18 total (0 active), Execution time: mean = 8.633 us, total = 155.395 us, Queueing time: mean = 105.561 us, max = 133.874 us, min = 29.147 us, total = 1.900 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 734.030 us, total = 8.808 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 95.296 us, total = 1.144 ms, Queueing time: mean = 132.082 us, max = 190.945 us, min = 32.307 us, total = 1.585 ms
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.298 ms, total = 7.791 ms, Queueing time: mean = 68.888 us, max = 137.944 us, min = 11.375 us, total = 413.330 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
GcsResourceManager::Update - 3 total (0 active), Execution time: mean = 104.086 us, total = 312.259 us, Queueing time: mean = 122.171 us, max = 138.303 us, min = 101.321 us, total = 366.512 us
GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 05:48:33,583 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 2608 total (1 active)
Queueing time: mean = 56.913 us, max = 7.604 ms, min = 3.143 us, total = 148.430 ms
Execution time: mean = 265.080 us, total = 691.328 ms
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1176 total (0 active), Execution time: mean = 28.731 us, total = 33.788 ms, Queueing time: mean = 117.164 us, max = 375.594 us, min = 5.071 us, total = 137.785 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 1176 total (0 active), Execution time: mean = 553.657 us, total = 651.100 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 25.972 us, total = 6.285 ms, Queueing time: mean = 40.153 us, max = 7.604 ms, min = 3.143 us, total = 9.717 ms
GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 11.023 us, total = 143.298 us, Queueing time: mean = 64.094 us, max = 86.697 us, min = 47.656 us, total = 833.225 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:48:33,583 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 394 total (25 active)
Queueing time: mean = 56.527 us, max = 8.828 ms, min = 2.456 us, total = 22.272 ms
Execution time: mean = 134.356 ms, total = 52.936 s
Event stats:
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 25.141 us, total = 6.084 ms, Queueing time: mean = 45.562 us, max = 8.828 ms, min = 2.456 us, total = 11.026 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 37 total (0 active), Execution time: mean = 30.784 us, total = 1.139 ms, Queueing time: mean = 138.328 us, max = 347.872 us, min = 5.878 us, total = 5.118 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 37 total (24 active), Execution time: mean = 1.430 s, total = 52.903 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 13 total (0 active), Execution time: mean = 109.377 us, total = 1.422 ms, Queueing time: mean = 130.402 us, max = 191.279 us, min = 44.178 us, total = 1.695 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 13 total (0 active), Execution time: mean = 657.092 us, total = 8.542 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:48:33,583 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 250 total (0 active)
Queueing time: mean = 15.984 us, max = 779.592 us, min = 621.000 ns, total = 3.996 ms
Execution time: mean = 34.306 us, total = 8.577 ms
Event stats:
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 33.609 us, total = 8.133 ms, Queueing time: mean = 14.269 us, max = 779.592 us, min = 4.534 us, total = 3.453 ms
RaySyncer.BroadcastMessage - 3 total (0 active), Execution time: mean = 32.858 us, total = 98.574 us, Queueing time: mean = 960.333 ns, max = 1.284 us, min = 621.000 ns, total = 2.881 us
- 3 total (0 active), Execution time: mean = 114.255 us, total = 342.764 us, Queueing time: mean = 116.620 us, max = 131.850 us, min = 102.511 us, total = 349.861 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:49:33,583 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 24
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 25
- cumulative published bytes: 22500
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 5
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 2
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 1
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:49:33,584 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 1832 total (4 active)
Queueing time: mean = 1.288 ms, max = 251.846 ms, min = 853.000 ns, total = 2.359 s
Execution time: mean = 408.039 us, total = 747.527 ms
Event stats:
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 21.475 us, total = 10.351 ms, Queueing time: mean = 39.622 us, max = 14.800 ms, min = 1.906 us, total = 19.098 ms
GcsInMemoryStore.Put - 239 total (0 active), Execution time: mean = 1.065 ms, total = 254.564 ms, Queueing time: mean = 7.389 ms, max = 251.330 ms, min = 2.247 us, total = 1.766 s
RayletLoadPulled - 120 total (1 active), Execution time: mean = 231.999 us, total = 27.840 ms, Queueing time: mean = 71.326 us, max = 194.192 us, min = 14.896 us, total = 8.559 ms
NodeManagerService.grpc_client.GetResourceLoad - 116 total (0 active), Execution time: mean = 1.783 ms, total = 206.826 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 116 total (0 active), Execution time: mean = 72.645 us, total = 8.427 ms, Queueing time: mean = 118.771 us, max = 2.124 ms, min = 13.688 us, total = 13.777 ms
GcsInMemoryStore.Get - 84 total (0 active), Execution time: mean = 23.472 us, total = 1.972 ms, Queueing time: mean = 19.463 us, max = 337.749 us, min = 3.595 us, total = 1.635 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 83 total (0 active), Execution time: mean = 47.107 us, total = 3.910 ms, Queueing time: mean = 103.671 us, max = 240.717 us, min = 13.972 us, total = 8.605 ms
InternalKVGcsService.grpc_server.InternalKVGet - 83 total (0 active), Execution time: mean = 658.563 us, total = 54.661 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 63 total (0 active), Execution time: mean = 38.782 us, total = 2.443 ms, Queueing time: mean = 108.795 us, max = 254.636 us, min = 14.068 us, total = 6.854 ms
InternalKVGcsService.grpc_server.InternalKVPut - 63 total (0 active), Execution time: mean = 573.490 us, total = 36.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 48 total (0 active), Execution time: mean = 32.126 us, total = 1.542 ms, Queueing time: mean = 105.330 us, max = 186.281 us, min = 22.398 us, total = 5.056 ms
NodeInfoGcsService.grpc_server.CheckAlive - 48 total (0 active), Execution time: mean = 558.891 us, total = 26.827 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 41 total (1 active), Execution time: mean = 10.981 us, total = 450.240 us, Queueing time: mean = 69.871 us, max = 151.718 us, min = 19.030 us, total = 2.865 ms
HealthCheck - 38 total (0 active), Execution time: mean = 7.884 us, total = 299.598 us, Queueing time: mean = 95.886 us, max = 174.442 us, min = 19.444 us, total = 3.644 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 24 total (0 active), Execution time: mean = 665.785 us, total = 15.979 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 24 total (0 active), Execution time: mean = 88.422 us, total = 2.122 ms, Queueing time: mean = 114.312 us, max = 190.945 us, min = 22.178 us, total = 2.743 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GCSServer.deadline_timer.debug_state_dump - 12 total (1 active), Execution time: mean = 1.406 ms, total = 16.875 ms, Queueing time: mean = 70.747 us, max = 137.944 us, min = 11.375 us, total = 848.965 us
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
GcsResourceManager::Update - 3 total (0 active), Execution time: mean = 104.086 us, total = 312.259 us, Queueing time: mean = 122.171 us, max = 138.303 us, min = 101.321 us, total = 366.512 us
GCSServer.deadline_timer.debug_state_event_stats_print - 2 total (1 active, 1 running), Execution time: mean = 496.070 us, total = 992.139 us, Queueing time: mean = 7.912 us, max = 15.823 us, min = 15.823 us, total = 15.823 us
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 05:49:33,584 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 5380 total (1 active)
Queueing time: mean = 52.259 us, max = 7.604 ms, min = 3.138 us, total = 281.152 ms
Execution time: mean = 249.239 us, total = 1.341 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 2436 total (0 active), Execution time: mean = 25.753 us, total = 62.735 ms, Queueing time: mean = 109.832 us, max = 401.821 us, min = 5.071 us, total = 267.551 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 2436 total (0 active), Execution time: mean = 519.562 us, total = 1.266 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 25.363 us, total = 12.225 ms, Queueing time: mean = 24.359 us, max = 7.604 ms, min = 3.138 us, total = 11.741 ms
GcsTaskManager.GcJobSummary - 25 total (1 active), Execution time: mean = 11.105 us, total = 277.632 us, Queueing time: mean = 70.569 us, max = 134.123 us, min = 29.930 us, total = 1.764 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:49:33,584 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 682 total (25 active)
Queueing time: mean = 39.918 us, max = 8.828 ms, min = 2.456 us, total = 27.224 ms
Execution time: mean = 165.780 ms, total = 113.062 s
Event stats:
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 25.439 us, total = 12.262 ms, Queueing time: mean = 27.648 us, max = 8.828 ms, min = 2.456 us, total = 13.326 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 49 total (0 active), Execution time: mean = 34.354 us, total = 1.683 ms, Queueing time: mean = 132.751 us, max = 347.872 us, min = 5.878 us, total = 6.505 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 49 total (24 active), Execution time: mean = 2.306 s, total = 113.013 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 25 total (0 active), Execution time: mean = 108.082 us, total = 2.702 ms, Queueing time: mean = 118.414 us, max = 191.279 us, min = 39.531 us, total = 2.960 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 25 total (0 active), Execution time: mean = 627.921 us, total = 15.698 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:49:33,585 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 490 total (0 active)
Queueing time: mean = 13.052 us, max = 779.592 us, min = 621.000 ns, total = 6.396 ms
Execution time: mean = 30.316 us, total = 14.855 ms
Event stats:
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 29.900 us, total = 14.412 ms, Queueing time: mean = 12.143 us, max = 779.592 us, min = 3.424 us, total = 5.853 ms
RaySyncer.BroadcastMessage - 3 total (0 active), Execution time: mean = 32.858 us, total = 98.574 us, Queueing time: mean = 960.333 ns, max = 1.284 us, min = 621.000 ns, total = 2.881 us
- 3 total (0 active), Execution time: mean = 114.255 us, total = 342.764 us, Queueing time: mean = 116.620 us, max = 131.850 us, min = 102.511 us, total = 349.861 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:50:33,585 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 36
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 38
- cumulative published bytes: 32718
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 8
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 3
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 2
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:50:33,586 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 2552 total (4 active)
Queueing time: mean = 935.142 us, max = 251.846 ms, min = 853.000 ns, total = 2.386 s
Execution time: mean = 361.272 us, total = 921.965 ms
Event stats:
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 21.590 us, total = 15.588 ms, Queueing time: mean = 29.264 us, max = 14.800 ms, min = 1.906 us, total = 21.129 ms
GcsInMemoryStore.Put - 335 total (0 active), Execution time: mean = 761.472 us, total = 255.093 ms, Queueing time: mean = 5.283 ms, max = 251.330 ms, min = 2.247 us, total = 1.770 s
RayletLoadPulled - 180 total (1 active), Execution time: mean = 241.374 us, total = 43.447 ms, Queueing time: mean = 70.843 us, max = 202.188 us, min = 14.896 us, total = 12.752 ms
NodeManagerService.grpc_client.GetResourceLoad - 176 total (0 active), Execution time: mean = 1.706 ms, total = 300.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 176 total (0 active), Execution time: mean = 73.586 us, total = 12.951 ms, Queueing time: mean = 115.323 us, max = 2.124 ms, min = 13.688 us, total = 20.297 ms
GcsInMemoryStore.Get - 96 total (0 active), Execution time: mean = 23.124 us, total = 2.220 ms, Queueing time: mean = 18.502 us, max = 337.749 us, min = 3.595 us, total = 1.776 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 95 total (0 active), Execution time: mean = 47.288 us, total = 4.492 ms, Queueing time: mean = 102.524 us, max = 240.717 us, min = 13.972 us, total = 9.740 ms
InternalKVGcsService.grpc_server.InternalKVGet - 95 total (0 active), Execution time: mean = 648.315 us, total = 61.590 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 87 total (0 active), Execution time: mean = 39.022 us, total = 3.395 ms, Queueing time: mean = 105.842 us, max = 254.636 us, min = 14.068 us, total = 9.208 ms
InternalKVGcsService.grpc_server.InternalKVPut - 87 total (0 active), Execution time: mean = 570.058 us, total = 49.595 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 72 total (0 active), Execution time: mean = 31.547 us, total = 2.271 ms, Queueing time: mean = 103.078 us, max = 190.114 us, min = 22.398 us, total = 7.422 ms
NodeInfoGcsService.grpc_server.CheckAlive - 72 total (0 active), Execution time: mean = 555.444 us, total = 39.992 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 61 total (1 active), Execution time: mean = 10.615 us, total = 647.532 us, Queueing time: mean = 66.860 us, max = 151.718 us, min = 19.030 us, total = 4.078 ms
HealthCheck - 58 total (0 active), Execution time: mean = 7.832 us, total = 454.259 us, Queueing time: mean = 91.104 us, max = 199.261 us, min = 18.886 us, total = 5.284 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 36 total (0 active), Execution time: mean = 666.409 us, total = 23.991 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 36 total (0 active), Execution time: mean = 84.915 us, total = 3.057 ms, Queueing time: mean = 114.260 us, max = 190.945 us, min = 22.178 us, total = 4.113 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GCSServer.deadline_timer.debug_state_dump - 18 total (1 active), Execution time: mean = 1.368 ms, total = 24.631 ms, Queueing time: mean = 63.672 us, max = 137.944 us, min = 11.375 us, total = 1.146 ms
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
GcsResourceManager::Update - 4 total (0 active), Execution time: mean = 124.977 us, total = 499.909 us, Queueing time: mean = 120.538 us, max = 138.303 us, min = 101.321 us, total = 482.154 us
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
GCSServer.deadline_timer.debug_state_event_stats_print - 3 total (1 active, 1 running), Execution time: mean = 926.751 us, total = 2.780 ms, Queueing time: mean = 35.602 us, max = 90.983 us, min = 15.823 us, total = 106.806 us
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 05:50:33,586 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 8152 total (1 active)
Queueing time: mean = 49.841 us, max = 7.604 ms, min = 2.691 us, total = 406.308 ms
Execution time: mean = 244.772 us, total = 1.995 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 3696 total (0 active), Execution time: mean = 24.991 us, total = 92.366 ms, Queueing time: mean = 105.370 us, max = 401.821 us, min = 5.071 us, total = 389.449 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 3696 total (0 active), Execution time: mean = 509.742 us, total = 1.884 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.751 us, total = 18.593 ms, Queueing time: mean = 19.325 us, max = 7.604 ms, min = 2.691 us, total = 13.953 ms
GcsTaskManager.GcJobSummary - 37 total (1 active), Execution time: mean = 10.975 us, total = 406.093 us, Queueing time: mean = 75.952 us, max = 143.193 us, min = 29.930 us, total = 2.810 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:50:33,586 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 974 total (25 active)
Queueing time: mean = 32.468 us, max = 8.828 ms, min = 2.456 us, total = 31.624 ms
Execution time: mean = 177.927 ms, total = 173.301 s
Event stats:
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.352 us, total = 18.304 ms, Queueing time: mean = 21.526 us, max = 8.828 ms, min = 2.456 us, total = 15.542 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 62 total (0 active), Execution time: mean = 35.252 us, total = 2.186 ms, Queueing time: mean = 121.970 us, max = 347.872 us, min = 5.878 us, total = 7.562 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 62 total (24 active), Execution time: mean = 2.794 s, total = 173.237 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 38 total (0 active), Execution time: mean = 101.689 us, total = 3.864 ms, Queueing time: mean = 107.569 us, max = 191.279 us, min = 23.555 us, total = 4.088 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 38 total (0 active), Execution time: mean = 610.504 us, total = 23.199 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:50:33,586 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 732 total (0 active)
Queueing time: mean = 11.809 us, max = 779.592 us, min = 456.000 ns, total = 8.644 ms
Execution time: mean = 28.539 us, total = 20.890 ms
Event stats:
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 28.262 us, total = 20.405 ms, Queueing time: mean = 11.188 us, max = 779.592 us, min = 3.198 us, total = 8.078 ms
RaySyncer.BroadcastMessage - 4 total (0 active), Execution time: mean = 27.779 us, total = 111.116 us, Queueing time: mean = 834.250 ns, max = 1.284 us, min = 456.000 ns, total = 3.337 us
- 4 total (0 active), Execution time: mean = 93.099 us, total = 372.394 us, Queueing time: mean = 93.332 us, max = 131.850 us, min = 23.468 us, total = 373.329 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:51:33,586 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 48
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 52
- cumulative published bytes: 42999
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 12
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 4
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 3
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:51:33,588 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 3273 total (4 active)
Queueing time: mean = 733.655 us, max = 251.846 ms, min = 853.000 ns, total = 2.401 s
Execution time: mean = 320.678 us, total = 1.050 s
Event stats:
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 20.938 us, total = 20.142 ms, Queueing time: mean = 23.807 us, max = 14.800 ms, min = 1.906 us, total = 22.902 ms
GcsInMemoryStore.Put - 431 total (0 active), Execution time: mean = 592.725 us, total = 255.465 ms, Queueing time: mean = 4.116 ms, max = 251.330 ms, min = 2.247 us, total = 1.774 s
RayletLoadPulled - 240 total (1 active), Execution time: mean = 236.425 us, total = 56.742 ms, Queueing time: mean = 62.763 us, max = 202.188 us, min = 14.896 us, total = 15.063 ms
NodeManagerService.grpc_client.GetResourceLoad - 236 total (0 active), Execution time: mean = 1.555 ms, total = 367.093 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 236 total (0 active), Execution time: mean = 70.426 us, total = 16.620 ms, Queueing time: mean = 94.973 us, max = 2.124 ms, min = 13.688 us, total = 22.414 ms
InternalKVGcsService.grpc_server.InternalKVPut - 111 total (0 active), Execution time: mean = 516.665 us, total = 57.350 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 111 total (0 active), Execution time: mean = 35.317 us, total = 3.920 ms, Queueing time: mean = 88.424 us, max = 254.636 us, min = 12.388 us, total = 9.815 ms
GcsInMemoryStore.Get - 108 total (0 active), Execution time: mean = 22.287 us, total = 2.407 ms, Queueing time: mean = 17.361 us, max = 337.749 us, min = 3.595 us, total = 1.875 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 107 total (0 active), Execution time: mean = 46.534 us, total = 4.979 ms, Queueing time: mean = 93.984 us, max = 240.717 us, min = 13.857 us, total = 10.056 ms
InternalKVGcsService.grpc_server.InternalKVGet - 107 total (0 active), Execution time: mean = 612.344 us, total = 65.521 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 96 total (0 active), Execution time: mean = 29.419 us, total = 2.824 ms, Queueing time: mean = 86.797 us, max = 190.114 us, min = 22.398 us, total = 8.332 ms
NodeInfoGcsService.grpc_server.CheckAlive - 96 total (0 active), Execution time: mean = 512.000 us, total = 49.152 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 81 total (1 active), Execution time: mean = 10.000 us, total = 810.034 us, Queueing time: mean = 59.949 us, max = 151.718 us, min = 19.030 us, total = 4.856 ms
HealthCheck - 78 total (0 active), Execution time: mean = 7.244 us, total = 565.050 us, Queueing time: mean = 76.806 us, max = 199.261 us, min = 17.698 us, total = 5.991 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 48 total (0 active), Execution time: mean = 605.169 us, total = 29.048 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 48 total (0 active), Execution time: mean = 80.613 us, total = 3.869 ms, Queueing time: mean = 94.495 us, max = 190.945 us, min = 22.178 us, total = 4.536 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GCSServer.deadline_timer.debug_state_dump - 24 total (1 active), Execution time: mean = 1.372 ms, total = 32.920 ms, Queueing time: mean = 57.811 us, max = 137.944 us, min = 11.375 us, total = 1.387 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
GcsResourceManager::Update - 6 total (0 active), Execution time: mean = 114.415 us, total = 686.489 us, Queueing time: mean = 123.233 us, max = 158.567 us, min = 98.679 us, total = 739.400 us
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
GCSServer.deadline_timer.debug_state_event_stats_print - 4 total (1 active, 1 running), Execution time: mean = 1.115 ms, total = 4.458 ms, Queueing time: mean = 40.390 us, max = 90.983 us, min = 15.823 us, total = 161.560 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
[2025-01-21 05:51:33,588 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 10924 total (1 active)
Queueing time: mean = 45.067 us, max = 7.604 ms, min = 2.691 us, total = 492.314 ms
Execution time: mean = 226.803 us, total = 2.478 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 4956 total (0 active), Execution time: mean = 23.035 us, total = 114.159 ms, Queueing time: mean = 95.402 us, max = 401.821 us, min = 4.632 us, total = 472.811 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 4956 total (0 active), Execution time: mean = 471.837 us, total = 2.338 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 25.466 us, total = 24.498 ms, Queueing time: mean = 16.479 us, max = 7.604 ms, min = 2.691 us, total = 15.853 ms
GcsTaskManager.GcJobSummary - 49 total (1 active), Execution time: mean = 10.391 us, total = 509.178 us, Queueing time: mean = 72.563 us, max = 167.540 us, min = 28.413 us, total = 3.556 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:51:33,588 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 1270 total (25 active)
Queueing time: mean = 27.110 us, max = 8.828 ms, min = 2.456 us, total = 34.429 ms
Execution time: mean = 183.795 ms, total = 233.419 s
Event stats:
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 24.461 us, total = 23.532 ms, Queueing time: mean = 18.133 us, max = 8.828 ms, min = 2.456 us, total = 17.444 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 76 total (0 active), Execution time: mean = 34.893 us, total = 2.652 ms, Queueing time: mean = 104.100 us, max = 347.872 us, min = 5.878 us, total = 7.912 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 76 total (24 active), Execution time: mean = 3.070 s, total = 233.342 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 52 total (0 active), Execution time: mean = 97.773 us, total = 5.084 ms, Queueing time: mean = 89.251 us, max = 191.279 us, min = 23.555 us, total = 4.641 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 52 total (0 active), Execution time: mean = 572.702 us, total = 29.780 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:51:33,588 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 976 total (0 active)
Queueing time: mean = 10.588 us, max = 779.592 us, min = 456.000 ns, total = 10.334 ms
Execution time: mean = 26.244 us, total = 25.614 ms
Event stats:
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 25.824 us, total = 24.843 ms, Queueing time: mean = 10.088 us, max = 779.592 us, min = 1.752 us, total = 9.704 ms
RaySyncer.BroadcastMessage - 6 total (0 active), Execution time: mean = 24.390 us, total = 146.342 us, Queueing time: mean = 934.333 ns, max = 1.284 us, min = 456.000 ns, total = 5.606 us
- 6 total (0 active), Execution time: mean = 103.789 us, total = 622.734 us, Queueing time: mean = 72.303 us, max = 131.850 us, min = 23.468 us, total = 433.820 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:52:33,588 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 60
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 64
- cumulative published bytes: 52815
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 12
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 4
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 3
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:52:33,589 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 3992 total (4 active)
Queueing time: mean = 608.107 us, max = 251.846 ms, min = 853.000 ns, total = 2.428 s
Execution time: mean = 305.390 us, total = 1.219 s
Event stats:
event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 21.062 us, total = 25.317 ms, Queueing time: mean = 20.749 us, max = 14.800 ms, min = 1.906 us, total = 24.940 ms
GcsInMemoryStore.Put - 527 total (0 active), Execution time: mean = 485.781 us, total = 256.006 ms, Queueing time: mean = 3.373 ms, max = 251.330 ms, min = 2.247 us, total = 1.778 s
RayletLoadPulled - 300 total (1 active), Execution time: mean = 239.126 us, total = 71.738 ms, Queueing time: mean = 63.560 us, max = 202.188 us, min = 14.896 us, total = 19.068 ms
NodeManagerService.grpc_client.GetResourceLoad - 296 total (0 active), Execution time: mean = 1.547 ms, total = 458.022 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 296 total (0 active), Execution time: mean = 71.344 us, total = 21.118 ms, Queueing time: mean = 96.022 us, max = 2.124 ms, min = 13.688 us, total = 28.423 ms
InternalKVGcsService.grpc_server.InternalKVPut - 135 total (0 active), Execution time: mean = 516.185 us, total = 69.685 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 135 total (0 active), Execution time: mean = 35.597 us, total = 4.806 ms, Queueing time: mean = 87.807 us, max = 254.636 us, min = 12.388 us, total = 11.854 ms
GcsInMemoryStore.Get - 120 total (0 active), Execution time: mean = 22.255 us, total = 2.671 ms, Queueing time: mean = 16.840 us, max = 337.749 us, min = 3.595 us, total = 2.021 ms
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 120 total (0 active), Execution time: mean = 29.901 us, total = 3.588 ms, Queueing time: mean = 90.754 us, max = 211.127 us, min = 22.398 us, total = 10.890 ms
NodeInfoGcsService.grpc_server.CheckAlive - 120 total (0 active), Execution time: mean = 525.557 us, total = 63.067 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 119 total (0 active), Execution time: mean = 46.698 us, total = 5.557 ms, Queueing time: mean = 92.308 us, max = 240.717 us, min = 12.541 us, total = 10.985 ms
InternalKVGcsService.grpc_server.InternalKVGet - 119 total (0 active), Execution time: mean = 601.147 us, total = 71.537 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 101 total (1 active), Execution time: mean = 9.856 us, total = 995.462 us, Queueing time: mean = 61.982 us, max = 151.718 us, min = 19.030 us, total = 6.260 ms
HealthCheck - 98 total (0 active), Execution time: mean = 7.413 us, total = 726.426 us, Queueing time: mean = 81.281 us, max = 199.261 us, min = 17.698 us, total = 7.965 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 60 total (0 active), Execution time: mean = 599.472 us, total = 35.968 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 60 total (0 active), Execution time: mean = 80.139 us, total = 4.808 ms, Queueing time: mean = 92.733 us, max = 190.945 us, min = 22.178 us, total = 5.564 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
GCSServer.deadline_timer.debug_state_dump - 30 total (1 active), Execution time: mean = 1.388 ms, total = 41.639 ms, Queueing time: mean = 59.311 us, max = 137.944 us, min = 11.375 us, total = 1.779 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsResourceManager::Update - 6 total (0 active), Execution time: mean = 114.415 us, total = 686.489 us, Queueing time: mean = 123.233 us, max = 158.567 us, min = 98.679 us, total = 739.400 us
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
GCSServer.deadline_timer.debug_state_event_stats_print - 5 total (1 active, 1 running), Execution time: mean = 1.235 ms, total = 6.177 ms, Queueing time: mean = 37.313 us, max = 90.983 us, min = 15.823 us, total = 186.565 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
[2025-01-21 05:52:33,589 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 13696 total (1 active)
Queueing time: mean = 45.412 us, max = 7.604 ms, min = 2.691 us, total = 621.963 ms
Execution time: mean = 230.707 us, total = 3.160 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 6216 total (0 active), Execution time: mean = 23.266 us, total = 144.621 ms, Queueing time: mean = 96.456 us, max = 401.821 us, min = 4.632 us, total = 599.573 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 6216 total (0 active), Execution time: mean = 480.080 us, total = 2.984 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 25.227 us, total = 30.323 ms, Queueing time: mean = 14.921 us, max = 7.604 ms, min = 2.691 us, total = 17.934 ms
GcsTaskManager.GcJobSummary - 61 total (1 active), Execution time: mean = 10.197 us, total = 622.010 us, Queueing time: mean = 71.485 us, max = 167.540 us, min = 28.413 us, total = 4.361 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:52:33,590 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 1559 total (25 active)
Queueing time: mean = 25.093 us, max = 8.828 ms, min = 2.456 us, total = 39.120 ms
Execution time: mean = 188.304 ms, total = 293.565 s
Event stats:
event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 24.268 us, total = 29.170 ms, Queueing time: mean = 16.278 us, max = 8.828 ms, min = 2.456 us, total = 19.566 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 88 total (0 active), Execution time: mean = 37.929 us, total = 3.338 ms, Queueing time: mean = 103.251 us, max = 347.872 us, min = 5.878 us, total = 9.086 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 88 total (24 active), Execution time: mean = 3.335 s, total = 293.473 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 64 total (0 active), Execution time: mean = 97.235 us, total = 6.223 ms, Queueing time: mean = 93.084 us, max = 191.279 us, min = 23.555 us, total = 5.957 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 64 total (0 active), Execution time: mean = 580.228 us, total = 37.135 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:52:33,590 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 1216 total (0 active)
Queueing time: mean = 10.528 us, max = 779.592 us, min = 456.000 ns, total = 12.802 ms
Execution time: mean = 26.346 us, total = 32.037 ms
Event stats:
event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 26.012 us, total = 31.266 ms, Queueing time: mean = 10.127 us, max = 779.592 us, min = 1.752 us, total = 12.173 ms
RaySyncer.BroadcastMessage - 6 total (0 active), Execution time: mean = 24.390 us, total = 146.342 us, Queueing time: mean = 934.333 ns, max = 1.284 us, min = 456.000 ns, total = 5.606 us
- 6 total (0 active), Execution time: mean = 103.789 us, total = 622.734 us, Queueing time: mean = 72.303 us, max = 131.850 us, min = 23.468 us, total = 433.820 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:53:33,590 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 72
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 77
- cumulative published bytes: 63094
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 17
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 5
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 4
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:53:33,591 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 4713 total (4 active)
Queueing time: mean = 521.136 us, max = 251.846 ms, min = 853.000 ns, total = 2.456 s
Execution time: mean = 296.451 us, total = 1.397 s
Event stats:
event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 21.174 us, total = 30.533 ms, Queueing time: mean = 18.727 us, max = 14.800 ms, min = 1.906 us, total = 27.004 ms
GcsInMemoryStore.Put - 623 total (0 active), Execution time: mean = 411.851 us, total = 256.583 ms, Queueing time: mean = 2.859 ms, max = 251.330 ms, min = 2.247 us, total = 1.781 s
RayletLoadPulled - 360 total (1 active), Execution time: mean = 243.595 us, total = 87.694 ms, Queueing time: mean = 64.708 us, max = 202.188 us, min = 14.896 us, total = 23.295 ms
NodeManagerService.grpc_client.GetResourceLoad - 356 total (0 active), Execution time: mean = 1.555 ms, total = 553.550 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 356 total (0 active), Execution time: mean = 72.585 us, total = 25.840 ms, Queueing time: mean = 98.002 us, max = 2.124 ms, min = 13.688 us, total = 34.889 ms
InternalKVGcsService.grpc_server.InternalKVPut - 159 total (0 active), Execution time: mean = 522.259 us, total = 83.039 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 159 total (0 active), Execution time: mean = 35.946 us, total = 5.715 ms, Queueing time: mean = 90.140 us, max = 254.636 us, min = 12.388 us, total = 14.332 ms
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 144 total (0 active), Execution time: mean = 30.488 us, total = 4.390 ms, Queueing time: mean = 94.624 us, max = 216.808 us, min = 22.398 us, total = 13.626 ms
NodeInfoGcsService.grpc_server.CheckAlive - 144 total (0 active), Execution time: mean = 529.306 us, total = 76.220 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Get - 132 total (0 active), Execution time: mean = 22.371 us, total = 2.953 ms, Queueing time: mean = 16.390 us, max = 337.749 us, min = 3.595 us, total = 2.164 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 131 total (0 active), Execution time: mean = 46.457 us, total = 6.086 ms, Queueing time: mean = 93.693 us, max = 240.717 us, min = 12.541 us, total = 12.274 ms
InternalKVGcsService.grpc_server.InternalKVGet - 131 total (0 active), Execution time: mean = 599.666 us, total = 78.556 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 121 total (1 active), Execution time: mean = 9.932 us, total = 1.202 ms, Queueing time: mean = 63.858 us, max = 151.718 us, min = 19.030 us, total = 7.727 ms
HealthCheck - 118 total (0 active), Execution time: mean = 7.616 us, total = 898.646 us, Queueing time: mean = 84.027 us, max = 199.261 us, min = 17.698 us, total = 9.915 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 72 total (0 active), Execution time: mean = 610.827 us, total = 43.980 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 72 total (0 active), Execution time: mean = 80.590 us, total = 5.802 ms, Queueing time: mean = 95.639 us, max = 190.945 us, min = 22.178 us, total = 6.886 ms
GCSServer.deadline_timer.debug_state_dump - 36 total (1 active), Execution time: mean = 1.399 ms, total = 50.379 ms, Queueing time: mean = 60.391 us, max = 137.944 us, min = 11.375 us, total = 2.174 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsResourceManager::Update - 8 total (0 active), Execution time: mean = 112.004 us, total = 896.028 us, Queueing time: mean = 126.100 us, max = 158.567 us, min = 98.679 us, total = 1.009 ms
GCSServer.deadline_timer.debug_state_event_stats_print - 6 total (1 active, 1 running), Execution time: mean = 1.308 ms, total = 7.849 ms, Queueing time: mean = 40.277 us, max = 90.983 us, min = 15.823 us, total = 241.663 us
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
[2025-01-21 05:53:33,591 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 16468 total (1 active)
Queueing time: mean = 46.567 us, max = 7.604 ms, min = 2.691 us, total = 766.869 ms
Execution time: mean = 236.090 us, total = 3.888 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 7476 total (0 active), Execution time: mean = 23.763 us, total = 177.651 ms, Queueing time: mean = 99.132 us, max = 401.821 us, min = 4.632 us, total = 741.110 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 7476 total (0 active), Execution time: mean = 491.289 us, total = 3.673 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 25.411 us, total = 36.643 ms, Queueing time: mean = 14.011 us, max = 7.604 ms, min = 2.691 us, total = 20.204 ms
GcsTaskManager.GcJobSummary - 73 total (1 active), Execution time: mean = 10.272 us, total = 749.880 us, Queueing time: mean = 74.801 us, max = 167.540 us, min = 28.413 us, total = 5.460 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:53:33,591 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 1851 total (25 active)
Queueing time: mean = 23.903 us, max = 8.828 ms, min = 2.456 us, total = 44.244 ms
Execution time: mean = 191.103 ms, total = 353.732 s
Event stats:
event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 24.248 us, total = 34.965 ms, Queueing time: mean = 15.082 us, max = 8.828 ms, min = 2.456 us, total = 21.748 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 101 total (0 active), Execution time: mean = 39.215 us, total = 3.961 ms, Queueing time: mean = 103.415 us, max = 347.872 us, min = 5.878 us, total = 10.445 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 101 total (24 active), Execution time: mean = 3.501 s, total = 353.622 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 77 total (0 active), Execution time: mean = 101.615 us, total = 7.824 ms, Queueing time: mean = 97.939 us, max = 191.279 us, min = 23.555 us, total = 7.541 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 77 total (0 active), Execution time: mean = 602.516 us, total = 46.394 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:53:33,591 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 1460 total (0 active)
Queueing time: mean = 10.580 us, max = 779.592 us, min = 456.000 ns, total = 15.447 ms
Execution time: mean = 26.493 us, total = 38.680 ms
Event stats:
event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 26.048 us, total = 37.562 ms, Queueing time: mean = 10.090 us, max = 779.592 us, min = 1.752 us, total = 14.550 ms
RaySyncer.BroadcastMessage - 8 total (0 active), Execution time: mean = 26.053 us, total = 208.422 us, Queueing time: mean = 965.875 ns, max = 1.284 us, min = 456.000 ns, total = 7.727 us
- 8 total (0 active), Execution time: mean = 113.459 us, total = 907.670 us, Queueing time: mean = 87.368 us, max = 146.934 us, min = 23.468 us, total = 698.947 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:54:33,592 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 84
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 90
- cumulative published bytes: 72943
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 17
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 5
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 4
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:54:33,593 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 5432 total (4 active)
Queueing time: mean = 457.183 us, max = 251.846 ms, min = 853.000 ns, total = 2.483 s
Execution time: mean = 289.700 us, total = 1.574 s
Event stats:
event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 21.243 us, total = 35.731 ms, Queueing time: mean = 17.260 us, max = 14.800 ms, min = 1.906 us, total = 29.032 ms
GcsInMemoryStore.Put - 719 total (0 active), Execution time: mean = 357.672 us, total = 257.166 ms, Queueing time: mean = 2.483 ms, max = 251.330 ms, min = 2.247 us, total = 1.785 s
RayletLoadPulled - 420 total (1 active), Execution time: mean = 246.664 us, total = 103.599 ms, Queueing time: mean = 65.672 us, max = 202.188 us, min = 14.896 us, total = 27.582 ms
NodeManagerService.grpc_client.GetResourceLoad - 416 total (0 active), Execution time: mean = 1.561 ms, total = 649.361 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 416 total (0 active), Execution time: mean = 73.159 us, total = 30.434 ms, Queueing time: mean = 98.309 us, max = 2.124 ms, min = 13.688 us, total = 40.897 ms
InternalKVGcsService.grpc_server.InternalKVPut - 183 total (0 active), Execution time: mean = 524.436 us, total = 95.972 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 183 total (0 active), Execution time: mean = 36.461 us, total = 6.672 ms, Queueing time: mean = 90.924 us, max = 254.636 us, min = 12.388 us, total = 16.639 ms
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 168 total (0 active), Execution time: mean = 30.483 us, total = 5.121 ms, Queueing time: mean = 95.312 us, max = 216.808 us, min = 22.398 us, total = 16.012 ms
NodeInfoGcsService.grpc_server.CheckAlive - 168 total (0 active), Execution time: mean = 524.842 us, total = 88.174 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Get - 144 total (0 active), Execution time: mean = 22.805 us, total = 3.284 ms, Queueing time: mean = 16.054 us, max = 337.749 us, min = 3.595 us, total = 2.312 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 143 total (0 active), Execution time: mean = 47.273 us, total = 6.760 ms, Queueing time: mean = 93.956 us, max = 240.717 us, min = 12.541 us, total = 13.436 ms
InternalKVGcsService.grpc_server.InternalKVGet - 143 total (0 active), Execution time: mean = 598.521 us, total = 85.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 141 total (1 active), Execution time: mean = 9.873 us, total = 1.392 ms, Queueing time: mean = 64.648 us, max = 151.718 us, min = 16.805 us, total = 9.115 ms
HealthCheck - 138 total (0 active), Execution time: mean = 7.684 us, total = 1.060 ms, Queueing time: mean = 87.606 us, max = 199.261 us, min = 17.698 us, total = 12.090 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 84 total (0 active), Execution time: mean = 615.923 us, total = 51.738 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 84 total (0 active), Execution time: mean = 82.166 us, total = 6.902 ms, Queueing time: mean = 97.472 us, max = 190.945 us, min = 22.178 us, total = 8.188 ms
GCSServer.deadline_timer.debug_state_dump - 42 total (1 active), Execution time: mean = 1.413 ms, total = 59.347 ms, Queueing time: mean = 62.353 us, max = 137.944 us, min = 11.375 us, total = 2.619 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsResourceManager::Update - 8 total (0 active), Execution time: mean = 112.004 us, total = 896.028 us, Queueing time: mean = 126.100 us, max = 158.567 us, min = 98.679 us, total = 1.009 ms
GCSServer.deadline_timer.debug_state_event_stats_print - 7 total (1 active, 1 running), Execution time: mean = 1.350 ms, total = 9.448 ms, Queueing time: mean = 41.430 us, max = 90.983 us, min = 15.823 us, total = 290.007 us
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
[2025-01-21 05:54:33,593 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 19240 total (1 active)
Queueing time: mean = 47.331 us, max = 7.604 ms, min = 2.691 us, total = 910.652 ms
Execution time: mean = 239.598 us, total = 4.610 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 8736 total (0 active), Execution time: mean = 24.309 us, total = 212.364 ms, Queueing time: mean = 100.961 us, max = 401.821 us, min = 4.632 us, total = 882.000 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 8736 total (0 active), Execution time: mean = 498.401 us, total = 4.354 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 25.298 us, total = 42.551 ms, Queueing time: mean = 13.228 us, max = 7.604 ms, min = 2.691 us, total = 22.250 ms
GcsTaskManager.GcJobSummary - 85 total (1 active), Execution time: mean = 10.596 us, total = 900.685 us, Queueing time: mean = 74.208 us, max = 167.540 us, min = 28.413 us, total = 6.308 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:54:33,593 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 2143 total (25 active)
Queueing time: mean = 23.000 us, max = 8.828 ms, min = 2.456 us, total = 49.289 ms
Execution time: mean = 193.141 ms, total = 413.902 s
Event stats:
event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 24.331 us, total = 40.925 ms, Queueing time: mean = 14.244 us, max = 8.828 ms, min = 2.456 us, total = 23.959 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 114 total (0 active), Execution time: mean = 39.182 us, total = 4.467 ms, Queueing time: mean = 105.139 us, max = 347.872 us, min = 5.878 us, total = 11.986 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 114 total (24 active), Execution time: mean = 3.630 s, total = 413.777 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 90 total (0 active), Execution time: mean = 100.418 us, total = 9.038 ms, Queueing time: mean = 98.157 us, max = 191.279 us, min = 23.555 us, total = 8.834 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 90 total (0 active), Execution time: mean = 603.694 us, total = 54.332 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:54:33,593 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 1700 total (0 active)
Queueing time: mean = 10.510 us, max = 779.592 us, min = 456.000 ns, total = 17.867 ms
Execution time: mean = 26.541 us, total = 45.120 ms
Event stats:
event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 26.161 us, total = 44.002 ms, Queueing time: mean = 10.089 us, max = 779.592 us, min = 1.752 us, total = 16.970 ms
RaySyncer.BroadcastMessage - 8 total (0 active), Execution time: mean = 26.053 us, total = 208.422 us, Queueing time: mean = 965.875 ns, max = 1.284 us, min = 456.000 ns, total = 7.727 us
- 8 total (0 active), Execution time: mean = 113.459 us, total = 907.670 us, Queueing time: mean = 87.368 us, max = 146.934 us, min = 23.468 us, total = 698.947 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:55:33,593 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 96
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 103
- cumulative published bytes: 83131
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 20
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 6
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 5
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:55:33,594 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 6152 total (4 active)
Queueing time: mean = 408.569 us, max = 251.846 ms, min = 853.000 ns, total = 2.514 s
Execution time: mean = 284.953 us, total = 1.753 s
Event stats:
event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 21.302 us, total = 40.942 ms, Queueing time: mean = 16.172 us, max = 14.800 ms, min = 1.906 us, total = 31.082 ms
GcsInMemoryStore.Put - 815 total (0 active), Execution time: mean = 316.264 us, total = 257.755 ms, Queueing time: mean = 2.196 ms, max = 251.330 ms, min = 2.247 us, total = 1.790 s
RayletLoadPulled - 480 total (1 active), Execution time: mean = 249.454 us, total = 119.738 ms, Queueing time: mean = 66.387 us, max = 202.188 us, min = 14.896 us, total = 31.866 ms
NodeManagerService.grpc_client.GetResourceLoad - 476 total (0 active), Execution time: mean = 1.565 ms, total = 744.792 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 476 total (0 active), Execution time: mean = 73.566 us, total = 35.018 ms, Queueing time: mean = 100.260 us, max = 2.124 ms, min = 13.688 us, total = 47.724 ms
InternalKVGcsService.grpc_server.InternalKVPut - 207 total (0 active), Execution time: mean = 531.626 us, total = 110.047 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 207 total (0 active), Execution time: mean = 37.358 us, total = 7.733 ms, Queueing time: mean = 92.613 us, max = 254.636 us, min = 12.388 us, total = 19.171 ms
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 192 total (0 active), Execution time: mean = 30.339 us, total = 5.825 ms, Queueing time: mean = 97.276 us, max = 216.808 us, min = 22.398 us, total = 18.677 ms
NodeInfoGcsService.grpc_server.CheckAlive - 192 total (0 active), Execution time: mean = 526.057 us, total = 101.003 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 161 total (1 active), Execution time: mean = 9.795 us, total = 1.577 ms, Queueing time: mean = 65.945 us, max = 151.718 us, min = 16.805 us, total = 10.617 ms
HealthCheck - 158 total (0 active), Execution time: mean = 7.734 us, total = 1.222 ms, Queueing time: mean = 89.494 us, max = 199.261 us, min = 17.698 us, total = 14.140 ms
GcsInMemoryStore.Get - 156 total (0 active), Execution time: mean = 23.030 us, total = 3.593 ms, Queueing time: mean = 15.772 us, max = 337.749 us, min = 3.595 us, total = 2.460 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 155 total (0 active), Execution time: mean = 47.539 us, total = 7.368 ms, Queueing time: mean = 95.770 us, max = 240.717 us, min = 12.541 us, total = 14.844 ms
InternalKVGcsService.grpc_server.InternalKVGet - 155 total (0 active), Execution time: mean = 599.794 us, total = 92.968 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 96 total (0 active), Execution time: mean = 83.684 us, total = 8.034 ms, Queueing time: mean = 100.710 us, max = 190.945 us, min = 22.178 us, total = 9.668 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 96 total (0 active), Execution time: mean = 626.306 us, total = 60.125 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GCSServer.deadline_timer.debug_state_dump - 48 total (1 active), Execution time: mean = 1.423 ms, total = 68.282 ms, Queueing time: mean = 63.136 us, max = 137.944 us, min = 11.375 us, total = 3.031 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
GcsResourceManager::Update - 9 total (0 active), Execution time: mean = 112.246 us, total = 1.010 ms, Queueing time: mean = 128.516 us, max = 158.567 us, min = 98.679 us, total = 1.157 ms
GCSServer.deadline_timer.debug_state_event_stats_print - 8 total (1 active, 1 running), Execution time: mean = 1.374 ms, total = 10.996 ms, Queueing time: mean = 45.210 us, max = 90.983 us, min = 15.823 us, total = 361.682 us
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
[2025-01-21 05:55:33,594 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 22012 total (1 active)
Queueing time: mean = 47.734 us, max = 7.604 ms, min = 2.691 us, total = 1.051 s
Execution time: mean = 242.040 us, total = 5.328 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 9996 total (0 active), Execution time: mean = 24.583 us, total = 245.727 ms, Queueing time: mean = 101.952 us, max = 401.821 us, min = 4.632 us, total = 1.019 s
TaskInfoGcsService.grpc_server.AddTaskEventData - 9996 total (0 active), Execution time: mean = 503.448 us, total = 5.032 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 25.266 us, total = 48.562 ms, Queueing time: mean = 12.642 us, max = 7.604 ms, min = 2.691 us, total = 24.298 ms
GcsTaskManager.GcJobSummary - 97 total (1 active), Execution time: mean = 10.483 us, total = 1.017 ms, Queueing time: mean = 74.521 us, max = 167.540 us, min = 28.413 us, total = 7.229 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:55:33,594 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 2435 total (25 active)
Queueing time: mean = 22.300 us, max = 8.828 ms, min = 2.456 us, total = 54.300 ms
Execution time: mean = 194.687 ms, total = 474.062 s
Event stats:
event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 24.684 us, total = 47.443 ms, Queueing time: mean = 13.658 us, max = 8.828 ms, min = 2.456 us, total = 26.251 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 127 total (0 active), Execution time: mean = 40.539 us, total = 5.148 ms, Queueing time: mean = 105.201 us, max = 347.872 us, min = 5.878 us, total = 13.360 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 127 total (24 active), Execution time: mean = 3.732 s, total = 473.920 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 103 total (0 active), Execution time: mean = 102.970 us, total = 10.606 ms, Queueing time: mean = 98.814 us, max = 191.279 us, min = 23.555 us, total = 10.178 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 103 total (0 active), Execution time: mean = 609.231 us, total = 62.751 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:55:33,595 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 1942 total (0 active)
Queueing time: mean = 10.490 us, max = 779.592 us, min = 456.000 ns, total = 20.372 ms
Execution time: mean = 26.623 us, total = 51.703 ms
Event stats:
event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 26.238 us, total = 50.429 ms, Queueing time: mean = 10.072 us, max = 779.592 us, min = 1.752 us, total = 19.359 ms
RaySyncer.BroadcastMessage - 9 total (0 active), Execution time: mean = 26.309 us, total = 236.777 us, Queueing time: mean = 967.667 ns, max = 1.284 us, min = 456.000 ns, total = 8.709 us
- 9 total (0 active), Execution time: mean = 115.051 us, total = 1.035 ms, Queueing time: mean = 90.475 us, max = 146.934 us, min = 23.468 us, total = 814.278 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:56:33,595 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 108
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 115
- cumulative published bytes: 92947
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 20
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 6
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 5
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:56:33,596 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 6871 total (4 active)
Queueing time: mean = 369.918 us, max = 251.846 ms, min = 853.000 ns, total = 2.542 s
Execution time: mean = 280.558 us, total = 1.928 s
Event stats:
event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 21.206 us, total = 45.847 ms, Queueing time: mean = 15.280 us, max = 14.800 ms, min = 1.906 us, total = 33.035 ms
GcsInMemoryStore.Put - 911 total (0 active), Execution time: mean = 283.559 us, total = 258.323 ms, Queueing time: mean = 1.968 ms, max = 251.330 ms, min = 2.247 us, total = 1.793 s
RayletLoadPulled - 540 total (1 active), Execution time: mean = 251.835 us, total = 135.991 ms, Queueing time: mean = 66.612 us, max = 202.188 us, min = 14.896 us, total = 35.971 ms
NodeManagerService.grpc_client.GetResourceLoad - 536 total (0 active), Execution time: mean = 1.567 ms, total = 839.808 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 536 total (0 active), Execution time: mean = 74.560 us, total = 39.964 ms, Queueing time: mean = 101.427 us, max = 2.124 ms, min = 13.688 us, total = 54.365 ms
InternalKVGcsService.grpc_server.InternalKVPut - 231 total (0 active), Execution time: mean = 532.666 us, total = 123.046 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 231 total (0 active), Execution time: mean = 37.260 us, total = 8.607 ms, Queueing time: mean = 93.995 us, max = 254.636 us, min = 12.388 us, total = 21.713 ms
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 216 total (0 active), Execution time: mean = 30.408 us, total = 6.568 ms, Queueing time: mean = 98.657 us, max = 216.808 us, min = 22.398 us, total = 21.310 ms
NodeInfoGcsService.grpc_server.CheckAlive - 216 total (0 active), Execution time: mean = 522.375 us, total = 112.833 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 181 total (1 active), Execution time: mean = 9.873 us, total = 1.787 ms, Queueing time: mean = 66.131 us, max = 151.718 us, min = 16.805 us, total = 11.970 ms
HealthCheck - 178 total (0 active), Execution time: mean = 7.732 us, total = 1.376 ms, Queueing time: mean = 90.008 us, max = 199.261 us, min = 17.698 us, total = 16.021 ms
GcsInMemoryStore.Get - 168 total (0 active), Execution time: mean = 23.814 us, total = 4.001 ms, Queueing time: mean = 15.568 us, max = 337.749 us, min = 3.595 us, total = 2.616 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 167 total (0 active), Execution time: mean = 47.880 us, total = 7.996 ms, Queueing time: mean = 96.127 us, max = 240.717 us, min = 12.541 us, total = 16.053 ms
InternalKVGcsService.grpc_server.InternalKVGet - 167 total (0 active), Execution time: mean = 598.527 us, total = 99.954 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 108 total (0 active), Execution time: mean = 83.147 us, total = 8.980 ms, Queueing time: mean = 104.636 us, max = 211.974 us, min = 22.178 us, total = 11.301 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 108 total (0 active), Execution time: mean = 629.266 us, total = 67.961 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GCSServer.deadline_timer.debug_state_dump - 54 total (1 active), Execution time: mean = 1.410 ms, total = 76.161 ms, Queueing time: mean = 61.739 us, max = 137.944 us, min = 11.375 us, total = 3.334 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
GCSServer.deadline_timer.debug_state_event_stats_print - 9 total (1 active, 1 running), Execution time: mean = 1.388 ms, total = 12.493 ms, Queueing time: mean = 45.555 us, max = 90.983 us, min = 15.823 us, total = 409.992 us
GcsResourceManager::Update - 9 total (0 active), Execution time: mean = 112.246 us, total = 1.010 ms, Queueing time: mean = 128.516 us, max = 158.567 us, min = 98.679 us, total = 1.157 ms
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
[2025-01-21 05:56:33,596 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 24784 total (1 active)
Queueing time: mean = 47.600 us, max = 7.604 ms, min = 2.691 us, total = 1.180 s
Execution time: mean = 242.233 us, total = 6.003 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 11256 total (0 active), Execution time: mean = 24.597 us, total = 276.865 ms, Queueing time: mean = 101.741 us, max = 401.821 us, min = 4.632 us, total = 1.145 s
TaskInfoGcsService.grpc_server.AddTaskEventData - 11256 total (0 active), Execution time: mean = 503.811 us, total = 5.671 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 25.249 us, total = 54.589 ms, Queueing time: mean = 12.189 us, max = 7.604 ms, min = 2.691 us, total = 26.352 ms
GcsTaskManager.GcJobSummary - 109 total (1 active), Execution time: mean = 10.302 us, total = 1.123 ms, Queueing time: mean = 74.094 us, max = 167.540 us, min = 28.413 us, total = 8.076 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:56:33,596 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 2723 total (25 active)
Queueing time: mean = 21.508 us, max = 8.828 ms, min = 2.456 us, total = 58.567 ms
Execution time: mean = 196.200 ms, total = 534.251 s
Event stats:
event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 24.706 us, total = 53.414 ms, Queueing time: mean = 13.192 us, max = 8.828 ms, min = 2.456 us, total = 28.520 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 139 total (0 active), Execution time: mean = 40.437 us, total = 5.621 ms, Queueing time: mean = 102.308 us, max = 347.872 us, min = 5.878 us, total = 14.221 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 139 total (24 active), Execution time: mean = 3.842 s, total = 534.095 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 115 total (0 active), Execution time: mean = 102.425 us, total = 11.779 ms, Queueing time: mean = 98.395 us, max = 191.279 us, min = 23.555 us, total = 11.315 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 115 total (0 active), Execution time: mean = 604.014 us, total = 69.462 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.824 us, total = 15.647 us, Queueing time: mean = 38.842 us, max = 77.683 us, min = 77.683 us, total = 77.683 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:56:33,596 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 2182 total (0 active)
Queueing time: mean = 10.447 us, max = 779.592 us, min = 456.000 ns, total = 22.795 ms
Execution time: mean = 26.748 us, total = 58.365 ms
Event stats:
event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 26.406 us, total = 57.090 ms, Queueing time: mean = 10.075 us, max = 779.592 us, min = 1.752 us, total = 21.782 ms
RaySyncer.BroadcastMessage - 9 total (0 active), Execution time: mean = 26.309 us, total = 236.777 us, Queueing time: mean = 967.667 ns, max = 1.284 us, min = 456.000 ns, total = 8.709 us
- 9 total (0 active), Execution time: mean = 115.051 us, total = 1.035 ms, Queueing time: mean = 90.475 us, max = 146.934 us, min = 23.468 us, total = 814.278 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:57:33,597 I 18551 18551] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 32
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 120
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 127
- cumulative published bytes: 102765
- current buffered bytes: 819
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 20
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 6
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 5
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 05:57:33,598 I 18551 18551] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 7590 total (4 active)
Queueing time: mean = 338.039 us, max = 251.846 ms, min = 853.000 ns, total = 2.566 s
Execution time: mean = 274.599 us, total = 2.084 s
Event stats:
event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 21.158 us, total = 50.822 ms, Queueing time: mean = 14.559 us, max = 14.800 ms, min = 1.906 us, total = 34.971 ms
GcsInMemoryStore.Put - 1007 total (0 active), Execution time: mean = 257.012 us, total = 258.811 ms, Queueing time: mean = 1.785 ms, max = 251.330 ms, min = 2.247 us, total = 1.797 s
RayletLoadPulled - 600 total (1 active), Execution time: mean = 252.532 us, total = 151.519 ms, Queueing time: mean = 66.206 us, max = 202.188 us, min = 14.896 us, total = 39.724 ms
NodeManagerService.grpc_client.GetResourceLoad - 596 total (0 active), Execution time: mean = 1.547 ms, total = 921.735 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 596 total (0 active), Execution time: mean = 74.630 us, total = 44.479 ms, Queueing time: mean = 99.525 us, max = 2.124 ms, min = 13.688 us, total = 59.317 ms
InternalKVGcsService.grpc_server.InternalKVPut - 255 total (0 active), Execution time: mean = 526.294 us, total = 134.205 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 255 total (0 active), Execution time: mean = 36.566 us, total = 9.324 ms, Queueing time: mean = 91.577 us, max = 254.636 us, min = 12.388 us, total = 23.352 ms
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 240 total (0 active), Execution time: mean = 30.998 us, total = 7.439 ms, Queueing time: mean = 97.956 us, max = 216.808 us, min = 14.918 us, total = 23.509 ms
NodeInfoGcsService.grpc_server.CheckAlive - 240 total (0 active), Execution time: mean = 516.369 us, total = 123.928 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 201 total (1 active), Execution time: mean = 9.748 us, total = 1.959 ms, Queueing time: mean = 65.449 us, max = 151.718 us, min = 16.805 us, total = 13.155 ms
HealthCheck - 198 total (0 active), Execution time: mean = 7.693 us, total = 1.523 ms, Queueing time: mean = 89.297 us, max = 199.261 us, min = 17.136 us, total = 17.681 ms
GcsInMemoryStore.Get - 180 total (0 active), Execution time: mean = 23.661 us, total = 4.259 ms, Queueing time: mean = 15.251 us, max = 337.749 us, min = 3.595 us, total = 2.745 ms
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 179 total (0 active), Execution time: mean = 47.566 us, total = 8.514 ms, Queueing time: mean = 96.182 us, max = 240.717 us, min = 12.541 us, total = 17.217 ms
InternalKVGcsService.grpc_server.InternalKVGet - 179 total (0 active), Execution time: mean = 593.606 us, total = 106.255 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 120 total (0 active), Execution time: mean = 82.496 us, total = 9.899 ms, Queueing time: mean = 103.536 us, max = 211.974 us, min = 21.535 us, total = 12.424 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 120 total (0 active), Execution time: mean = 624.939 us, total = 74.993 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GCSServer.deadline_timer.debug_state_dump - 60 total (1 active), Execution time: mean = 1.403 ms, total = 84.209 ms, Queueing time: mean = 62.894 us, max = 187.024 us, min = 11.375 us, total = 3.774 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 32 total (0 active), Execution time: mean = 45.754 us, total = 1.464 ms, Queueing time: mean = 96.826 us, max = 200.145 us, min = 4.641 us, total = 3.098 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 32 total (0 active), Execution time: mean = 588.463 us, total = 18.831 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 1.622 ms, total = 34.060 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 32.752 us, total = 687.799 us, Queueing time: mean = 112.379 us, max = 329.473 us, min = 17.925 us, total = 2.360 ms
GCSServer.deadline_timer.debug_state_event_stats_print - 10 total (1 active, 1 running), Execution time: mean = 1.432 ms, total = 14.315 ms, Queueing time: mean = 47.371 us, max = 90.983 us, min = 15.823 us, total = 473.711 us
GcsResourceManager::Update - 9 total (0 active), Execution time: mean = 112.246 us, total = 1.010 ms, Queueing time: mean = 128.516 us, max = 158.567 us, min = 98.679 us, total = 1.157 ms
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.808 us, total = 76.850 us, Queueing time: mean = 63.707 us, max = 93.369 us, min = 12.191 us, total = 382.240 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 594.980 us, total = 2.975 ms, Queueing time: mean = 454.913 us, max = 1.926 ms, min = 37.965 us, total = 2.275 ms
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.901 ms, total = 9.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 356.690 us, total = 1.427 ms, Queueing time: mean = 127.566 ms, max = 251.846 ms, min = 2.946 ms, total = 510.264 ms
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 647.711 us, total = 647.711 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 210.583 us, total = 210.583 us, Queueing time: mean = 146.426 us, max = 146.426 us, min = 146.426 us, total = 146.426 us
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 20.427 us, total = 20.427 us, Queueing time: mean = 11.010 us, max = 11.010 us, min = 11.010 us, total = 11.010 us
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 27.593 us, total = 27.593 us, Queueing time: mean = 11.858 us, max = 11.858 us, min = 11.858 us, total = 11.858 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 691.858 us, total = 691.858 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 39.511 us, total = 39.511 us, Queueing time: mean = 125.761 us, max = 125.761 us, min = 125.761 us, total = 125.761 us
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 839.122 us, total = 839.122 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 45.900 us, total = 45.900 us, Queueing time: mean = 126.144 us, max = 126.144 us, min = 126.144 us, total = 126.144 us
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 29.468 us, total = 29.468 us, Queueing time: mean = 53.587 us, max = 53.587 us, min = 53.587 us, total = 53.587 us
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 850.573 us, total = 850.573 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.811 us, total = 17.811 us, Queueing time: mean = 853.000 ns, max = 853.000 ns, min = 853.000 ns, total = 853.000 ns
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.226 ms, total = 1.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 497.095 us, total = 497.095 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 575.218 us, total = 575.218 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.741 us, total = 77.741 us, Queueing time: mean = 97.930 us, max = 97.930 us, min = 97.930 us, total = 97.930 us
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 27.480 us, total = 27.480 us, Queueing time: mean = 181.711 us, max = 181.711 us, min = 181.711 us, total = 181.711 us
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 154.576 us, total = 154.576 us, Queueing time: mean = 71.587 us, max = 71.587 us, min = 71.587 us, total = 71.587 us
[2025-01-21 05:57:33,598 I 18551 18551] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 27556 total (1 active)
Queueing time: mean = 46.976 us, max = 7.604 ms, min = 2.473 us, total = 1.294 s
Execution time: mean = 239.789 us, total = 6.608 s
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 12516 total (0 active), Execution time: mean = 24.311 us, total = 304.279 ms, Queueing time: mean = 100.456 us, max = 401.821 us, min = 4.212 us, total = 1.257 s
TaskInfoGcsService.grpc_server.AddTaskEventData - 12516 total (0 active), Execution time: mean = 498.698 us, total = 6.242 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 25.159 us, total = 60.433 ms, Queueing time: mean = 11.820 us, max = 7.604 ms, min = 2.473 us, total = 28.393 ms
GcsTaskManager.GcJobSummary - 121 total (1 active), Execution time: mean = 10.059 us, total = 1.217 ms, Queueing time: mean = 71.671 us, max = 167.540 us, min = 25.713 us, total = 8.672 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.865 us, total = 11.865 us, Queueing time: mean = 94.975 us, max = 94.975 us, min = 94.975 us, total = 94.975 us
[2025-01-21 05:57:33,598 I 18551 18551] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 3058 total (25 active)
Queueing time: mean = 21.363 us, max = 8.828 ms, min = 2.456 us, total = 65.329 ms
Execution time: mean = 4.674 s, total = 14294.617 s
Event stats:
event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 24.610 us, total = 59.112 ms, Queueing time: mean = 12.748 us, max = 8.828 ms, min = 2.456 us, total = 30.620 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 174 total (0 active), Execution time: mean = 37.518 us, total = 6.528 ms, Queueing time: mean = 101.571 us, max = 347.872 us, min = 5.878 us, total = 17.673 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 174 total (24 active), Execution time: mean = 82.152 s, total = 14294.445 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 127 total (0 active), Execution time: mean = 101.531 us, total = 12.894 ms, Queueing time: mean = 98.114 us, max = 200.459 us, min = 23.555 us, total = 12.461 ms
InternalPubSubGcsService.grpc_server.GcsPublish - 127 total (0 active), Execution time: mean = 599.919 us, total = 76.190 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 585.600 us, total = 14.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 67.427 us, total = 1.686 ms, Queueing time: mean = 125.752 us, max = 232.674 us, min = 22.924 us, total = 3.144 ms
Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 237.114 us, total = 711.342 us, Queueing time: mean = 47.608 us, max = 77.683 us, min = 65.142 us, total = 142.825 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 44.551 us, total = 44.551 us, Queueing time: mean = 1.289 ms, max = 1.289 ms, min = 1.289 ms, total = 1.289 ms
[2025-01-21 05:57:33,598 I 18551 18551] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 2422 total (0 active)
Queueing time: mean = 10.347 us, max = 779.592 us, min = 456.000 ns, total = 25.061 ms
Execution time: mean = 26.511 us, total = 64.210 ms
Event stats:
event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 26.201 us, total = 62.936 ms, Queueing time: mean = 10.012 us, max = 779.592 us, min = 1.752 us, total = 24.048 ms
RaySyncer.BroadcastMessage - 9 total (0 active), Execution time: mean = 26.309 us, total = 236.777 us, Queueing time: mean = 967.667 ns, max = 1.284 us, min = 456.000 ns, total = 8.709 us
- 9 total (0 active), Execution time: mean = 115.051 us, total = 1.035 ms, Queueing time: mean = 90.475 us, max = 146.934 us, min = 23.468 us, total = 814.278 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 983.000 ns, total = 1.966 us, Queueing time: mean = 95.018 us, max = 96.396 us, min = 93.640 us, total = 190.036 us
[2025-01-21 05:58:01,517 I 18551 18551] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-21 05:58:01,636 I 18551 18551] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,636 I 18551 18551] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,636 I 18551 18551] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,636 I 18551 18551] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,799 I 18551 18600] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,799 I 18551 18551] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-21 05:58:01,799 I 18551 18600] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
[2025-01-21 05:58:01,801 I 18551 18551] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-21 05:58:01,931 I 18551 18551] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-21 05:58:01,932 I 18551 18551] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-21 05:58:02,048 I 18551 18551] (gcs_server) stats.h:120: Stats module has shutdown.
|