status+osd df:
data:
pools: 14 pools, 680 pgs
objects: 19.10M objects, 24.4TiB
usage: 77.3TiB used, 276TiB / 354TiB avail
pgs: 449 active+clean
195 active+clean+snaptrim_wait
36 active+clean+snaptrim
io:
client: 3.86MiB/s rd, 32.5MiB/s wr, 2.94kop/s rd, 216op/s wr
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
10 ssd 6.98489 1.00000 6.98TiB 3.49TiB 3.50TiB 49.91 2.28 83
11 ssd 6.98560 1.00000 6.99TiB 3.60TiB 3.39TiB 51.46 2.35 98
12 ssd 2.29799 1.00000 2.30TiB 1.40TiB 922GiB 60.82 2.78 35
13 ssd 2.29799 1.00000 2.30TiB 931GiB 1.39TiB 39.55 1.81 24
14 ssd 3.49300 1.00000 3.49TiB 2.24TiB 1.26TiB 64.00 2.93 51
41 ssd 6.98489 1.00000 6.98TiB 4.00TiB 2.98TiB 57.29 2.62 107
42 ssd 6.98560 1.00000 6.99TiB 3.26TiB 3.73TiB 46.63 2.13 90
24 ssd 6.98489 1.00000 6.98TiB 3.21TiB 3.77TiB 45.97 2.10 85
25 ssd 6.98560 1.00000 6.99TiB 3.76TiB 3.23TiB 53.81 2.46 101
27 ssd 2.29799 1.00000 2.30TiB 1.44TiB 882GiB 62.53 2.86 32
28 ssd 2.29799 1.00000 2.30TiB 1.43TiB 891GiB 62.14 2.84 32
29 ssd 3.49300 1.00000 3.49TiB 2.62TiB 894GiB 75.00 3.43 58
43 ssd 6.98489 1.00000 6.98TiB 3.07TiB 3.92TiB 43.93 2.01 87
44 ssd 6.98560 1.00000 6.99TiB 3.67TiB 3.32TiB 52.52 2.40 93
26 ssd 6.98489 1.00000 6.98TiB 4.15TiB 2.83TiB 59.42 2.72 112
38 ssd 2.29990 1.00000 2.30TiB 1.15TiB 1.15TiB 49.89 2.28 29
39 ssd 2.29549 1.00000 2.30TiB 1.40TiB 921GiB 60.80 2.78 33
40 ssd 3.49300 1.00000 3.49TiB 2.15TiB 1.34TiB 61.58 2.82 40
45 ssd 6.98489 1.00000 6.98TiB 4.37TiB 2.61TiB 62.58 2.86 105
46 ssd 6.98560 1.00000 6.99TiB 4.05TiB 2.94TiB 57.91 2.65 88
47 ssd 6.98560 1.00000 6.99TiB 3.33TiB 3.65TiB 47.68 2.18 81
Osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -19 0 root default -16 108.08926 root ssd -9 36.02995 host-ssd 2001-ssd 10 ssd 6.98489 osd.10 up 1.00000 1.00000 11 ssd 6.98560 osd.11 up 1.00000 1.00000 12 ssd 2.29799 osd.12 up 1.00000 1.00000 13 ssd 2.29799 osd.13 up 1.00000 1.00000 14 ssd 3.49300 osd.14 up 1.00000 1.00000 41 ssd 6.98489 osd.41 up 1.00000 1.00000 42 ssd 6.98560 osd.42 up 1.00000 1.00000 -12 36.02995 host-ssd 2002-ssd 24 ssd 6.98489 osd.24 up 1.00000 1.00000 25 ssd 6.98560 osd.25 up 1.00000 1.00000 27 ssd 2.29799 osd.27 up 1.00000 1.00000 28 ssd 2.29799 osd.28 up 1.00000 1.00000 29 ssd 3.49300 osd.29 up 1.00000 1.00000 43 ssd 6.98489 osd.43 up 1.00000 1.00000 44 ssd 6.98560 osd.44 up 1.00000 1.00000 -22 36.02936 host-ssd 2003-ssd 26 ssd 6.98489 osd.26 up 1.00000 1.00000 38 ssd 2.29990 osd.38 up 1.00000 1.00000 39 ssd 2.29549 osd.39 up 1.00000 1.00000 40 ssd 3.49300 osd.40 up 1.00000 1.00000 45 ssd 6.98489 osd.45 up 1.00000 1.00000 46 ssd 6.98560 osd.46 up 1.00000 1.00000 47 ssd 6.98560 osd.47 up 1.00000 1.00000 -1 245.58824 root hdd -3 81.86394 host-hdd 2001-hdd 0 hdd 9.09599 osd.0 up 1.00000 1.00000 1 hdd 9.09599 osd.1 up 1.00000 1.00000 2 hdd 9.09599 osd.2 up 1.00000 1.00000 3 hdd 9.09599 osd.3 up 1.00000 1.00000 4 hdd 9.09599 osd.4 up 1.00000 1.00000 5 hdd 9.09599 osd.5 up 1.00000 1.00000 6 hdd 9.09599 osd.6 up 1.00000 1.00000 7 hdd 9.09599 osd.7 up 1.00000 1.00000 8 hdd 9.09599 osd.8 up 1.00000 1.00000 -5 81.86394 host-hdd 2002-hdd 15 hdd 9.09599 osd.15 up 1.00000 1.00000 16 hdd 9.09599 osd.16 up 1.00000 1.00000 17 hdd 9.09599 osd.17 up 1.00000 1.00000 18 hdd 9.09599 osd.18 up 1.00000 1.00000 19 hdd 9.09599 osd.19 up 1.00000 1.00000 20 hdd 9.09599 osd.20 up 1.00000 1.00000 21 hdd 9.09599 osd.21 up 1.00000 1.00000 22 hdd 9.09599 osd.22 up 1.00000 1.00000 23 hdd 9.09599 osd.23 up 1.00000 1.00000 -25 81.86037 host-hdd 2003-hdd 9 hdd 9.09560 osd.9 up 1.00000 1.00000 30 hdd 9.09560 osd.30 up 1.00000 1.00000 31 hdd 9.09560 osd.31 up 1.00000 1.00000 32 hdd 9.09560 osd.32 up 1.00000 1.00000 33 hdd 9.09560 osd.33 up 1.00000 1.00000 34 hdd 9.09560 osd.34 up 1.00000 1.00000 35 hdd 9.09560 osd.35 up 1.00000 1.00000 36 hdd 9.09560 osd.36 up 1.00000 1.00000 37 hdd 9.09560 osd.37 up 1.00000 1.00000
Affected pools:
pool 20 'dbs-realtime-staging-w-financedb' replicated size 3 min_size 1 crush_rule 1 object_hash rjenkins pg_num 256 pgp_num 256 last_change 8931 lfor 0/2260 flags hashpspool,nearfull max_bytes 19999515213824 stripe_width 0 application rbd
pool 21 'dbs-realtime-staging-client' replicated size 3 min_size 1 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128 last_change 8931 flags hashpspool,nearfull max_bytes 9999757606912 stripe_width 0 application rbd
ceph osd crush rule dump
[
{
"rule_id": 0,
"rule_name": "replicated_hdd_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "hdd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host-hdd"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "replicated_ssd_rule",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -16,
"item_name": "ssd"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "host-ssd"
},
{
"op": "emit"
}
]
}
]
rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR
.rgw.root 5.43KiB 20 0 60 0 0 0 105486961 67.1GiB 44 31KiB
dbosql-realtime-staging 63.5GiB 17260 0 51780 0 0 0 4385039 214GiB 24517805 641GiB
dbs-realtime-staging-client 11.6GiB 2973 2 8919 0 0 0 309779150 7.75TiB 141481908 11.5TiB
dbs-realtime-staging-w-financedb 18.1TiB 17216113 13519214 51648339 0 0 0 498012004 14.5TiB 621641675 10.2TiB
default.rgw.buckets.data 5.94TiB 1701672 0 5105016 0 0 0 385105812 22.6TiB 14857535 9.27TiB
default.rgw.buckets.index 0B 47100 0 141300 0 0 0 9358464185 8.72TiB 1076303 0B
default.rgw.buckets.non-ec 0B 20 0 60 0 0 0 4972496 3.26GiB 3065667 0B
default.rgw.control 0B 8 0 24 0 0 0 0 0B 0 0B
default.rgw.log 178B 205 0 615 0 0 0 116379863 1.18TiB 83680335 19.1MiB
default.rgw.meta 39.2KiB 188 0 564 0 0 0 677531397 540GiB 626911 612KiB
k8s-chatbot 0B 0 0 0 0 0 0 0 0B 0 0B
k8s-infraops 200GiB 51860 0 155580 0 0 0 1256335 126GiB 19012858 944GiB
k8s-mobile-devops 64.3GiB 16886 0 50658 0 0 0 977773 81.8GiB 32660137 1.12TiB
perftest 3.80GiB 984 0 2952 0 0 0 4743574373 17.7TiB 10035604710 44.4TiB
total_objects 19055289
total_used 77.4TiB
total_avail 276TiB
total_space 354TiB
ceph df detail
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
354TiB 276TiB 77.4TiB 21.88 19.05M
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED
.rgw.root 1 N/A N/A 5.43KiB 0 7.18TiB 20 20 101MiB 44B 16.3KiB
default.rgw.control 2 N/A N/A 0B 0 7.18TiB 8 8 0B 0B 0B
default.rgw.meta 3 N/A N/A 39.2KiB 0 7.18TiB 188 188 646MiB 612KiB 118KiB
default.rgw.log 4 N/A N/A 178B 0 7.18TiB 205 205 111MiB 79.8MiB 534B
default.rgw.buckets.index 5 N/A N/A 0B 0 7.18TiB 47100 47.10k 8.72GiB 1.03MiB 0B
default.rgw.buckets.data 6 N/A N/A 5.94TiB 8.48 64.1TiB 1701778 1.70M 367MiB 14.2MiB 17.8TiB
k8s-chatbot 7 N/A 10GiB 0B 0 64.1TiB 0 0 0B 0B 0B
k8s-infraops 8 N/A 210GiB 200GiB 0.30 64.1TiB 51860 51.86k 1.20MiB 18.1MiB 600GiB
k8s-mobile-devops 9 N/A 200GiB 64.3GiB 0.10 64.1TiB 16886 16.89k 955KiB 31.1MiB 193GiB
default.rgw.buckets.non-ec 12 N/A N/A 0B 0 64.1TiB 20 20 4.74MiB 2.92MiB 0B
perftest 15 N/A 4GiB 3.80GiB 0 64.1TiB 984 984 4.42GiB 9.35GiB 11.4GiB
dbosql-realtime-staging 19 N/A 931GiB 63.5GiB 0.86 7.18TiB 17260 17.26k 4.18MiB 23.4MiB 191GiB
dbs-realtime-staging-w-financedb 20 N/A 18.2TiB 18.1TiB 71.64 7.18TiB 17215222 17.22M 475MiB 593MiB 54.4TiB
dbs-realtime-staging-client 21 N/A 9.09TiB 11.6GiB 0.16 7.18TiB 2973 2.97k 295MiB 135MiB 34.7GiB