RisingWave fails to bulk delete objects from Minio - Unexpected (permanent) at batch, context: { service: s3 } => deserialize xml, source: missing field Key

[pjpringlenom]

Describe the bug

Since 18 January compactor has been erroring with source: missing field Key

As a I result I now have several Tb of unnecessary objects in the store.

Error message/log

Compactor logs have repeating errors of this pattern

2025-02-07T15:45:25+08:00	2025-02-07T07:45:25.122965725+00:00 stdout F 2025-02-07T07:45:25.122437813Z  INFO risingwave_storage::hummock::vacuum: Try to vacuum SSTs [39379057, 34630848, 35208955, 38413716, 33940203, 34006595, 38313058, 33146390, 39008136, 36988587, 40149782, 41369309, 33047958, 38659262, 39865468, 32773140, 33111348, 35383498, 38032132, 41132357, 39371814, 37226249, 34051258, 40012903, 34126281, 35192924, 34091272, 32485643, 35978046, 31756049, 41203778, 31827796, 37525321, 39051200, 38112854, 39882827, 40521052, 38890727, 41711660, 38740632, 32563259, 36596446, 37059066, 40663773, 36642034, 39775610, 37069737, 32712605, 33466095, 36458694, 31867924, 39302606, 33193460, 33426526, 40954445, 37417431, 32476847, 39730144, 33096754, 41573834, 37739612, 36635373, 34136971, 37632238, 35260267, 34045635, 36906241, 35652751, 32354421, 32930456, 37625331, 38657225, 35287774, 40811687, 37154588, 36788855, 36852568, 36407060, 36276526, 36370681, 35655620, 32837424, 37711046, 41378132, 39181880, 34669016, 33317172, 34429514, 33655366, 34741515, 32658081, 37682076, 39908957, 39635283, 32182722, 32702285, 41080261, 37226866, 37784429, 33357317, 41068304, 39366078, 37222199, 32572800, 35260068, 36717685, 32979356, 36643327, 32799917, 32193285, 32999152, 32464997, 37659220, 37925972, 41338980, 41433042, 32473909, 35619601, 39638675, 36383997, 36833083, 40284752, 36247902, 32079725, 39555165, 39658166, 37841033, 35968583, 41013049, 37067275, 35043184, 36419335, 33636507, 38345389, 35274351, 34064336, 39968045, 39924989, 40121497, 31725180, 32091169, 37804028, 34006099, 34006944, 32953941, 37851879, 35274309, 32540890, 41155367, 37835127, 37631629, 33395669, 40971276, 31751190, 37213138, 32342569, 40410481, 38625295, 36109421, 38342163, 40814634, 37090902, 33933645, 39303694, 41605920, 38698870, 40025904, 36997748, 33447812, 37291767, 36528052, 40810069, 39014836, 38065312, 37129435, 40708628, 40145827, 38397397, 40124122, 33234657, 36408204, 35405068, 40020599, 40130288, 41327382, 33672673, 33165129, 38258630, 38930134, 40244174, 37940256, 33405170, 38768629, 33215428, 33734444, 38812623, 38759178, 35425410, 41701741, 38594425, 39542002, 31854682, 37832645, 36576025, 35631668, 35350522, 41332625, 39491243, 32343451, 37127252, 39934590, 39037715, 33346851, 36468887, 36782147, 41186328, 40538596, 35309240, 41405251, 34204145, 40251590, 40782228, 38313581, 35962452, 35538691, 36887608, 39332748, 31949489, 38961203, 32433266, 38627764, 40238791, 38753511, 33872878, 33313415, 33350142, 33220559, 38304136, 39459602, 37188013, 34692110, 36441215, 38552249, 33844324, 38085693, 31967043, 41230315, 38772705, 40112941, 34281935, 32539285, 35917125, 33853969, 39204602, 39962113, 32744102, 41134148, 36393307, 37293059, 38684228, 37587168, 39262911, 32756442, 39193504, 36041947, 41199974, 36007200, 35260137, 37005894, 32195961, 32368008, 41745828, 34004795, 33586091, 38902978, 33996551, 36967832, 34571368, 34654357, 41434136, 35627738, 37413604, 36991891, 32334947, 37447893, 34357631, 37892099, 37576072, 36097566, 37742812, 32562471, 32242547, 35505775, 34366380, 36860901, 40334261, 39518418, 38696324, 39619337, 32223229, 36813442, 35571250, 35386105, 38522562, 41285003, 34874640, 33208300, 35826616, 32436799, 39152908, 38330937, 40292352, 36954585, 34636581, 32074589, 41004168, 35394990, 34076874, 38136380, 38344553, 35716919, 31736368, 37725457, 31762400, 40156009, 35193298, 31893164, 37301430, 39196671, 39394391, 39354791, 33889469, 34030469, 37759211, 32503615, 33128263, 41256321, 41554291, 32831029, 35573910, 38298840, 37180049, 36057421, 33007911, 32228103, 38384326, 36176048, 35689904, 32231354, 35535857, 31775167, 34469316, 38431008, 32708459, 35719941, 33827731, 35323081, 35242499, 33868231, 39649151, 41060765, 41059051, 38093554, 32539903, 39268990, 34199033, 40826421, 34467880, 37913431, 35867036, 31949685, 37651103, 38314340, 32226470, 36257749, 35146128, 39621982, 33060213, 32728878, 34855811, 35901220, 36275976, 38168743, 33151925, 33030633, 37965072, 38577586, 33702969, 32205754, 40053543, 41658750, 36357130, 36226142, 36294470, 32554577, 36660745, 37267432, 40696625, 35597196, 38189569, 34286184, 37320977, 37652214, 39074159, 33943269, 33484069, 41205746, 38934791, 33976870, 36504099, 38792713, 38258074, 32793202, 37612293, 31745606, 34382852, 34929823, 38962225, 33417587, 38633319, 34420514, 36836427, 32142267, 31766185, 35680124, 35824714, 41642357, 40538224, 37041916, 35374642, 34622565, 38993109, 32975755, 38952278, 37224606, 32378573, 36720263, 36888786, 39010494, 31735912, 35549309, 32814613, 37838516, 36946031, 37695949, 34109771, 40298191, 35291882, 41402717, 34345214, 34157731, 34639823, 34129037, 34589870, 40191091, 37004470, 34771229, 32507439, 37744849, 40216659, 33546284, 39711547, 37400929, 37465686, 33483286, 32871195, 38044309, 41680635, 32493751, 38896356, 38889388, 37982656, 38168686, 34278682, 38837994, 32997345, 34880703, 37218476, 40006843, 41606904, 41593806, 34655375, 41403785, 38464576, 32695359, 37398136, 34758597, 39504492, 39528273, 39931904, 36796982, 33185016, 39661974, 36841370, 40723458, 38212268, 37478470, 37322824, 35172661, 41431575]
2025-02-07T15:45:25+08:00	2025-02-07T07:45:25.324960576+00:00 stdout F 2025-02-07T07:45:25.324343646Z  INFO risingwave_storage::hummock::vacuum: Try to vacuum SSTs [35862071, 35148735, 40034225, 39563093, 36578848, 32887487, 34214791, 41666286, 34037949, 32740112, 34036313, 36814244, 40651240, 37778645, 36259963, 41433701, 39785782, 40606685, 40294140, 34625207, 40497684, 35928790, 39762670, 37617812, 41359681, 41140393, 34727587, 37186292, 36745868, 35179564, 37508075, 38967441, 40119959, 36561600, 39226328, 39847379, 38567136, 34296501, 33642602, 35533870, 40209554, 35151638, 38699037, 34032193, 41067703, 37923315, 36357375, 40130359, 38077369, 33393815, 38126371, 40994327, 41500776, 39907023, 40252157, 32905678, 40263997, 41063038, 38195552, 40844575, 32275545, 36253360, 40664205, 34083260, 39777074, 35812355, 40653363, 37067216, 37228305, 32429413, 33078966, 38639423, 37006161, 35026913, 37869982, 38576300, 36705047, 39599162, 38713263, 34068525, 40886351, 34224598, 31729346, 39638363, 35657088, 37157857, 31739849, 36452695, 38915930, 40674010, 38075738, 39008078, 38254643, 39205408, 40875416, 39571203, 34127538, 37082540, 40193658, 40987738, 33960290, 32789254, 40445112, 36153675, 31836523, 31727225, 41254733, 38971477, 38440663, 39719893, 34531963, 33447694, 36028817, 34443280, 37757777, 39895233, 39134921, 34903829, 38115929, 40483054, 34115535, 41097489, 38311741, 34731258, 37701251, 40697671, 41680186, 37918439, 34459914, 40775738, 41396128, 32870447, 32041783, 32556398, 38485399, 39349758, 41727300, 32098005, 32877259, 36607760, 39686308, 32267032, 39534448, 41715746, 35814085, 38505259, 38303134, 40824028, 40038687, 35900231, 35828663, 39963162, 40018209, 34873307, 39894683, 35340741, 40044350, 33468454, 34453819, 40639612, 37603049, 37273549, 38867567, 32648435, 32094275, 38144376, 34342026, 32776465, 35365703, 38406669, 36286589, 32705885, 37011652, 38912537, 41278475, 35494968, 38837589, 36154440, 39842166, 39373419, 39295331, 39639597, 31787035, 37437776, 33360935, 33001297, 37042228, 40995245, 34116145, 38297577, 39133461, 39716107, 40458658, 37279931, 39671490, 34641858, 32417135, 34511996, 40539334, 35707173, 35966421, 39092516, 38440072, 41294694, 37805943, 36308902, 34399424, 33582185, 38052463, 35494394, 37918884, 32219014, 35628668, 35489465, 37370993, 32778366, 36229381, 36602669, 33011713, 33692520, 34257323, 33771578, 33967205, 35219319, 34327903, 36585051, 37283533, 41722053, 35561675, 37741252, 38334186, 34917981, 33568257, 35457157, 37061015, 41357147, 31807583, 34225334, 37826907, 40798818, 39881643, 40926873, 41451070, 41430444, 32730445, 36197124, 38303772, 40203095, 36499683, 32178569, 35728591, 37034481, 32352787, 35389828, 34119086, 39363504, 34586268, 32284456, 37568066, 38745973, 33036336, 32029697, 41715555, 36608855, 37765084, 33225277, 32890098, 32447506, 40807609, 33676627, 36425678, 33402768, 39779322, 34853311, 32198297, 34234937, 36087964, 37492147, 36904225, 40678353, 40931142, 41610901, 37245767, 36919667, 35935684, 40480492, 36836529, 39088554, 38823846, 32139580, 41514194, 33271466, 38195815, 35140501, 39430185, 39603895, 38051784, 38369288, 33921163, 36637148, 38578564, 38971684, 39024652, 32297369, 32996794, 38572069, 36156958, 41650724, 34447782, 33388942, 36580925, 41338830, 39522513, 35194680, 40225715, 31907045, 34262557, 34839916, 38319659, 40522957, 32901030, 37558065, 39791310, 39225097, 33335831, 32659443, 34997030, 37768279, 41722812, 40636353, 33295622, 34898145, 39506988, 34151242, 37677745, 34041384, 39905342, 34863857, 36229945, 35350176, 34345479, 39338327, 32087737, 37415542, 36735970, 34930337, 34993675, 35809142, 33997274, 32438717, 38188718, 35154593, 40690787, 35022874, 31945580, 32457482, 39225818, 39937239, 39917993, 41640395, 32281481, 33062789, 34280263, 38656633, 35926475, 33181878, 37898645, 40886954, 39094502, 37897644, 39942351, 34116316, 39257907, 37757123, 38127655, 31986483, 37423205, 36681576, 40795172, 34745914, 37540153, 36388096, 31763489, 34347760, 32026512, 31747872, 39624601, 38157543, 41737655, 40858762, 40733990, 34404359, 41254735, 39729096, 38562654, 31746236, 36926853, 36840918, 37246170, 41120178, 41383243, 34024746, 34598632, 38361768, 33050346, 37155597, 40552850, 39183468, 39066047, 33283177, 36021766, 35529079, 40020467, 40937800, 32171292, 41331247, 31852349, 39550698, 33942402, 40485873, 41458176, 35605262, 39168710, 38127925, 39021610, 39908521, 36135647, 32920715, 34078199, 36719461, 35125256, 35660505, 32449303, 39503093, 41301541, 31879302, 41568368, 39789483, 33342294, 39400304, 33079608, 35218520, 38397889, 37144786, 33813645, 39575060, 39714878, 34992626, 32798837, 35027028, 32277128, 36223655, 41622122, 41641777, 33253840, 32009274, 37426829, 37973536, 32477395, 41462446, 40538269, 38622739, 38891918, 35450890, 40073091, 34301559, 39241408, 32327424, 35069825, 40193471, 34248213, 38855902, 35965818, 41297696, 33889843, 31734793, 34663149, 41669788, 40761179, 31747030, 35377163, 32647586, 35436553, 41327563, 35181908, 36938009, 36183983, 34547059, 34942690, 33958789, 32825217, 33300740, 39229885, 38835846, 34424082, 37950911, 40817560, 34678074, 41381154, 40333180]
2025-02-07T15:45:25+08:00	2025-02-07T07:45:25.384314984+00:00 stdout F 2025-02-07T07:45:25.384233652Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2025-02-07T15:45:25+08:00	2025-02-07T07:45:25.614378740+00:00 stdout F 2025-02-07T07:45:25.614264729Z ERROR opendal::services: service=s3 operation=batch-delete count=500 -> Unexpected (permanent) at batch, context: { service: s3 } => deserialize xml, source: missing field `Key`    
2025-02-07T15:45:25+08:00	2025-02-07T07:45:25.614378740+00:00 stdout F 2025-02-07T07:45:25.614306858Z ERROR risingwave_object_store::object: delete_objects failed error=Unexpected (permanent) at batch, context: { service: s3 } => deserialize xml, source: missing field `Key`
2025-02-07T15:45:25+08:00	2025-02-07T07:45:25.614378740+00:00 stdout F 2025-02-07T07:45:25.61433338Z  WARN risingwave_storage::hummock::compactor: Failed to vacuum task error=ObjectStore failed with IO error: Unexpected (permanent) at batch, context: { service: s3 } => deserialize xml, source: missing field `Key`
2025-02-07T15:45:25+08:00	2025-02-07T07:45:25.725965424+00:00 stdout F 2025-02-07T07:45:25.725610556Z  INFO risingwave_storage::hummock::vacuum: Try to vacuum SSTs [38613992, 33319305, 39520829, 38519514, 32446353, 34003144, 35576756, 32175569, 34721280, 38223933, 31846008, 34814626, 38796905, 40482978, 34727146, 39697005, 32418715, 34029854, 33551105, 40610728, 37691562, 39888688, 35946193, 32966187, 39134731, 35947535, 39512837, 39332191, 39167330, 35128820, 36713547, 38335166, 38175512, 41654542, 41058395, 36192772, 36591557, 40174222, 38627610, 41719718, 37809934, 34534492, 39200786, 36355656, 34246741, 37463046, 39706473, 33145425, 36361033, 39985339, 34303485, 32033955, 34060466, 39658653, 39426064, 34398948, 33559077, 38051442, 36878200, 38746526, 32278575, 36829015, 35448383, 31935380, 41690223, 37295648, 39200486, 36955676, 36097130, 32065726, 31868609, 37805569, 40350198, 36683739, 36293442, 37670287, 36554530, 39431204, 32657606, 33930220, 40048338, 36477700, 34827137, 38119147, 36628227, 36652926, 37028104, 36169487, 35085302, 32932593, 35349909, 31772219, 32220348, 34585871, 34822226, 32230252, 37954070, 36938007, 36076618, 34138357, 39201476, 37499396, 38667232, 37590734, 33689427, 36952127, 41619904, 39684912, 36091886, 41616872, 40648058, 39778114, 32417610, 35857895, 36551627, 39619819, 36022047, 39042100, 36732971, 37552489, 32725204, 37583635, 34175050, 40593357, 35725348, 41504538, 37699827, 41495292, 41286303, 36097143, 38276394, 37315733, 39863672, 39362154, 33735198, 31898312, 38246018, 39514254, 40657706, 37227851, 31835658, 32447195, 38721596, 31931302, 35576530, 31878511, 36843165, 34054149, 39316354, 32118532, 37133721, 37399230, 37023963, 34768664, 37646754, 40528163, 35748181, 34047256, 33691832, 36617588, 41747855, 38838574, 36239308, 36003160, 36807521, 36315432, 32227683, 40867085, 40925159, 32694994, 33044050, 38702180, 37262439, 41221908, 36121594, 34629732, 36928109, 33048364, 41108708, 41743474, 36082760, 35233163, 40074682, 36745734, 36000333, 37925295, 37652212, 37211518, 32153875, 34826764, 39142653, 32396415, 37210795, 35095416, 34775716, 35474948, 35411778, 34907314, 40736679, 40262233, 34537684, 40096239, 33747928, 32098488, 37731706, 36558185, 38064676, 40683772, 37496346, 36567722, 34196766, 38388188, 34131609, 38560927, 34951876, 35800530, 39891808, 31958493, 32493024, 39606365, 34316121, 34898669, 41026134, 36286664, 36834497, 39308106, 40448603, 33114484, 37171571, 41518548, 35992273, 41367445, 34519481, 32805683, 32598210, 41369298, 36353086, 40561271, 37316494, 35060817, 37667076, 38843357, 39158597, 40790800, 38384383, 32981646, 32801952, 39284217, 40824146, 39515336, 36476952, 34792403, 40348818, 36202067, 37318698, 34964253, 34596028, 36910350, 35071727, 38866744, 40856624, 40192020, 39704170, 33052339, 32213909, 37410928, 39032795, 41737971, 34182814, 40778426, 39525469, 32765766, 39785166, 38223232, 36985046, 40954673, 40668603, 40588187, 33552896, 40482937, 31961807, 41520076, 31871601, 41303501, 34723771, 34250056, 35196721, 36698431, 40274586, 33995502, 32233512, 37573392, 36267106, 32290631, 38649139, 37864482, 40536083, 41592475, 39171710, 40501117, 33861675, 32983537, 38679782, 39034354, 39121057, 41048231, 33958642, 34798235, 35792896, 36371348, 39442346, 32749776, 33999367, 32490449, 41710764, 33352781, 32399956, 41277673, 37225873, 33429718, 33186946, 33631555, 34093793, 38633347, 41149018, 34085380, 34198302, 34226275, 37063080, 35812680, 34033473, 33477644, 38120171, 37463121, 37677853, 35841251, 38158159, 34258887, 32653381, 34001565, 38248850, 36112559, 33294417, 34964193, 41363754, 33015227, 38194136, 41361897, 38727903, 39504976, 37709623, 34273681, 39539254, 34758996, 34591416, 36799264, 37899092, 40805067, 41185722, 36273006, 37965252, 38452328, 33191811, 32461064, 38271119, 37791692, 37888580, 34585134, 36485257, 32893635, 41467973, 37604841, 38173318, 31738771, 37562680, 32794202, 36343689, 32804018, 40052734, 39732938, 34729177, 35468887, 41503318, 35962391, 33816845, 36305411, 39836784, 35612018, 37914136, 34963387, 40199471, 33380734, 40960469, 35353130, 36689772, 36842907, 39170773, 37228036, 33148467, 34693457, 32870202, 38805253, 37717771, 32842414, 32508199, 33672459, 32212236, 39094965, 40026308, 38692510, 36353192, 40179718, 34924888, 36508134, 40025117, 33307789, 39443676, 40987094, 35082219, 37317454, 34771340, 35772887, 37641647, 36113645, 36595214, 41576847, 32047023, 39229025, 35812240, 35951726, 40907006, 34802677, 38528296, 33523744, 38322554, 40577865, 37588541, 31761578, 38378160, 35992999, 33790807, 33693732, 38875131, 37274988, 37699651, 41616140, 39274530, 32779892, 36502073, 36166510, 33455987, 38304799, 35847949, 34735680, 32926738, 33482187, 32761685, 38381322, 34792194, 38283890, 37055030, 36675714, 33963403, 41211011, 34829432, 37370201, 31992956, 37556501, 31888098, 31850720, 37504511, 40200943, 37233856, 38500281, 35210546, 39185869, 34287621, 39562939, 33269854, 31951344, 38842733, 39659954, 41139533, 36773298, 41389525, 41436733, 32181009, 38661599, 37476080, 32668062, 35550600, 38558538, 33573562, 33873980, 34460792, 38827590, 38693694, 38623011, 39002862, 35257369]
2025-02-07T15:45:26+08:00	2025-02-07T07:45:25.797210580+00:00 stdout F 2025-02-07T07:45:25.797081778Z ERROR opendal::services: service=s3 operation=batch-delete count=500 -> Unexpected (permanent) at batch, context: { service: s3 } => deserialize xml, source: missing field `Key`    
2025-02-07T15:45:26+08:00	2025-02-07T07:45:25.797210580+00:00 stdout F 2025-02-07T07:45:25.797117539Z ERROR risingwave_object_store::object: delete_objects failed error=Unexpected (permanent) at batch, context: { service: s3 } => deserialize xml, source: missing field `Key`
2025-02-07T15:45:26+08:00	2025-02-07T07:45:25.797210580+00:00 stdout F 2025-02-07T07:45:25.797170589Z  WARN risingwave_storage::hummock::compactor: Failed to vacuum task error=ObjectStore failed with IO error: Unexpected (permanent) at batch, context: { service: s3 } => deserialize xml, source: missing field `Key`

To Reproduce

Restart compactor and it will continue with this error.

Expected behavior

Expect files to be deleted from s3.

How did you deploy RisingWave?

Risingwave with docker compose
External Minio instance

The version of RisingWave

PostgreSQL 13.14.0-RisingWave-2.0.1 (0d15632)

Additional context

No response