Distributed storage framework design for various data reliability requirements

Yong Liu; Yingxun Fu

doi:10.1117/12.2641014

2 December 2022 Distributed storage framework design for various data reliability requirements

Yong Liu, Yingxun Fu

Proceedings Volume 12288, International Conference on Computer, Artificial Intelligence, and Control Engineering (CAICE 2022); 122880I (2022) https://doi.org/10.1117/12.2641014
Event: International Conference on Computer, Artificial Intelligence, and Control Engineering (CAICE 2022), 2022, Zhuhai, China

Abstract

Erasure Coding is a advanced redundancy technique to ensure data reliability of the distributed storage system(DSS). However, when data is first written into DSS, it may be hesitantly to choose the specific parameters of erasure coding to balance the repair cost and storage utilization. In this paper, we design and implement VDRR-EC, a new distributed storage framework which exploit deep q network (DQN) mind to automatically decide the parameters of erasure coding to optimize traditional distributed storage system performance. Experimental results in real cluster environment show that VDRR-EC reduces data repair frequency and storage cost by 10.7%~19.2% and approximately 3.5% respectively, the penalty is acceptable additional network transport overhead and computing overhead.

Citation Download Citation

Yong Liu and Yingxun Fu "Distributed storage framework design for various data reliability requirements", Proc. SPIE 12288, International Conference on Computer, Artificial Intelligence, and Control Engineering (CAICE 2022), 122880I (2 December 2022); https://doi.org/10.1117/12.2641014

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available