A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning

Autores
Barsce, Juan Cruz; Palombarini, Jorge Andrés; Martínez, Ernesto Carlos
Año de publicación
2020
Idioma
inglés
Tipo de recurso
artículo
Estado
versión publicada
Descripción
Optimization of hyper-parameters in real-world applications of reinforcement learning (RL) is a key issue, because their settings determine how fast the agent will learn its policy by interacting with its environment due to the information content of data gathered. In this work, an approach that uses Bayesian optimization to perform an autonomous two-tier optimization of both representation decisions and algorithm hyper-parameters is proposed: first, categorical / structural RL hyper-parameters are taken as binary variables and optimized with an acquisition function tailored for such type of variables. Then, at a lower level of abstraction, solution-level hyper-parameters are optimized by resorting to the expected improvement acquisition function, whereas the categorical hyper-parameters found in the optimization at the upperlevelof abstraction are fixed. This two-tier approach is validated with a tabular and neural network setting of the value function, in a classic simulated control task. Results obtained are promising and open the way for more user-independent applications of reinforcement learning.
Fil: Barsce, Juan Cruz. Universidad Tecnologica Nacional. Facultad Regional Villa Maria; Argentina
Fil: Palombarini, Jorge Andrés. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Centro de Investigaciones y Transferencia de Villa María. Universidad Nacional de Villa María. Centro de Investigaciones y Transferencia de Villa María; Argentina. Universidad Tecnologica Nacional. Facultad Regional Villa Maria; Argentina
Fil: Martínez, Ernesto Carlos. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Santa Fe. Instituto de Desarrollo y Diseño. Universidad Tecnológica Nacional. Facultad Regional Santa Fe. Instituto de Desarrollo y Diseño; Argentina
Materia
REINFORCEMENT LEARNING
BAYESIAN OPTIMIZATION
AUTONOMOUS SYSTEMS
HYPER-PARAMETERS TUNING
Nivel de accesibilidad
acceso abierto
Condiciones de uso
https://creativecommons.org/licenses/by-nc/2.5/ar/
Repositorio
CONICET Digital (CONICET)
Institución
Consejo Nacional de Investigaciones Científicas y Técnicas
OAI Identificador
oai:ri.conicet.gov.ar:11336/113002
Acceder
id CONICETDig_4942662d434527f2f7efdd8296e8f3ab
oai_identifier_str oai:ri.conicet.gov.ar:11336/113002
network_acronym_str CONICETDig
repository_id_str 3498
network_name_str CONICET Digital (CONICET)
spelling A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learningBarsce, Juan CruzPalombarini, Jorge AndrésMartínez, Ernesto CarlosREINFORCEMENT LEARNINGBAYESIAN OPTIMIZATIONAUTONOMOUS SYSTEMSHYPER-PARAMETERS TUNINGhttps://purl.org/becyt/ford/2.2https://purl.org/becyt/ford/2Optimization of hyper-parameters in real-world applications of reinforcement learning (RL) is a key issue, because their settings determine how fast the agent will learn its policy by interacting with its environment due to the information content of data gathered. In this work, an approach that uses Bayesian optimization to perform an autonomous two-tier optimization of both representation decisions and algorithm hyper-parameters is proposed: first, categorical / structural RL hyper-parameters are taken as binary variables and optimized with an acquisition function tailored for such type of variables. Then, at a lower level of abstraction, solution-level hyper-parameters are optimized by resorting to the expected improvement acquisition function, whereas the categorical hyper-parameters found in the optimization at the upperlevelof abstraction are fixed. This two-tier approach is validated with a tabular and neural network setting of the value function, in a classic simulated control task. Results obtained are promising and open the way for more user-independent applications of reinforcement learning.Fil: Barsce, Juan Cruz. Universidad Tecnologica Nacional. Facultad Regional Villa Maria; ArgentinaFil: Palombarini, Jorge Andrés. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Centro de Investigaciones y Transferencia de Villa María. Universidad Nacional de Villa María. Centro de Investigaciones y Transferencia de Villa María; Argentina. Universidad Tecnologica Nacional. Facultad Regional Villa Maria; ArgentinaFil: Martínez, Ernesto Carlos. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Santa Fe. Instituto de Desarrollo y Diseño. Universidad Tecnológica Nacional. Facultad Regional Santa Fe. Instituto de Desarrollo y Diseño; ArgentinaSociedad Argentina de Informática E Investigación Operativa2020-05-19info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionhttp://purl.org/coar/resource_type/c_6501info:ar-repo/semantics/articuloapplication/pdfapplication/pdfhttp://hdl.handle.net/11336/113002Barsce, Juan Cruz; Palombarini, Jorge Andrés; Martínez, Ernesto Carlos; A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning; Sociedad Argentina de Informática E Investigación Operativa; SADIO Electronic Journal of Informatic and Operation Research; 19; 2; 19-5-2020; 2-271514-6774CONICET DigitalCONICETenginfo:eu-repo/semantics/altIdentifier/url/https://publicaciones.sadio.org.ar/index.php/EJS/article/view/165info:eu-repo/semantics/openAccesshttps://creativecommons.org/licenses/by-nc/2.5/ar/reponame:CONICET Digital (CONICET)instname:Consejo Nacional de Investigaciones Científicas y Técnicas2025-10-15T15:38:33Zoai:ri.conicet.gov.ar:11336/113002instacron:CONICETInstitucionalhttp://ri.conicet.gov.ar/Organismo científico-tecnológicoNo correspondehttp://ri.conicet.gov.ar/oai/requestdasensio@conicet.gov.ar; lcarlino@conicet.gov.arArgentinaNo correspondeNo correspondeNo correspondeopendoar:34982025-10-15 15:38:33.582CONICET Digital (CONICET) - Consejo Nacional de Investigaciones Científicas y Técnicasfalse
dc.title.none.fl_str_mv A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning
title A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning
spellingShingle A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning
Barsce, Juan Cruz
REINFORCEMENT LEARNING
BAYESIAN OPTIMIZATION
AUTONOMOUS SYSTEMS
HYPER-PARAMETERS TUNING
title_short A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning
title_full A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning
title_fullStr A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning
title_full_unstemmed A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning
title_sort A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning
dc.creator.none.fl_str_mv Barsce, Juan Cruz
Palombarini, Jorge Andrés
Martínez, Ernesto Carlos
author Barsce, Juan Cruz
author_facet Barsce, Juan Cruz
Palombarini, Jorge Andrés
Martínez, Ernesto Carlos
author_role author
author2 Palombarini, Jorge Andrés
Martínez, Ernesto Carlos
author2_role author
author
dc.subject.none.fl_str_mv REINFORCEMENT LEARNING
BAYESIAN OPTIMIZATION
AUTONOMOUS SYSTEMS
HYPER-PARAMETERS TUNING
topic REINFORCEMENT LEARNING
BAYESIAN OPTIMIZATION
AUTONOMOUS SYSTEMS
HYPER-PARAMETERS TUNING
purl_subject.fl_str_mv https://purl.org/becyt/ford/2.2
https://purl.org/becyt/ford/2
dc.description.none.fl_txt_mv Optimization of hyper-parameters in real-world applications of reinforcement learning (RL) is a key issue, because their settings determine how fast the agent will learn its policy by interacting with its environment due to the information content of data gathered. In this work, an approach that uses Bayesian optimization to perform an autonomous two-tier optimization of both representation decisions and algorithm hyper-parameters is proposed: first, categorical / structural RL hyper-parameters are taken as binary variables and optimized with an acquisition function tailored for such type of variables. Then, at a lower level of abstraction, solution-level hyper-parameters are optimized by resorting to the expected improvement acquisition function, whereas the categorical hyper-parameters found in the optimization at the upperlevelof abstraction are fixed. This two-tier approach is validated with a tabular and neural network setting of the value function, in a classic simulated control task. Results obtained are promising and open the way for more user-independent applications of reinforcement learning.
Fil: Barsce, Juan Cruz. Universidad Tecnologica Nacional. Facultad Regional Villa Maria; Argentina
Fil: Palombarini, Jorge Andrés. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Córdoba. Centro de Investigaciones y Transferencia de Villa María. Universidad Nacional de Villa María. Centro de Investigaciones y Transferencia de Villa María; Argentina. Universidad Tecnologica Nacional. Facultad Regional Villa Maria; Argentina
Fil: Martínez, Ernesto Carlos. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Santa Fe. Instituto de Desarrollo y Diseño. Universidad Tecnológica Nacional. Facultad Regional Santa Fe. Instituto de Desarrollo y Diseño; Argentina
description Optimization of hyper-parameters in real-world applications of reinforcement learning (RL) is a key issue, because their settings determine how fast the agent will learn its policy by interacting with its environment due to the information content of data gathered. In this work, an approach that uses Bayesian optimization to perform an autonomous two-tier optimization of both representation decisions and algorithm hyper-parameters is proposed: first, categorical / structural RL hyper-parameters are taken as binary variables and optimized with an acquisition function tailored for such type of variables. Then, at a lower level of abstraction, solution-level hyper-parameters are optimized by resorting to the expected improvement acquisition function, whereas the categorical hyper-parameters found in the optimization at the upperlevelof abstraction are fixed. This two-tier approach is validated with a tabular and neural network setting of the value function, in a classic simulated control task. Results obtained are promising and open the way for more user-independent applications of reinforcement learning.
publishDate 2020
dc.date.none.fl_str_mv 2020-05-19
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
http://purl.org/coar/resource_type/c_6501
info:ar-repo/semantics/articulo
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv http://hdl.handle.net/11336/113002
Barsce, Juan Cruz; Palombarini, Jorge Andrés; Martínez, Ernesto Carlos; A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning; Sociedad Argentina de Informática E Investigación Operativa; SADIO Electronic Journal of Informatic and Operation Research; 19; 2; 19-5-2020; 2-27
1514-6774
CONICET Digital
CONICET
url http://hdl.handle.net/11336/113002
identifier_str_mv Barsce, Juan Cruz; Palombarini, Jorge Andrés; Martínez, Ernesto Carlos; A hierarchical two-tier approach to hyper-parameter optimization in reinforcement learning; Sociedad Argentina de Informática E Investigación Operativa; SADIO Electronic Journal of Informatic and Operation Research; 19; 2; 19-5-2020; 2-27
1514-6774
CONICET Digital
CONICET
dc.language.none.fl_str_mv eng
language eng
dc.relation.none.fl_str_mv info:eu-repo/semantics/altIdentifier/url/https://publicaciones.sadio.org.ar/index.php/EJS/article/view/165
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
https://creativecommons.org/licenses/by-nc/2.5/ar/
eu_rights_str_mv openAccess
rights_invalid_str_mv https://creativecommons.org/licenses/by-nc/2.5/ar/
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv Sociedad Argentina de Informática E Investigación Operativa
publisher.none.fl_str_mv Sociedad Argentina de Informática E Investigación Operativa
dc.source.none.fl_str_mv reponame:CONICET Digital (CONICET)
instname:Consejo Nacional de Investigaciones Científicas y Técnicas
reponame_str CONICET Digital (CONICET)
collection CONICET Digital (CONICET)
instname_str Consejo Nacional de Investigaciones Científicas y Técnicas
repository.name.fl_str_mv CONICET Digital (CONICET) - Consejo Nacional de Investigaciones Científicas y Técnicas
repository.mail.fl_str_mv dasensio@conicet.gov.ar; lcarlino@conicet.gov.ar
_version_ 1846083504037691392
score 12.891075

Publicaciones similares