0 cumulative citations
View corpus contextA skill-filtering twist speeds up hierarchical RL: QDOS weights offline skill discovery by estimated advantage and enforces quality-diversity, producing markedly faster exploration and higher returns on complex simulated manipulation and locomotion benchmarks.
Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning
QDOS uses advantage-weighted quality-diversity objectives to extract diverse, high-value latent skills from mixed-quality offline datasets and reuses pseudo-labeled offline data to accelerate online hierarchical RL, improving sample efficiency and final returns on several sparse-reward simulated tasks.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Recent studies investigate how to leverage pre-collected datasets to improve the policy performance and sample efficiency of RL. One promising approach to achieve this goal is to employ a two-stage strategy: In the first stage, diverse skills are extracted as a low-level policy from a given dataset, and a high-level policy is trained to solve a specific task in the second stage. Typically, extraction of the low-level policy is performed based on unsupervised learning such as trajectory VAE. However, a limitation of this approach is that the quality of the low-level policy highly depends on the quality of the dataset. To address this issue, we introduce QDOS (Quality-Diversity Offline Skill learning), a unified pipeline for robust offline-to-online learning. Our approach incorporates an Advantage-Weighted Quality-Diversity pretraining objective, which weights the skill extraction and diversity objectives by the estimated advantage of each trajectory segment. This approach allows the model to extract diverse and high-value skills. By providing robust and task-relevant skill representations, QDOS significantly improves the quality of the embedded skill space used by the low-level policy. We further integrate this with a dual dataset reuse strategy, where offline data is used both for skill pretraining and for populating the online replay buffer via pseudo-labeling. Experiments demonstrate that QDOS significantly outperforms strong baselines in structured manipulation tasks and unstructured locomotion tasks, confirming its ability to accelerate exploration and improve final returns in challenging sparse-reward domains.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic empirical evidence on multiple challenging simulated benchmarks (antmaze, humanoidmaze, kitchen, antsoccer), compares against several strong baselines, and reports averages over random seeds; however, evidence is limited to simulated environments, relies on tuned hyperparameters, uses only 5 seeds, and lacks real-world/robotic validation or formal guarantees, reducing external credibility.
Methods Rigormedium — The experimental design uses well-known baselines (SUPE, IQL, SAC, etc.), a clear pretraining/online pipeline, and multiple environments; key components (IQL-based advantage estimation, weighted VAE objective, mutual-information regularization, pseudo-labeling) are explicitly described. Weaknesses include limited reporting on hyperparameter sensitivity, reliance on optimistic pseudo-reward heuristics, modest number of random seeds, and absence of real-world experiments or theoretical identification of when advantage-weighting will or will not help.
SampleMixed-quality offline trajectory datasets collected in simulated control environments (antmaze, antsoccer, humanoidmaze, kitchen), containing good moves, failed attempts, and noise; trajectories are segmented into fixed-length segments (H=20); offline pretraining uses these datasets to learn latent skills; online finetuning uses pseudo-labeled offline segments plus online transitions; experiments averaged over 5 random seeds.
Themesskills_training innovation
GeneralizabilityEvaluated only in simulated benchmarks; real-world robotics performance is untested., Performance depends on the composition and quality of the offline dataset (unclear robustness across different dataset mixes)., Hyperparameters (e.g., temperature λ, mutual-information weight α, expectile τ) tuned per environment may limit out-of-the-box transfer., Pseudo-labeling and optimistic reward estimation (RND/UCB) may bias learning in ways that depend on environment dynamics and reward sparsity., Results reported over 5 seeds — limited assessment of variability and statistical significance across tasks.
Incomplete processing:
Summary missing.