FAQ.md

FAQ

I got a mujoco error (e.g. C error/python error in installation or running), what should I do?

Since mujoco requires C part (and rendering), such error may occur if your system misses packages. Please double check your packages and contact mujoco for help.

I see multiple options in gym to render. What should I use to render?

As long as it comes with GPU acceleration, it should be fast. EGL is used in the benchmarks and is recommended.

What are the --update_rewards and --use_inv in the example command?

These are legacy for compatability purpose. This codebase supports predicting the rewards and inverse dynamics as well as rotation as a side training task. These are useful for running policy adaptation during deployment (PAD experiments). This is enabled in training for compatability in loading our previously-trained checkpoints. This is not enabled in inference since it adds to latency and we don't need adaptation in the current setting. We experimented and we discovered no significant change when removing the flag of reward prediction.

How do you benchmark video backgrounds?

We use the script above with video0 for seed 0, video1 for seed 1, etc. In the end, we take mean and std of all inference rewards sum.

Hints for troubleshooting?

RL + vision is a task which is inherently very unstable and difficult to solve and even different runs could lead to different results if the algorithm is not stable enough. If you see something is wrong, debug in the following sequence: if the rewards in training environment do not meet the expectation, then it is probably not a problem of domain or visual changes, and in this case, you could try reducing augmentation and starting to tune the RL parameters (You may add, remove, or change RL augmentations or clamp in encoder.py). If the rewards in training environment do meet expectation, but the rewards in test environment do not, then you should check the performance of adapter in testing env. If you have no problem in color hard but problems in video tasks (as they are very challenging), you can export the frames before and after attention module and visualize (this is very helpful) in your attention training hyperparameters (you can do it before training RL and it is really helpful to use a good adapter, see forward_conv in encoder.py as an example).

In addition, pixels of background leakage often happen and it is normal to have it without influencing the performance. Also, since the training environments used by works in this series are with little distractions (e.g. not much texture in the background and easy to separate), the quality of masks will satisfy the requirement of the attention module most of the time (also, leaking patches of background is not as harmful as missing the foreground, so use a lower threshold). What is difficult is to adapt in video scenarios as they come with confusions in terms of texture and color, and tuning the hyperparameters may be required, although augmentation based on natural images already provides a pretty good augmentation when compared to hand-designed augmentations.

For hyperparam tuning, we have given a set of hyperparams that perform well enough in many scenarios, and so in Step 1, since the training environments do not have much noise and the foreground and background are relative distinguishable given their different color, the hyperparameters are relatively stable across environments. Including background patches do not influence much of the quality of the trained adapter module in stage 2, but missing information will cause issues in RL. You can use provided "dilate" function to make the mask larger, if you find the masks to miss information, although it's not used in our final training to get foreground in stage 2. In addition, for Step 2, a common thing is that foreground patches are missing in videos. This is mode severe than including background patches. Adjusting weights for foreground could help because the pixels for positive/negative samples are imbalanced.

For DeepMind Control, although Places dataset is not required to get improvements on the color hard environment, Places dataset is recommended for the video environment since it involves complicated textures which make the adapter hard to distinguish. For DrawerWorld, since the texture is repeating (relatively uniform), the adapter works even without Places dataset. If you decide to use Places augmentation, you need to control the strength of it to avoid mis-classify patches. In addition, in the experience of training adapter, the DrawerWorld benchmark is not difficult in training environment and the training environment has simple foreground and background, so hyperparam "k" in Stage 1 does not matter much for DrawerWorld and algorithm can select foreground efficiently even if you over-specify "k". The benchmark is designed to be difficult in the test environment and they are very challenging in terms of foreground and background separation because the texture has never appeared to the model in training time, so Stage 2 needs a lot of attention.

Are Pretrained Models Available?

Yes, and please see the README.