Legacy UIShift (v1)

⚠️ This README contains instructions for the legacy UIShift-7B version.
The updated GUI-Shift (v2) code and instructions will be released soon.

Data Preparation

We provide our 2K training samples for UIShift-7B ($k=1$) training here. These samples require no reasoning requirements for GRPO training. Since UI Transition is an annotation-free and easily scalable training task, we also provide the data processing workflow and corresponding scripts to support flexible scaling.

• AndroidControl preparation: Download the original AndroidControl dataset in TFRecord format and convert it into episode-wise folders. We provide a reference script: androidcontrol_parse.py.

• Metadata extraction: Run androidcontrol_metadata_extraction.py to extract screenshot metadata, including step instruction and groundtruth action. Each screenshot is identified by its episode_id and step index. The output will be saved to a CSV file. You can modify the script to exclude specific action types during training.

• UI bbox extraction: For click actions, UIShift assigns the correctness reward if the clicked point falls within the correct UI element. Since AndroidControl provides only the clicked point, we locate the smallest bounding box in the accessibility tree that contasin the point. See the reference script: androidcontrol_ui_bbox_extraction.py.

• Training set conversion: Once the metadata is prepared, it can be converted into the appropriate json or jsonl format depending on the training framework. We use VLM-R1 for UIShift training, and provide a reference script: metadata_to_trainingset.py.

Training

• Follow the installation instructions in VLM-R1.

• Replace the original get_question_template() in the VLM-R1/src/open-r1-multimudal/src/open_r1/vlm_modules/qwen_module.py with the following implementation before starting GRPO training. This ensures that no reasoning prompts are appended during training.

  @staticmethod
  def get_question_template(task_type: str):
      match task_type:
          case "rec":
              return "{Question} First output the thinking process in <think> </think> tags and then output the final answer in <answer> </answer> tags. Output the final answer in JSON format."
          case "uishift_no_reasoning":
              return "{Question} Output the final answer in <answer> </answer> tags. Do not output any extra text."
          case _:
              return "{Question} First output the thinking process in <think> </think> tags and then output the final answer in <answer> </answer> tags."
  

• Copy grpo_uishift.py to VLM-R1/src/open-r1-multimudal/src/open_r1/trainer. This script implements the reward mechanism used during training.

• Copy run_grpo_uishift.sh to VLM-R1/src/open-r1-multimudal/run_scripts. This is the training launch script and contains the hyperparameters used in UIShift.

Evaluation

The test set for AndroidControl is available here, and the prompts used for evaluating both the Qwen2.5-VL baseline and UIShift are provided here.