multi-taskmodal.md

Multi-task/modal

• (arXiv 2021.02) Transformer is All You Need: Multimodal Multitask Learning with a Unified Transformer, [Paper], [Code]
• (arXiv 2021.04) MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding, [Paper], [Code]
• (arXiv 2021.04) Multi-Modal Fusion Transformer for End-to-End Autonomous Driving, [Paper]
• (arXiv 2021.04) VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text, [Paper]
• (arXiv 2021.04) Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning, [Paper]
• (arXiv 2021.06) Scene Transformer: A Unified Multi-task Model for Behavior Prediction and Planning, [Paper]
• (arXiv 2021.06) Spatio-Temporal Multi-Task Learning Transformer for Joint Moving Object Detection and Segmentation, [Paper]
• (arXiv 2021.06) A Transformer-based Cross-modal Fusion Model with Adversarial Training, [Paper]
• (arXiv 2021.07) Attention Bottlenecks for Multimodal Fusion, [Paper]
• (arXiv 2021.07) Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robots, [Paper]
• (arXiv 2021.07) Case Relation Transformer: A Crossmodal Language Generation Model for Fetching Instructions, [Paper]
• (arXiv 2021.07) Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers, [Paper], [Code]
• (arXiv 2021.08) StrucTexT: Structured Text Understanding with Multi-Modal Transformers, [Paper]
• (arXiv 2021.08) Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations, [Paper]
• (arXiv 2021.09) TxT: Crossmodal End-to-End Learning with Transformers, [Paper]
• (arXiv 2021.09) Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers, [Paper]
• (arXiv 2021.09) Temporal Pyramid Transformer with Multimodal Interaction for Video Question Answering, [Paper]
• (arXiv 2021.09) On Pursuit of Designing Multi-modal Transformer for Video Grounding, [Paper], [Code]
• (arXiv 2021.09) Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactions, [Paper]
• (arXiv 2021.09) KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation, [Paper]
• (arXiv 2021.10) Unifying Multimodal Transformer for Bi-directional Image and Text Generation, [Paper], [Code]
• (arXiv 2021.10) VLDeformer: Learning Visual-Semantic Embeddings by Vision-Language Transformer Decomposing, [Paper]
• (arXiv 2021.10) Detecting Dementia from Speech and Transcripts using Transformers, [Paper]
• (arXiv 2021.11) MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition, [Paper]
• (arXiv 2021.11) VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts, [Paper], [Code]
• (arXiv 2021.11) An Empirical Study of Training End-to-End Vision-and-Language Transformers, [Paper], [Code]
• (arXiv 2021.11) CLIP2TV: An Empirical Study on Transformer-based Methods for Video-Text Retrieval, [Paper]
• (arXiv 2021.11) Graph Relation Transformer: Incorporating pairwise object features into the Transformer architecture, [Paper], [Code1], [Code2]
• (arXiv 2021.11) UFO: A UniFied TransfOrmer for Vision-Language Representation Learning, [Paper]
• (arXiv 2021.11) Multi-modal Transformers Excel at Class-agnostic Object Detection, [Paper], [Code]
• (arXiv 2021.11) Sparse Fusion for Multimodal Transformers, [Paper]
• (arXiv 2021.11) VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling, [Paper], [Code]
• (arXiv 2021.11) Cerberus Transformer: Joint Semantic, Affordance and Attribute Parsing, [Paper], [Code]
• (arXiv 2021.11) PolyViT: Co-training Vision Transformers on Images, Videos and Audio, [Paper]
• (arXiv 2021.11) End-to-End Referring Video Object Segmentation with Multimodal Transformers, [Paper], [Code]
• (arXiv 2021.12) TransMEF: A Transformer-Based Multi-Exposure Image Fusion Framework using Self-Supervised Multi-Task Learning, [Paper], [Code]
• (arXiv 2021.12) LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences, [Paper]
• (arXiv 2021.12) Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation, [Paper]
• (arXiv 2021.12) VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modeling, [Paper]
• (arXiv 2021.12) VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks, [Paper],[Code]
• (arXiv 2021.12) Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text, [Paper]
• (arXiv 2021.12) Distilled Dual-Encoder Model for Vision-Language Understanding, [Paper],[Code]
• (arXiv 2021.12) Multimodal Personality Recognition using Cross-Attention Transformer and Behaviour Encoding, [Paper]
• (arXiv 2021.12) SLIP: Self-supervision meets Language-Image Pre-training, [Paper],[Code]
• (arXiv 2021.12) Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation, [Paper],[Code]
• (arXiv 2022.01) Robust Self-Supervised Audio-Visual Speech Recognition, [Paper],[Code]
• (arXiv 2022.01) Self-Training Vision Language BERTs with a Unified Conditional Model, [Paper]
• (arXiv 2022.01) Uniformer: Unified Transformer for Efficient Spatiotemporal Representation Learning, [Paper],[Code]
• (arXiv 2022.01) BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions, [Paper],[Code]
• (arXiv 2022.01) OMNIVORE: A Single Model for Many Visual Modalities, [Paper],[Code]
• (arXiv 2022.01) A Pre-trained Audio-Visual Transformer for Emotion Recognition, [Paper]
• (arXiv 2022.01) Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition, [Paper]
• (arXiv 2022.02) Towards Weakly-Supervised Text Spotting using a Multi-Task Transformer, [Paper]
• (arXiv 2022.03) DXM-TransFuse U-net: Dual Cross-Modal Transformer Fusion U-net for Automated Nerve Identification, [Paper]
• (arXiv 2022.03) LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives, [Paper]
• (arXiv 2022.03) VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer, [Paper],[Project]
• (arXiv 2022.03) MDMMT-2: Multidomain Multimodal Transformer for Video Retrieval, One More Step Towards Generalization, [Paper],[Project]
• (arXiv 2022.03) Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation, [Paper]
• (arXiv 2022.03) Inverted Pyramid Multi-task Transformer for Dense Scene Understanding, [Paper]
• (arXiv 2022.03) UNIMO-2: End-to-End Unified Vision-Language Grounded Learning, [Paper],[Project]
• (arXiv 2022.03) Multi-Modal Learning for AU Detection Based on Multi-Head Fused Transformers, [Paper]
• (arXiv 2022.03) UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection, [Paper],[Project]
• (arXiv 2022.03) Multi-modal Multi-label Facial Action Unit Detection with Transformer, [Paper],[Project]
• (arXiv 2022.03) Multimodal Fusion Transformer for Remote Sensing Image Classification, [Paper]
• (arXiv 2022.03) VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers, [Paper]
• (arXiv 2022.04) MultiMAE: Multi-modal Multi-task Masked Autoencoders, [Paper],[Project]
• (arXiv 2022.04) Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutation, [Paper]
• (arXiv 2022.04) MHMS: Multimodal Hierarchical Multimedia Summarization, [Paper]
• (arXiv 2022.04) Multimodal Transformer for Nursing Activity Recognition, [Paper]
• (arXiv 2022.04) Are Multimodal Transformers Robust to Missing Modality?, [Paper]
• (arXiv 2022.04) X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks, [Paper]
• (arXiv 2022.04) Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks, [Paper]
• (arXiv 2022.04) Multimodal Token Fusion for Vision Transformers, [Paper]
• (arXiv 2022.04) Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval, [Paper], [Code]
• (arXiv 2022.04) ParkPredict+: Multimodal Intent and Motion Prediction for Vehicles in Parking Lots with CNN and Transformer, [Paper]
• (arXiv 2022.05) Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training, [Paper]
• (arXiv 2022.05) MulT: An End-to-End Multitask Learning Transformer, [Paper]), [Project]
• (arXiv 2022.05) Training Vision-Language Transformers from Captions Alone, [Paper], [Code]
• (arXiv 2022.05) GIT: A Generative Image-to-text Transformer for Vision and Language, [Paper]
• (arXiv 2022.05) Multi-Task Learning with Multi-query Transformer for Dense Prediction, [Paper]
• (arXiv 2022.06) VL-BEIT: Generative Vision-Language Pretraining, [Paper], [Code]
• (arXiv 2022.06) AntPivot: Livestream Highlight Detection via Hierarchical Attention Mechanism, [Paper]
• (arXiv 2022.06) A Unified Sequence Interface for Vision Tasks, [Paper]
• (arXiv 2022.06) Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos, [Paper]
• (arXiv 2022.06) Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks, [Paper]
• (arXiv 2022.06) M&M Mix: A Multimodal Multiview Transformer Ensemble, [Paper]
• (arXiv 2022.06) RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrieval, [Paper]
• (arXiv 2022.07) You Only Need One Detector: Unified Object Detector for Different Modalities based on Vision Transformers, [Paper], [Code]
• (arXiv 2022.07) Open-Vocabulary Multi-Label Classification via Multi-modal Knowledge Transfer, [Paper], [Code]
• (arXiv 2022.07) Audio鈭扸isual Segmentation, [Paper], [Code]
• (arXiv 2022.07) FashionViL: Fashion-Focused Vision-and-Language Representation Learning, [Paper], [Code]
• (arXiv 2022.07) Multimodal Transformer for Automatic 3D Annotation and Object Detection, [Paper], [Code]
• (arXiv 2022.07) UFO: Unified Feature Optimization, [Paper], [Code]
• (arXiv 2022.07) An Ensemble Approach for Multiple Emotion Descriptors Estimation Using Multi-task Learning, [Paper]
• (arXiv 2022.07) STrajNet: Occupancy Flow Prediction via Multi-modal Swin Transformer, [Paper]
• (arXiv 2022.08) Multi-Task Transformer with uncertainty modelling for Face Based Affective Computing, [Paper]
• (arXiv 2022.08) Multi-modal Transformer Path Prediction for Autonomous Vehicle, [Paper]
• (arXiv 2022.08) Efficient Multimodal Transformer with Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis, [Paper]
• (arXiv 2022.08) VAuLT: Augmenting the Vision-and-Language Transformer with the Propagation of Deep Language Representations, [Paper], [Code]
• (arXiv 2022.08) Flat Multi-modal Interaction Transformer for Named Entity Recognition, [Paper], [Code]
• (arXiv 2022.08) TFusion: Transformer based N-to-One Multimodal Fusion Block, [Paper]
• (arXiv 2022.09) Multi-task Swin Transformer for Motion Artifacts Classification and Cardiac Magnetic Resonance Image Segmentation, [Paper]
• (arXiv 2022.09) TMSS: An End-to-End Transformer-based Multimodal Network for Segmentation and Survival Prediction, [Paper], [Code]
• (arXiv 2022.09) Can We Solve 3D Vision Tasks Starting from A 2D Vision Transformer, [Paper], [Code]
• (arXiv 2022.09) UniColor: A Unified Framework for Multi-Modal Colorization with Transformer, [Paper], [Code]
• (arXiv 2022.09) TVLT: Textless Vision-Language Transformer, [Paper], [Code]
• (arXiv 2022.10) A Strong Transfer Baseline for RGB-D Fusion in Vision Transformers, [Paper]
• (arXiv 2022.10) Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data, [Paper]
• (arXiv 2022.10) VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment, [Paper]
• (arXiv 2022.10) Transformer-based Localization from Embodied Dialog with Large-scale Pre-training, [Paper]
• (arXiv 2022.10) AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization, [Paper]
• (arXiv 2022.10) Understanding Embodied Reference with Touch-Line Transformer, [Paper]
• (arXiv 2022.10) Foundation Transformers, [Paper]
• (arXiv 2022.10) PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learning, [Paper]
• (arXiv 2022.10) Multimodal Image Fusion based on Hybrid CNN-Transformer and Non-local Cross-modal Attention, [Paper], [Code]
• (arXiv 2022.10) Multi-Source Transformer Architectures for Audiovisual Scene Classification, [Paper]
• (arXiv 2022.10) Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies, [Paper]
• (arXiv 2022.10) M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design, [Paper], [Code]
• (arXiv 2022.10) TAMFormer: Multi-Modal Transformer with Learned Attention Mask for Early Intent Prediction, [Paper], [Code]
• (arXiv 2022.10) Multimodal Transformer Distillation for Audio-Visual Synchronization, [Paper]
• (arXiv 2022.10) Masked Vision-Language Transformer in Fashion, [Paper], [Code]
• (arXiv 2022.10) Multimodal Transformer for Parallel Concatenated Variational Autoencoders, [Paper]
• (arXiv 2022.10) RCDPT: Radar-Camera fusion Dense Prediction Transformer, [Paper]
• (arXiv 2022.11) Efficient Joint Detection and Multiple Object Tracking with Spatially Aware Transformer, [Paper]
• (arXiv 2022.11) OneFormer: One Transformer to Rule Universal Image Segmentation, [Paper], [Code]
• (arXiv 2022.11) TransCC: Transformer-based Multiple Illuminant Color Constancy Using Multitask Learning, [Paper]
• (arXiv 2022.11) Unifying Vision-Language Representation Space with Single-tower Transformer, [Paper]
• (arXiv 2022.11) Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion, [Paper], [Code]
• (arXiv 2022.12) Multimodal Vision Transformers with Forced Attention for Behavior Analysis, [Paper]
• (arXiv 2022.12) Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformers, [Paper], [Code]
• (arXiv 2022.12) Hierarchical multimodal transformers for Multi-Page DocVQA, [Paper]
• (arXiv 2022.12) Vision Transformers are Parameter-Efficient Audio-Visual Learners, [Paper], [Code]
• (arXiv 2022.12) Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program, [Paper]
• (arXiv 2023.01) Cross Modal Transformer via Coordinates Encoding for 3D Object Dectection, [Paper], [Code]
• (arXiv 2023.01) DeMT: Deformable Mixer Transformer for Multi-Task Learning of Dense Prediction, [Paper], [Code]
• (arXiv 2023.01) Multi-scale multi-modal micro-expression recognition algorithm based on transformer, [Paper]
• (arXiv 2023.01) Logically at Factify 2023: A Multi-Modal Fact Checking System Based on Evidence Retrieval techniques and Transformer Encoder Architecture, [Paper]
• (arXiv 2023.01) ViTs for SITS: Vision Transformers for Satellite Image ries, [Paper]
• (arXiv 2023.01) Zorro: the masked multimodal transformer, [Paper]
• (arXiv 2023.01) Multimodal Event Transformer for Image-guided Story Ending Generation, [Paper]
• (arXiv 2023.01) UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers, [Paper]
• (arXiv 2023.02) Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing, [Paper]
• (arXiv 2023.02) ViM: Vision Middleware for Unified Downstream Transferring, [Paper]
• (arXiv 2023.03) One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale, [Paper], [Code]
• (arXiv 2023.03) MAGVLT: Masked Generative Vision-and-Language Transformer, [Paper]
• (arXiv 2023.03) LiDARFormer: A Unified Transformer-based Multi-task Network for LiDAR Perception, [Paper]
• (arXiv 2023.03) MMFormer: Multimodal Transformer Using Multiscale Self-Attention for Remote Sensing Image Classification, [Paper]
• (arXiv 2023.04) Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification, [Paper]
• (arXiv 2023.04) PARFormer: Transformer-based Multi-Task Network for Pedestrian Attribute Recognition, [Paper], [Code]
• (arXiv 2023.04) AutoTaskFormer: Searching Vision Transformers for Multi-task Learning, [Paper]
• (arXiv 2023.05) MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer, [Paper], [Code]
• (arXiv 2023.05) MMoT: Mixture-of-Modality-Tokens Transformer for Composed Multimodal Conditional Image Synthesis, [Paper], [Project]
• (arXiv 2023.05) JOINEDTrans: Prior Guided Multi-task Transformer for Joint Optic Disc/Cup Segmentation and Fovea Detection, [Paper], [Project]
• (arXiv 2023.05) Brain encoding models based on multimodal transformers can transfer across language and vision, [Paper]
• (arXiv 2023.05) CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers, [Paper], [Code]
• (arXiv 2023.05) Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts, [Paper], [Code]
• (arXiv 2023.06) Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification, [Paper], [Code]
• (arXiv 2023.06) Energy-Based Models for Cross-Modal Localization using Convolutional Transformers, [Paper]
• (arXiv 2023.06) Efficient Multi-Task Scene Analysis with RGB-D Transformers, [Paper], [Code]
• (arXiv 2023.06) ContentCTR: Frame-level Live Streaming Click-Through Rate Prediction with Multimodal Transformer, [Paper]
• (arXiv 2023.07) End-To-End Prediction of Knee Osteoarthritis Progression With Multi-Modal Transformers, [Paper]
• (arXiv 2023.07) Interactive Image Segmentation with Cross-Modality Vision Transformers, [Paper], [Code]
• (arXiv 2023.07) TransNuSeg: A Lightweight Multi-Task Transformer for Nuclei Segmentation, [Paper], [Code]
• (arXiv 2023.07) Meta-Transformer: A Unified Framework for Multimodal Learning, [Paper], [Project]
• (arXiv 2023.07) ComPtr: Towards Diverse Bi-source Dense Prediction Tasks via A Simple yet General Complementary Transformer, [Paper], [Code]
• (arXiv 2023.07) Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation, [Paper]
• (arXiv 2023.07) Prompt Guided Transformer for Multi-Task Dense Prediction, [Paper]
• (arXiv 2023.07) Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics, [Paper]
• (arXiv 2023.08) FusionAD: Multi-modality Fusion for Prediction and Planning Tasks of Autonomous Driving, [Paper]
• (arXiv 2023.08) Multimodal Neurons in Pretrained Text-Only Transformers, [Paper], [Project]
• (arXiv 2023.08) A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models, [Paper]
• (arXiv 2023.08) 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment, [Paper], [Project]
• (arXiv 2023.08) Vision Transformer Adapters for Generalizable Multitask Learning, [Paper], [Code]
• (arXiv 2023.08) UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization, [Paper], [Code]
• (arXiv 2023.09) Exchanging-based Multimodal Fusion with Transformer, [Paper], [Code]
• (arXiv 2023.09) Multimodal Transformer for Material Segmentation, [Paper], [Code]
• (arXiv 2023.09) Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens, [Paper]
• (arXiv 2023.09) MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi-Modal Spatial-Temporal Vision Transformer, [Paper], [Code]
• (arXiv 2023.09) Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation, [Paper]
• (arXiv 2023.09) Tile Classification Based Viewport Prediction with Multi-modal Fusion Transformer, [Paper]
• (arXiv 2023.10) LeTFuser: Light-weight End-to-end Transformer-Based Sensor Fusion for Autonomous Driving with Multi-Task Learning, [Paper], [Code]
• (arXiv 2023.10) 3M-TRANSFORMER: A Multi-Stage Multi-Stream Multimodal Transformer for Embodied Turn-Taking Prediction, [Paper]
• (arXiv 2023.10) MMTF-DES: A Fusion of Multimodal Transformer Models for Desire, Emotion, and Sentiment Analysis of Social Media Data, [Paper]
• (arXiv 2023.11) Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretation, [Paper]
• (arXiv 2023.11) Self-MI: Efficient Multimodal Fusion via Self-Supervised Multi-Task Learning with Auxiliary Mutual Information Maximization, [Paper]
• (arXiv 2023.11) PolyMaX: General Dense Prediction with Mask Transformer, [Paper]
• (arXiv 2023.11) Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain, [Paper]
• (arXiv 2023.11) Language Grounded QFormer for Efficient Vision Language Understanding, [Paper]
• (arXiv 2023.11) DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models, [Paper]
• (arXiv 2023.11) VIT-LENS-2: Gateway to Omni-modal Intelligence, [Paper], [[Code]](https://github.com/TencentARC/ViT-Lens锛?
• (arXiv 2023.11) You Only Learn One Query: Learning Unified Human Query for Single-Stage Multi-Person Multi-Task Human-Centric Perception, [Paper]
• (arXiv 2023.12) VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation, [Paper]
• (arXiv 2024.01) Multimodal Informative ViT: Information Aggregation and Distribution for Hyperspectral and LiDAR Classification, [Paper], [Code]
• (arXiv 2024.01) SeTformer is What You Need for Vision and Language, [Paper]
• (arXiv 2024.01) Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities, [Paper], [Code]
• (arXiv 2024.01) Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition, [Paper]
• (arXiv 2024.02) Question Aware Vision Transformer for Multimodal Reasoning, [Paper]
• (arXiv 2024.02) A Touch, Vision, and Language Dataset for Multimodal Alignment, [Paper], [Code]
• (arXiv 2024.02) Multimodal Transformer With a Low-Computational-Cost Guarantee, [Paper]
• (arXiv 2024.03) Task Indicating Transformer for Task-conditional Dense Predictions, [Paper]
• (arXiv 2024.03) Multimodal Transformer for Comics Text-Cloze, [Paper]
• (arXiv 2024.03) MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer, [Paper], [Code]
• (arXiv 2024.03) GiT: Towards Generalist Vision Transformer through Universal Language Interface, [Paper], [Code]
• (arXiv 2024.03) Uni-SMART: Universal Science Multimodal Analysis and Research Transformer, [Paper]
• (arXiv 2024.03) Affective Behaviour Analysis via Integrating Multi-Modal Knowledge, [Paper]
• (arXiv 2024.03) M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving, [Paper], [Code]
• (arXiv 2024.04) ViTamin: Designing Scalable Vision Models in the Vision-Language Era, [Paper], [Code]
• (arXiv 2024.04) GLID: Pre-training a Generalist Encoder-Decoder Vision Model, [Paper]
• (arXiv 2024.04) The revenge of BiSeNet: Efficient Multi-Task Image Segmentation, [Paper]
• (arXiv 2024.04) HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild, [Paper], [Code]
• (arXiv 2024.04) UniRGB-IR: A Unified Framework for Visible-Infrared Downstream Tasks via Adapter Tuning, [Paper], [Code]
• (arXiv 2024.05) Similarity Guided Multimodal Fusion Transformer for Semantic Location Prediction in Social Media, [Paper]
• (arXiv 2024.05) CamViG: Camera Aware Image-to-Video Generation with Multimodal Transformers, [Paper]
• (arXiv 2024.06) GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer, [Paper], [Code]
• (arXiv 2024.07) DaBiT: Depth and Blur informed Transformer for Joint Refocusing and Super-Resolution, [Paper], [Code]
• (arXiv 2024.07) Instruct-IPT: All-in-One Image Processing Transformer via Weight Modulation, [Paper], [Code]
• (arXiv 2024.07) CorMulT: A Semi-supervised Modality Correlation-aware Multimodal Transformer for Sentiment Analysis, [Paper]
• (arXiv 2024.07) A Multimodal Transformer for Live Streaming Highlight Prediction, [Paper]
• (arXiv 2024.07) Hierarchical Multi-modal Transformer for Cross-modal Long Document Classification, [Paper]
• (arXiv 2024.08) Algorithm Research of ELMo Word Embedding and Deep Learning Multimodal Transformer in Image Description, [Paper]
• (arXiv 2024.08) Show-o: One Single Transformer to Unify Multimodal Understanding and Generation, [Paper], [Code]
• (arXiv 2024.08) Enhanced Parking Perception by Multi-Task Fisheye Cross-view Transformers, [Paper]
• (arXiv 2024.08) SITransformer: Shared Information-Guided Transformer for Extreme Multimodal Summarization, [Paper], [Code]
• (arXiv 2024.09) Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy, [Paper]
• (arXiv 2024.09) Genetic Information Analysis of Age-Related Macular Degeneration Fellow Eye Using Multi-Modal Selective ViT, [Paper]
• (arXiv 2024.11) UIFormer: A Unified Transformer-based Framework for Incremental Few-Shot Object Detection and Instance Segmentation, [Paper]
• (arXiv 2024.11) CMAViT: Integrating Climate, Managment, and Remote Sensing Data for Crop Yield Estimation with Multimodel Vision Transformers, [Paper], [Code]
• (arXiv 2024.12) VibrantVS: A high-resolution multi-task transformer for forest canopy height estimation, [Paper]
• (arXiv 2024.12) UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation, [Paper]
• (arXiv 2025.01) TADFormer: Task-Adaptive Dynamic Transformer for Efficient Multi-Task Learning, [Paper]
• (arXiv 2025.01) Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning, [Paper]
• (arXiv 2025.01) CroMe: Multimodal Fake News Detection using Cross-Modal Tri-Transformer and Metric Learning, [Paper]
• (arXiv 2025.02) UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths, [Paper], [Code]
• (arXiv 2025.03) Solar Multimodal Transformer: Intraday Solar Irradiance Predictor using Public Cameras and Time Series, [Paper]
• (arXiv 2025.03) CFSum: A Transformer-Based Multi-Modal Video Summarization Framework With Coarse-Fine Fusion, [Paper]
• (arXiv 2025.03) SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation, [Paper]
• (arXiv 2025.03) CoCMT: Communication-Efficient Cross-Modal Transformer for Collaborative Perception, [Paper], [Code]
• (arXiv 2025.03) LIAM: Multimodal Transformer for Language Instructions, Images, Actions and Semantic Maps, [Paper], [Code]
• (arXiv 2025.04) Audio and Multiscale Visual Cues Driven Cross-modal Transformer for Idling Vehicle Detection, [Paper]
• (arXiv 2025.05) OpenVision : A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning, [Paper], [Code]
• (arXiv 2025.06) MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution, [Paper], [Code]
• (arXiv 2025.06) RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer, [Paper], [Code]
• (arXiv 2025.07) CoSMo: A Multimodal Transformer for Page Stream Segmentation in Comic Books, [Paper]
• (arXiv 2025.07) Resolving Token-Space Gradient Conflicts: Token Space Manipulation for Transformer-Based Multi-Task Learning, [Paper], [Code]
• (arXiv 2025.07) OmniVec2 - A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning, [Paper]
• (arXiv 2025.08) Efficient Inter-Task Attention for Multitask Transformer Models, [Paper]
• (arXiv 2025.08) MMFformer: Multimodal Fusion Transformer Network for Depression Detection, [Paper], [Code]
• (arXiv 2025.09) Sensor-Adaptive Flood Mapping with Pre-trained Multi-Modal Transformers across SAR and Multispectral Modalities, [Paper]