Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models

Jinta Weng; Donghao Li; Yifan Deng; Jie Zhang; Yue Hu; Heyan Huang

doi:10.1007/978-981-99-8148-9_5

Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models

Jinta Weng, Donghao Li, Yifan Deng, Jie Zhang, Yue Hu^*, Heyan Huang

^*Corresponding author for this work

School of Computer Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The effectiveness of large language models (LLMs) and instruction learning has been demonstrated in different pre-trained language models (such as ChatGPT). However, current prompt learning methods usually use a unified template for the same tasks, and the template is difficult to capture significant information from different instances. To integrate the semantic attention dynamically on the instance level, We propose ISPrompt, an instance-semantic-aware prompt learning model. Specifically, the instance-driven prompt generated from the semantic dependency tree is introduced. Then, the proposed model would select a suitable semantic prompt from the prompt selection pool to motivate the prompt-based fine-tuning process. Our results show that the proposed model achieves state-of-the-art performance on few-shot learning tasks, which proves that ISPrompt integrating the instance semantics dynamically could assume as a better knowledge-mining tool for PLMs.

Original language	English
Title of host publication	Neural Information Processing - 30th International Conference, ICONIP 2023, Proceedings
Editors	Biao Luo, Long Cheng, Zheng-Guang Wu, Hongyi Li, Chaojie Li
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	55-67
Number of pages	13
ISBN (Print)	9789819981472
DOIs	https://doi.org/10.1007/978-981-99-8148-9_5
Publication status	Published - 2024
Event	30th International Conference on Neural Information Processing, ICONIP 2023 - Changsha, China Duration: 20 Nov 2023 → 23 Nov 2023

Publication series

Name	Communications in Computer and Information Science
Volume	1966 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	30th International Conference on Neural Information Processing, ICONIP 2023
Country/Territory	China
City	Changsha
Period	20/11/23 → 23/11/23

Keywords

AIGC
Large Language Models
deep learning
instruction learning
prompt learning

Access to Document

10.1007/978-981-99-8148-9_5

Cite this

Weng, J., Li, D., Deng, Y., Zhang, J., Hu, Y., & Huang, H. (2024). Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models. In B. Luo, L. Cheng, Z.-G. Wu, H. Li, & C. Li (Eds.), Neural Information Processing - 30th International Conference, ICONIP 2023, Proceedings (pp. 55-67). (Communications in Computer and Information Science; Vol. 1966 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-981-99-8148-9_5

Weng, Jinta ; Li, Donghao ; Deng, Yifan et al. / Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models. Neural Information Processing - 30th International Conference, ICONIP 2023, Proceedings. editor / Biao Luo ; Long Cheng ; Zheng-Guang Wu ; Hongyi Li ; Chaojie Li. Springer Science and Business Media Deutschland GmbH, 2024. pp. 55-67 (Communications in Computer and Information Science).

@inproceedings{1658e9ebe9984378af9df335132e63ad,

title = "Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models",

abstract = "The effectiveness of large language models (LLMs) and instruction learning has been demonstrated in different pre-trained language models (such as ChatGPT). However, current prompt learning methods usually use a unified template for the same tasks, and the template is difficult to capture significant information from different instances. To integrate the semantic attention dynamically on the instance level, We propose ISPrompt, an instance-semantic-aware prompt learning model. Specifically, the instance-driven prompt generated from the semantic dependency tree is introduced. Then, the proposed model would select a suitable semantic prompt from the prompt selection pool to motivate the prompt-based fine-tuning process. Our results show that the proposed model achieves state-of-the-art performance on few-shot learning tasks, which proves that ISPrompt integrating the instance semantics dynamically could assume as a better knowledge-mining tool for PLMs.",

keywords = "AIGC, Large Language Models, deep learning, instruction learning, prompt learning",

author = "Jinta Weng and Donghao Li and Yifan Deng and Jie Zhang and Yue Hu and Heyan Huang",

note = "Publisher Copyright: {\textcopyright} 2024, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.; 30th International Conference on Neural Information Processing, ICONIP 2023 ; Conference date: 20-11-2023 Through 23-11-2023",

year = "2024",

doi = "10.1007/978-981-99-8148-9_5",

language = "English",

isbn = "9789819981472",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "55--67",

editor = "Biao Luo and Long Cheng and Zheng-Guang Wu and Hongyi Li and Chaojie Li",

booktitle = "Neural Information Processing - 30th International Conference, ICONIP 2023, Proceedings",

address = "Germany",

}

Weng, J, Li, D, Deng, Y, Zhang, J, Hu, Y & Huang, H 2024, Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models. in B Luo, L Cheng, Z-G Wu, H Li & C Li (eds), Neural Information Processing - 30th International Conference, ICONIP 2023, Proceedings. Communications in Computer and Information Science, vol. 1966 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 55-67, 30th International Conference on Neural Information Processing, ICONIP 2023, Changsha, China, 20/11/23. https://doi.org/10.1007/978-981-99-8148-9_5

Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models. / Weng, Jinta; Li, Donghao; Deng, Yifan et al.
Neural Information Processing - 30th International Conference, ICONIP 2023, Proceedings. ed. / Biao Luo; Long Cheng; Zheng-Guang Wu; Hongyi Li; Chaojie Li. Springer Science and Business Media Deutschland GmbH, 2024. p. 55-67 (Communications in Computer and Information Science; Vol. 1966 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models

AU - Weng, Jinta

AU - Li, Donghao

AU - Deng, Yifan

AU - Zhang, Jie

AU - Hu, Yue

AU - Huang, Heyan

PY - 2024

Y1 - 2024

N2 - The effectiveness of large language models (LLMs) and instruction learning has been demonstrated in different pre-trained language models (such as ChatGPT). However, current prompt learning methods usually use a unified template for the same tasks, and the template is difficult to capture significant information from different instances. To integrate the semantic attention dynamically on the instance level, We propose ISPrompt, an instance-semantic-aware prompt learning model. Specifically, the instance-driven prompt generated from the semantic dependency tree is introduced. Then, the proposed model would select a suitable semantic prompt from the prompt selection pool to motivate the prompt-based fine-tuning process. Our results show that the proposed model achieves state-of-the-art performance on few-shot learning tasks, which proves that ISPrompt integrating the instance semantics dynamically could assume as a better knowledge-mining tool for PLMs.

AB - The effectiveness of large language models (LLMs) and instruction learning has been demonstrated in different pre-trained language models (such as ChatGPT). However, current prompt learning methods usually use a unified template for the same tasks, and the template is difficult to capture significant information from different instances. To integrate the semantic attention dynamically on the instance level, We propose ISPrompt, an instance-semantic-aware prompt learning model. Specifically, the instance-driven prompt generated from the semantic dependency tree is introduced. Then, the proposed model would select a suitable semantic prompt from the prompt selection pool to motivate the prompt-based fine-tuning process. Our results show that the proposed model achieves state-of-the-art performance on few-shot learning tasks, which proves that ISPrompt integrating the instance semantics dynamically could assume as a better knowledge-mining tool for PLMs.

KW - AIGC

KW - Large Language Models

KW - deep learning

KW - instruction learning

KW - prompt learning

UR - http://www.scopus.com/inward/record.url?scp=85178604300&partnerID=8YFLogxK

U2 - 10.1007/978-981-99-8148-9_5

DO - 10.1007/978-981-99-8148-9_5

M3 - Conference contribution

AN - SCOPUS:85178604300

SN - 9789819981472

T3 - Communications in Computer and Information Science

SP - 55

EP - 67

BT - Neural Information Processing - 30th International Conference, ICONIP 2023, Proceedings

A2 - Luo, Biao

A2 - Cheng, Long

A2 - Wu, Zheng-Guang

A2 - Li, Hongyi

A2 - Li, Chaojie

PB - Springer Science and Business Media Deutschland GmbH

T2 - 30th International Conference on Neural Information Processing, ICONIP 2023

Y2 - 20 November 2023 through 23 November 2023

ER -

Weng J, Li D, Deng Y, Zhang J, Hu Y, Huang H. Instance-Aware and Semantic-Guided Prompt for Few-Shot Learning in Large Language Models. In Luo B, Cheng L, Wu ZG, Li H, Li C, editors, Neural Information Processing - 30th International Conference, ICONIP 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2024. p. 55-67. (Communications in Computer and Information Science). doi: 10.1007/978-981-99-8148-9_5