Hugging Face's logo

Spaces:
markqiu
/
prinvest_mate
Sleeping

App Files Community
Tuchuanhuhuhu commited on
Commit
e0bc027
·
1 Parent(s): 05bf1ca

token计数改为直接请求模型

Browse files
Files changed (1) hide show
  1. utils.py +1 -1
utils.py CHANGED
@@ -49,7 +49,7 @@ def postprocess(
49
  return y
50
 
51
  def count_token(input_str):
52
- encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")
53
  length = len(encoding.encode(input_str))
54
  return length
55
 
 
49
  return y
50
 
51
  def count_token(input_str):
52
+ encoding = tiktoken.get_encoding("cl100k_base")
53
  length = len(encoding.encode(input_str))
54
  return length
55