瀏覽代碼

automatically strip empty data entries (for better alpaca dataset compat)

Alex "mcmonkey" Goodwin 2 年之前
父節點
當前提交
7bf601107c
共有 1 個文件被更改,包括 1 次插入1 次删除
  1. 1 1
      modules/training.py

+ 1 - 1
modules/training.py

@@ -74,7 +74,7 @@ def do_train(loraName: str, microBatchSize: int, batchSize: int, epochs: int, le
         }
     def generate_prompt(data_point: dict[str, str]):
         for options, data in formatData.items():
-            if set(options.split(',')) == set(data_point.keys()):
+            if set(options.split(',')) == set(x[0] for x in data_point.items() if len(x[1].strip()) > 0):
                 for key, val in data_point.items():
                     data = data.replace(f'%{key}%', val)
             return data