Spaces:

dl4ds
/

tutor_dev

Configuration error

iamhks14 commited on Mar 25, 2024

Commit

fe158b7

2 Parent(s): 7b87020 5a03eca

Merge pull request #17 from DL4DS/dataloaderfix

Files changed (1) hide show

code/modules/data_loader.py CHANGED Viewed

@@ -39,12 +39,14 @@ class DataLoader:
                     chunk_size=config["splitter_options"]["chunk_size"],
                     chunk_overlap=config["splitter_options"]["chunk_overlap"],
                     separators=config["splitter_options"]["chunk_separators"],
                 )
             else:
                 self.splitter = RecursiveCharacterTextSplitter(
                     chunk_size=config["splitter_options"]["chunk_size"],
                     chunk_overlap=config["splitter_options"]["chunk_overlap"],
                     separators=config["splitter_options"]["chunk_separators"],
                 )
         else:
             self.splitter = None

                     chunk_size=config["splitter_options"]["chunk_size"],
                     chunk_overlap=config["splitter_options"]["chunk_overlap"],
                     separators=config["splitter_options"]["chunk_separators"],
+                    disallowed_special=()
                 )
             else:
                 self.splitter = RecursiveCharacterTextSplitter(
                     chunk_size=config["splitter_options"]["chunk_size"],
                     chunk_overlap=config["splitter_options"]["chunk_overlap"],
                     separators=config["splitter_options"]["chunk_separators"],
+                    disallowed_special=()
                 )
         else:
             self.splitter = None