title: Dataset viewer issue for Winoground
comments: related (same dataset): https://github.com/huggingface/datasets/issues/4149. But the issue is different. Looking at it I thought this issue was related to the error I was seeing, but upon consideration I'd think the dataset viewer would return a 500 (unable to create the split like me) or a 404 (unable to load split b/c it was never created) error if it was having the issue I was seeing in #4149. 401 message makes it look like dataset viewer isn't passing through the identity of the user who has signed the licensing agreement when making the request to GET [examples.jsonl](https://huggingface.co/datasets/facebook/winoground/resolve/a86a60456fbbd242e9a744199071a6bd3e7fd9de/examples.jsonl). Pinging @SBrandeis, as it seems related to gated datasets and access tokens. To replicate:
  
  ```python
  >>> import datasets
  >>> dataset= datasets.load_dataset('facebook/winoground', name='facebook--winoground', split='train', use_auth_token="hf_app_...", streaming=True)
  >>> next(iter(dataset))
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
      for key, example in self._iter():
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
      yield from ex_iterable
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
      yield from self.generate_examples_fn(**self.kwargs)
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 439, in wrapper
      for key, table in generate_tables_fn(**kwargs):
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/packaged_modules/json/json.py", line 85, in _generate_tables
      for file_idx, file in enumerate(files):
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 679, in __iter__
      yield from self.generator(*self.args, **self.kwargs)
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 731, in _iter_from_urlpaths
      for dirpath, _, filenames in xwalk(urlpath, use_auth_token=use_auth_token):
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 623, in xwalk
      for dirpath, dirnames, filenames in fs.walk(main_hop):
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/spec.py", line 372, in walk
      listing = self.ls(path, detail=True, **kwargs)
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 85, in wrapper
      return sync(self.loop, func, *args, **kwargs)
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 65, in sync
      raise return_result
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
      result[0] = await coro
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 196, in _ls
      out = await self._ls_real(url, detail=detail, **kwargs)
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 150, in _ls_real
      self._raise_not_found_for_status(r, url)
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 208, in _raise_not_found_for_status
      response.raise_for_status()
    File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
      raise ClientResponseError(
  aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/facebook/winoground/resolve/a86a60456fbbd242e9a744199071a6bd3e7fd9de/examples.jsonl')
  ```
  
  *edited to fix `use_token` -> `use_auth_token`, thx @odellus* ~~Using your command to replicate and changing `use_token` to `use_auth_token` fixes the problem I was seeing in #4149.~~
  Nevermind it gave me an iterator to a method returning the same 401s. Changing `use_token` to `use_auth_token` does not fix the issue. After investigation with @severo , we found a potential culprit: https://github.com/huggingface/datasets/blob/3cd0a009a43f9f174056d70bfa2ca32216181926/src/datasets/utils/streaming_download_manager.py#L610-L624
  
  The streaming manager does not seem to pass `use_auth_token` to `fsspec` when streaming and not iterating content of a zip archive
  
  cc @albertvillanova @lhoestq  I was able to reproduce it on a private dataset, let me work on a fix Hey @lhoestq, Thanks for working on a fix! Any plans to merge #4173 into master?  Thanks for the heads up, I still need to fix some tests that are failing in the CI before merging ;) The fix has been merged, we'll do a new release soon, and update the dataset viewer
body: ## Dataset viewer issue for 'Winoground'
  
  **Link:** [*link to the dataset viewer page*](https://huggingface.co/datasets/facebook/winoground/viewer/facebook--winoground/train)
  
  *short description of the issue*
  Getting 401, message='Unauthorized'
  The dataset is subject to authorization, but I can access the files from the interface, so I assume I'm granted to access it. I'd assume the permission somehow doesn't propagate to the dataset viewer tool.
  
  Am I the one who added this dataset ? No
  