first upload

ResizeRight/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Assaf Shocher
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ResizeRight/README.md

@@ -0,0 +1,87 @@
+# ResizeRight
+This is a resizing packge for images or tensors, that supports both Numpy and PyTorch seamlessly. The main motivation for creating this is to address some **crucial incorrectness issues** that exist in all other resizing packages I am aware of. As far as I know, it is the only one that performs correctly in all cases.  ResizeRight is specially made for machine learning, image enhancement and restoration challenges.
+
+In the Pytorch case, it is **fully differentiable and can be used inside a neural network** both as a dynamic function (eg. inside some "forward" method) or as a PyTorch layer (nn.Module).
+ 
+The code is inspired by MATLAB's imresize function, but with crucial differences. It is specifically useful due to the following reasons:
+
+1. ResizeRight produces results **identical (PSNR>60dB) to MATLAB for the simple cases** (scale_factor * in_size is integer). None of the Python packages I am aware of, currently resize images with results similar to MATLAB's imresize (which is a common benchmark for image resotration tasks, especially super-resolution). 
+
+2. No other **differntiable** method I am aware of supports **AntiAliasing** as in MATLAB. Actually very few non-differentiable ones, including popular ones, do. This causes artifacts and inconsistency in downscaling. (see [this tweet](https://twitter.com/jaakkolehtinen/status/1258102168176951299) by [Jaakko Lehtinen](https://users.aalto.fi/~lehtinj7/)
+ for example).
+
+3. The most important part: In the general case where scale_factor * in_size is non-integer, **no existing resizing method I am aware of (including MATLAB) performs consistently.** ResizeRight is accurate and consistent due to its ability to process **both scale-factor and output-size** provided by the user. This is a super important feature for super-resolution and learning. One must acknowledge that the same output-size can be resulted with varying scale-factors as output-size is usually determined by *ceil(input_size * scale_factor)*. This situation creates dangerous lack of consistency. Best explained by example: say you have an image of size 9x9 and you resize by scale-factor of 0.5. Result size is 5x5. now you resize with scale-factor of 2. you get result sized 10x10. "no big deal", you must be thinking now, "I can resize it according to output-size 9x9", right? but then you will not get the correct scale-fcator which is calculated as output-size / input-size = 1.8.
+Due to a simple observation regarding the projection of the output grid to the input grid, ResizeRight is the only one that consistently maintains the image centered, as in optical zoom while complying with the exact scale-factor and output size the user requires. 
+This is one of the main reasons for creating this repository. this downscale-upscale consistency is often crucial for learning based tasks (e.g. ["Zero-Shot Super-Resolution"](http://www.wisdom.weizmann.ac.il/~vision/zssr/)), and does not exist in other python packages nor in MATLAB.
+
+4. Misalignment in resizing is a pandemic! Many existing packages actually return misaligned results. it is visually not apparent but can cause great damage to image enhancement tasks.(for example, see [how tensorflow's image resize stole 60 days of my life](https://hackernoon.com/how-tensorflows-tf-image-resize-stole-60-days-of-my-life-aba5eb093f35)). I personally also suffered from many misfortunate consequences of such missalignment before and throughout making this method.
+
+5. Resizing supports **both Numpy and PyTorch** tensors seamlessly, just by the type of input tensor given. Results are checked to be identical in both modes, so you can safely apply to different tensor types and maintain consistency. No Numpy <-> Torch conversion takes part at any step. The process is done exclusively with one of the frameworks. No direct dependency is needed, so you can run it without having PyTorch installed at all, or without Numpy. You only need one of them.
+
+6. For PyTorch you can either use a **ResizeLayer (nn.Module)** that calculates the resizing weights once on initialization and then applies them at each network pass. This also means resizing can be performed efficiently on **GPU** and on batches of images.
+
+7. On the other hand, you can use a **dynamic resize** function to scale to different scal-factors at each pass. Both ways built upon the same building-blocks and produce identical results. Both are differntiable and can be used inside a neural network.
+
+8. Differently from some existing methods, including MATLAB, You can **resize N-D tensors in M-D dimensions.** for any M<=N.
+
+9. You can specify a list of scale-factors to resize each dimension using a different scale-factor.
+
+10. You can easily add and embed your own interpolation methods for the resizer to use (see interp_mehods.py)
+
+11. Some calculations are done more efficiently than the MATLAB version (one example is that MATLAB extends the kernel size by 2, and then searches for zero columns in the weights and cancels them. ResizeRight uses an observation that resizing is actually a continuous convolution and avoids having these redundancies ahead, see Shocher et al. ["From Discrete to  Continuous Convolution Layers"](https://arxiv.org/abs/2006.11120)).
+--------
+
+### Usage:
+For dynamic resize using either Numpy or PyTorch:
+```
+resize_reight.resize(input, scale_factors=None, out_shape=None, 
+                     interp_method=interp_methods.cubic, support_sz=None, 
+                     antialiasing=True)
+```
+For a PyTorch layer (nn.Module):
+```
+resize_layer = resize_reight.ResizeLayer(in_shape, scale_factors=None, out_shape=None,
+                                         interp_method=interp_methods.cubic, support_sz=None,
+                                         antialiasing=True
+                           
+resize_layer(input)
+```
+
+__input__ :   
+the input image/tensor, a Numpy or Torch tensor.
+
+__in_shape__  (only specified for a static layer):   
+the input tensor shape. a list of integers.
+
+__scale_factors__:    
+can be specified as-  
+1. one scalar scale - then it will be assumed that you want to resize first two dims with this scale for Numpy or last two dims for PyTorch.  
+2. a list or tupple of scales - one for each dimension you want to resize. note: if length of the list is L then first L dims will be rescaled for Numpy and last L for PyTorch. 
+3. not specified - then it will be calculated using output_size. this is not recomended (see advantage 3 in the list above).   
+
+__out_shape__:   
+A list or tupple. if shorter than input.shape then only the first/last (depending np/torch) dims are resized. if not specified, can be calcualated from scale_factor.
+
+__interp_method__:   
+The type of interpolation used to calculate the weights. this is a scalar to scalar function that can be applied to tensors pointwise. The classical methods are implemented and can be found in interp_methods.py. (cubic, linear, laczos2, lanczos3, box).
+
+__support_sz__:   
+This is the support of the interpolation function, i.e length of non-zero segment over its 1d input domain. this is a characteristic of the function. eg. for bicubic 4, linear 2, laczos2 4, lanczos3 6, box 1.
+
+__antialiasing__:   
+This is an option similar to MATLAB's default. only relevant for downscaling. if true it basicly means that the kernel is stretched with 1/scale_factor to prevent aliasing (low-pass filtering)
+
+--------
+
+### Cite / credit
+If you find our work useful in your research or publication, please cite this work:
+```
+@misc{ResizeRight,
+  author = {Shocher, Assaf},
+  title = {ResizeRight},
+  year = {2018},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/assafshocher/ResizeRight}},
+}
+```

ResizeRight/interp_methods.py

@@ -0,0 +1,65 @@
+from math import pi
+
+try:
+    import torch
+except ImportError:
+    torch = None
+
+try:
+    import numpy
+except ImportError:
+    numpy = None
+
+if numpy is None and torch is None:
+    raise ImportError("Must have either Numpy or PyTorch but both not found")
+
+
+def set_framework_dependencies(x):
+    if type(x) is numpy.ndarray:
+        to_dtype = lambda a: a
+        fw = numpy
+    else:
+        to_dtype = lambda a: a.to(x.dtype)
+        fw = torch
+    eps = fw.finfo(fw.float32).eps
+    return fw, to_dtype, eps
+
+
+def support_sz(sz):
+    def wrapper(f):
+        f.support_sz = sz
+        return f
+    return wrapper
+
+@support_sz(4)
+def cubic(x):
+    fw, to_dtype, eps = set_framework_dependencies(x)
+    absx = fw.abs(x)
+    absx2 = absx ** 2
+    absx3 = absx ** 3
+    return ((1.5 * absx3 - 2.5 * absx2 + 1.) * to_dtype(absx <= 1.) +
+            (-0.5 * absx3 + 2.5 * absx2 - 4. * absx + 2.) *
+            to_dtype((1. < absx) & (absx <= 2.)))
+
+@support_sz(4)
+def lanczos2(x):
+    fw, to_dtype, eps = set_framework_dependencies(x)
+    return (((fw.sin(pi * x) * fw.sin(pi * x / 2) + eps) /
+            ((pi**2 * x**2 / 2) + eps)) * to_dtype(abs(x) < 2))
+
+@support_sz(6)
+def lanczos3(x):
+    fw, to_dtype, eps = set_framework_dependencies(x)
+    return (((fw.sin(pi * x) * fw.sin(pi * x / 3) + eps) /
+            ((pi**2 * x**2 / 3) + eps)) * to_dtype(abs(x) < 3))
+
+@support_sz(2)
+def linear(x):
+    fw, to_dtype, eps = set_framework_dependencies(x)
+    return ((x + 1) * to_dtype((-1 <= x) & (x < 0)) + (1 - x) *
+            to_dtype((0 <= x) & (x <= 1)))
+
+@support_sz(1)
+def box(x):
+    fw, to_dtype, eps = set_framework_dependencies(x)
+    return to_dtype((-1 <= x) & (x < 0)) + to_dtype((0 <= x) & (x <= 1))

ResizeRight/resize_right.py

@@ -0,0 +1,341 @@
+import warnings
+from math import ceil
+from . import interp_methods
+
+
+class NoneClass:
+    pass
+
+try:
+    import torch
+    from torch import nn
+    nnModuleWrapped = nn.Module
+except ImportError:
+    warnings.warn('No PyTorch found, will work only with Numpy')
+    torch = None
+    nnModuleWrapped = NoneClass
+
+try:
+    import numpy
+except ImportError:
+    warnings.warn('No Numpy found, will work only with PyTorch')
+    numpy = None
+
+
+if numpy is None and torch is None:
+    raise ImportError("Must have either Numpy or PyTorch but both not found")
+
+
+def resize(input, scale_factors=None, out_shape=None,
+           interp_method=interp_methods.cubic, support_sz=None,
+           antialiasing=True):
+    # get properties of the input tensor
+    in_shape, n_dims = input.shape, input.ndim
+
+    # fw stands for framework that can be either numpy or torch,
+    # determined by the input type
+    fw = numpy if type(input) is numpy.ndarray else torch
+    eps = fw.finfo(fw.float32).eps
+
+    # set missing scale factors or output shapem one according to another,
+    # scream if both missing
+    scale_factors, out_shape = set_scale_and_out_sz(in_shape, out_shape,
+                                                    scale_factors, fw)
+
+    # sort indices of dimensions according to scale of each dimension.
+    # since we are going dim by dim this is efficient
+    sorted_filtered_dims_and_scales = [(dim, scale_factors[dim])
+                                       for dim in sorted(range(n_dims),
+                                       key=lambda ind: scale_factors[ind])
+                                       if scale_factors[dim] != 1.]
+
+    # unless support size is specified by the user, it is an attribute
+    # of the interpolation method
+    if support_sz is None:
+        support_sz = interp_method.support_sz
+
+    # when using pytorch, we need to know what is the input tensor device
+    device = input.device if fw is torch else None
+
+    # output begins identical to input and changes with each iteration
+    output = input
+
+    # iterate over dims
+    for dim, scale_factor in sorted_filtered_dims_and_scales:
+
+        # get 1d set of weights and fields of view for each output location
+        # along this dim
+        field_of_view, weights = prepare_weights_and_field_of_view_1d(
+            dim, scale_factor, in_shape[dim], out_shape[dim], interp_method,
+            support_sz, antialiasing, fw, eps, device)
+
+        # multiply the weights by the values in the field of view and
+        # aggreagate
+        output = apply_weights(output, field_of_view, weights, dim, n_dims,
+                               fw)
+    return output
+
+
+class ResizeLayer(nnModuleWrapped):
+    def __init__(self, in_shape, scale_factors=None, out_shape=None,
+                 interp_method=interp_methods.cubic, support_sz=None,
+                 antialiasing=True):
+        super(ResizeLayer, self).__init__()
+
+        # fw stands for framework, that can be either numpy or torch. since
+        # this is a torch layer, only one option in this case.
+        fw = torch
+        eps = fw.finfo(fw.float32).eps
+
+        # set missing scale factors or output shapem one according to another,
+        # scream if both missing
+        scale_factors, out_shape = set_scale_and_out_sz(in_shape, out_shape,
+                                                        scale_factors, fw)
+
+        # unless support size is specified by the user, it is an attribute
+        # of the interpolation method
+        if support_sz is None:
+            support_sz = interp_method.support_sz
+
+        self.n_dims = len(in_shape)
+
+        # sort indices of dimensions according to scale of each dimension.
+        # since we are going dim by dim this is efficient
+        self.sorted_filtered_dims_and_scales = [(dim, scale_factors[dim])
+                                                for dim in
+                                                sorted(range(self.n_dims),
+                                                key=lambda ind:
+                                                scale_factors[ind])
+                                                if scale_factors[dim] != 1.]
+
+        # iterate over dims
+        field_of_view_list = []
+        weights_list = []
+        for dim, scale_factor in self.sorted_filtered_dims_and_scales:
+
+            # get 1d set of weights and fields of view for each output
+            # location along this dim
+            field_of_view, weights = prepare_weights_and_field_of_view_1d(
+                dim, scale_factor, in_shape[dim], out_shape[dim],
+                interp_method, support_sz, antialiasing, fw, eps, input.device)
+
+            # keep weights and fields of views for all dims
+            weights_list.append(nn.Parameter(weights, requires_grad=False))
+            field_of_view_list.append(nn.Parameter(field_of_view,
+                                      requires_grad=False))
+
+        self.field_of_view = nn.ParameterList(field_of_view_list)
+        self.weights = nn.ParameterList(weights_list)
+        self.in_shape = in_shape
+
+    def forward(self, input):
+        # output begins identical to input and changes with each iteration
+        output = input
+
+        for (dim, scale_factor), field_of_view, weights in zip(
+                self.sorted_filtered_dims_and_scales,
+                self.field_of_view,
+                self.weights):
+            # multiply the weights by the values in the field of view and
+            # aggreagate
+            output = apply_weights(output, field_of_view, weights, dim,
+                                   self.n_dims, torch)
+        return output
+
+
+def prepare_weights_and_field_of_view_1d(dim, scale_factor, in_sz, out_sz,
+                                         interp_method, support_sz,
+                                         antialiasing, fw, eps, device=None):
+    # If antialiasing is taking place, we modify the window size and the
+    # interpolation method (see inside function)
+    interp_method, cur_support_sz = apply_antialiasing_if_needed(
+                                                             interp_method,
+                                                             support_sz,
+                                                             scale_factor,
+                                                             antialiasing)
+
+    # STEP 1- PROJECTED GRID: The non-integer locations of the projection of
+    # output pixel locations to the input tensor
+    projected_grid = get_projected_grid(in_sz, out_sz, scale_factor, fw, device)
+
+    # STEP 2- FIELDS OF VIEW: for each output pixels, map the input pixels
+    # that influence it
+    field_of_view = get_field_of_view(projected_grid, cur_support_sz, in_sz,
+                                      fw, eps, device)
+
+    # STEP 3- CALCULATE WEIGHTS: Match a set of weights to the pixels in the
+    # field of view for each output pixel
+    weights = get_weights(interp_method, projected_grid, field_of_view)
+
+    return field_of_view, weights
+
+
+def apply_weights(input, field_of_view, weights, dim, n_dims, fw):
+    # STEP 4- APPLY WEIGHTS: Each output pixel is calculated by multiplying
+    # its set of weights with the pixel values in its field of view.
+    # We now multiply the fields of view with their matching weights.
+    # We do this by tensor multiplication and broadcasting.
+    # this step is separated to a different function, so that it can be
+    # repeated with the same calculated weights and fields.
+
+    # for this operations we assume the resized dim is the first one.
+    # so we transpose and will transpose back after multiplying
+    tmp_input = fw_swapaxes(input, dim, 0, fw)
+
+    # field_of_view is a tensor of order 2: for each output (1d location
+    # along cur dim)- a list of 1d neighbors locations.
+    # note that this whole operations is applied to each dim separately,
+    # this is why it is all in 1d.
+    # neighbors = tmp_input[field_of_view] is a tensor of order image_dims+1:
+    # for each output pixel (this time indicated in all dims), these are the
+    # values of the neighbors in the 1d field of view. note that we only
+    # consider neighbors along the current dim, but such set exists for every
+    # multi-dim location, hence the final tensor order is image_dims+1.
+    neighbors = tmp_input[field_of_view]
+
+    # weights is an order 2 tensor: for each output location along 1d- a list
+    # of weighs matching the field of view. we augment it with ones, for
+    # broadcasting, so that when multiplies some tensor the weights affect
+    # only its first dim.
+    tmp_weights = fw.reshape(weights, (*weights.shape, * [1] * (n_dims - 1)))
+
+    # now we simply multiply the weights with the neighbors, and then sum
+    # along the field of view, to get a single value per out pixel
+    tmp_output = (neighbors * tmp_weights).sum(1)
+
+    # we transpose back the resized dim to its original position
+    return fw_swapaxes(tmp_output, 0, dim, fw)
+
+
+def set_scale_and_out_sz(in_shape, out_shape, scale_factors, fw):
+    # eventually we must have both scale-factors and out-sizes for all in/out
+    # dims. however, we support many possible partial arguments
+    if scale_factors is None and out_shape is None:
+        raise ValueError("either scale_factors or out_shape should be "
+                         "provided")
+    if out_shape is not None:
+        # if out_shape has less dims than in_shape, we defaultly resize the
+        # first dims for numpy and last dims for torch
+        # out_shape = (list(out_shape) + list(in_shape[:-len(out_shape)])
+                     # if fw is numpy
+                     # else list(in_shape[:-len(out_shape)]) + list(out_shape))
+        out_shape = (list(out_shape) + list(in_shape[-len(out_shape):])
+                     if fw is numpy
+                     else list(in_shape[:-len(out_shape)]) + list(out_shape))
+        if scale_factors is None:
+            # if no scale given, we calculate it as the out to in ratio
+            # (not recomended)
+            scale_factors = [out_sz / in_sz for out_sz, in_sz
+                             in zip(out_shape, in_shape)]
+    if scale_factors is not None:
+        # by default, if a single number is given as scale, we assume resizing
+        # two dims (most common are images with 2 spatial dims)
+        scale_factors = (scale_factors
+                         if isinstance(scale_factors, (list, tuple))
+                         else [scale_factors, scale_factors])
+        # if less scale_factors than in_shape dims, we defaultly resize the
+        # first dims for numpy and last dims for torch
+        scale_factors = (list(scale_factors) + [1] *
+                         (len(in_shape) - len(scale_factors)) if fw is numpy
+                         else [1] * (len(in_shape) - len(scale_factors)) +
+                         list(scale_factors))
+        if out_shape is None:
+            # when no out_shape given, it is calculated by multiplying the
+            # scale by the in_shape (not recomended)
+            out_shape = [ceil(scale_factor * in_sz)
+                         for scale_factor, in_sz in
+                         zip(scale_factors, in_shape)]
+        # next line intentionally after out_shape determined for stability
+        scale_factors = [float(sf) for sf in scale_factors]
+    return scale_factors, out_shape
+
+
+def get_projected_grid(in_sz, out_sz, scale_factor, fw, device=None):
+    # we start by having the ouput coordinates which are just integer locations
+    out_coordinates = fw.arange(out_sz)
+
+    # if using torch we need to match the grid tensor device to the input device
+    out_coordinates = fw_set_device(out_coordinates, device, fw)
+
+    # This is projecting the ouput pixel locations in 1d to the input tensor,
+    # as non-integer locations.
+    # the following fomrula is derived in the paper
+    # "From Discrete to Continuous Convolutions" by Shocher et al.
+    return (out_coordinates / scale_factor +
+            (in_sz - 1) / 2 - (out_sz - 1) / (2 * scale_factor))
+
+
+def get_field_of_view(projected_grid, cur_support_sz, in_sz, fw, eps, device):
+    # for each output pixel, map which input pixels influence it, in 1d.
+    # we start by calculating the leftmost neighbor, using half of the window
+    # size (eps is for when boundary is exact int)
+    left_boundaries = fw_ceil(projected_grid - cur_support_sz / 2 - eps, fw)
+
+    # then we simply take all the pixel centers in the field by counting
+    # window size pixels from the left boundary
+    ordinal_numbers = fw.arange(ceil(cur_support_sz - eps))
+    # in case using torch we need to match the device
+    ordinal_numbers = fw_set_device(ordinal_numbers, device, fw)
+    field_of_view = left_boundaries[:, None] + ordinal_numbers
+
+    # next we do a trick instead of padding, we map the field of view so that
+    # it would be like mirror padding, without actually padding
+    # (which would require enlarging the input tensor)
+    mirror = fw_cat((fw.arange(in_sz), fw.arange(in_sz - 1, -1, step=-1)), fw)
+    field_of_view = mirror[fw.remainder(field_of_view, mirror.shape[0])]
+    field_of_view = fw_set_device(field_of_view, device, fw)
+    return field_of_view
+
+
+def get_weights(interp_method, projected_grid, field_of_view):
+    # the set of weights per each output pixels is the result of the chosen
+    # interpolation method applied to the distances between projected grid
+    # locations and the pixel-centers in the field of view (distances are
+    # directed, can be positive or negative)
+    weights = interp_method(projected_grid[:, None] - field_of_view)
+
+    # we now carefully normalize the weights to sum to 1 per each output pixel
+    sum_weights = weights.sum(1, keepdims=True)
+    sum_weights[sum_weights == 0] = 1
+    return weights / sum_weights
+
+
+def apply_antialiasing_if_needed(interp_method, support_sz, scale_factor,
+                                 antialiasing):
+    # antialiasing is "stretching" the field of view according to the scale
+    # factor (only for downscaling). this is low-pass filtering. this
+    # requires modifying both the interpolation (stretching the 1d
+    # function and multiplying by the scale-factor) and the window size.
+    if scale_factor >= 1.0 or not antialiasing:
+        return interp_method, support_sz
+    cur_interp_method = (lambda arg: scale_factor *
+                         interp_method(scale_factor * arg))
+    cur_support_sz = support_sz / scale_factor
+    return cur_interp_method, cur_support_sz
+
+
+def fw_ceil(x, fw):
+    if fw is numpy:
+        return fw.int_(fw.ceil(x))
+    else:
+        return x.ceil().long()
+
+
+def fw_cat(x, fw):
+    if fw is numpy:
+        return fw.concatenate(x)
+    else:
+        return fw.cat(x)
+
+
+def fw_swapaxes(x, ax_1, ax_2, fw):
+    if fw is numpy:
+        return fw.swapaxes(x, ax_1, ax_2)
+    else:
+        return x.transpose(ax_1, ax_2)
+
+def fw_set_device(x, device, fw):
+    if fw is numpy:
+        return x
+    else:
+        return x.to(device)

basicsr/__init__.py

@@ -0,0 +1,12 @@
+# https://github.com/xinntao/BasicSR
+# flake8: noqa
+from .archs import *
+from .data import *
+from .losses import *
+from .metrics import *
+from .models import *
+from .ops import *
+from .test import *
+from .train import *
+from .utils import *
+# from .version import __gitsha__, __version__

basicsr/archs/__init__.py

@@ -0,0 +1,24 @@
+import importlib
+from copy import deepcopy
+from os import path as osp
+
+from basicsr.utils import get_root_logger, scandir
+from basicsr.utils.registry import ARCH_REGISTRY
+
+__all__ = ['build_network']
+
+# automatically scan and import arch modules for registry
+# scan all the files under the 'archs' folder and collect files ending with '_arch.py'
+arch_folder = osp.dirname(osp.abspath(__file__))
+arch_filenames = [osp.splitext(osp.basename(v))[0] for v in scandir(arch_folder) if v.endswith('_arch.py')]
+# import all the arch modules
+_arch_modules = [importlib.import_module(f'basicsr.archs.{file_name}') for file_name in arch_filenames]
+
+
+def build_network(opt):
+    opt = deepcopy(opt)
+    network_type = opt.pop('type')
+    net = ARCH_REGISTRY.get(network_type)(**opt)
+    logger = get_root_logger()
+    logger.info(f'Network [{net.__class__.__name__}] is created.')
+    return net

basicsr/archs/arch_util.py

@@ -0,0 +1,313 @@
+import collections.abc
+import math
+import torch
+import torchvision
+import warnings
+from distutils.version import LooseVersion
+from itertools import repeat
+from torch import nn as nn
+from torch.nn import functional as F
+from torch.nn import init as init
+from torch.nn.modules.batchnorm import _BatchNorm
+
+from basicsr.ops.dcn import ModulatedDeformConvPack, modulated_deform_conv
+from basicsr.utils import get_root_logger
+
+
+@torch.no_grad()
+def default_init_weights(module_list, scale=1, bias_fill=0, **kwargs):
+    """Initialize network weights.
+
+    Args:
+        module_list (list[nn.Module] | nn.Module): Modules to be initialized.
+        scale (float): Scale initialized weights, especially for residual
+            blocks. Default: 1.
+        bias_fill (float): The value to fill bias. Default: 0
+        kwargs (dict): Other arguments for initialization function.
+    """
+    if not isinstance(module_list, list):
+        module_list = [module_list]
+    for module in module_list:
+        for m in module.modules():
+            if isinstance(m, nn.Conv2d):
+                init.kaiming_normal_(m.weight, **kwargs)
+                m.weight.data *= scale
+                if m.bias is not None:
+                    m.bias.data.fill_(bias_fill)
+            elif isinstance(m, nn.Linear):
+                init.kaiming_normal_(m.weight, **kwargs)
+                m.weight.data *= scale
+                if m.bias is not None:
+                    m.bias.data.fill_(bias_fill)
+            elif isinstance(m, _BatchNorm):
+                init.constant_(m.weight, 1)
+                if m.bias is not None:
+                    m.bias.data.fill_(bias_fill)
+
+
+def make_layer(basic_block, num_basic_block, **kwarg):
+    """Make layers by stacking the same blocks.
+
+    Args:
+        basic_block (nn.module): nn.module class for basic block.
+        num_basic_block (int): number of blocks.
+
+    Returns:
+        nn.Sequential: Stacked blocks in nn.Sequential.
+    """
+    layers = []
+    for _ in range(num_basic_block):
+        layers.append(basic_block(**kwarg))
+    return nn.Sequential(*layers)
+
+
+class ResidualBlockNoBN(nn.Module):
+    """Residual block without BN.
+
+    Args:
+        num_feat (int): Channel number of intermediate features.
+            Default: 64.
+        res_scale (float): Residual scale. Default: 1.
+        pytorch_init (bool): If set to True, use pytorch default init,
+            otherwise, use default_init_weights. Default: False.
+    """
+
+    def __init__(self, num_feat=64, res_scale=1, pytorch_init=False):
+        super(ResidualBlockNoBN, self).__init__()
+        self.res_scale = res_scale
+        self.conv1 = nn.Conv2d(num_feat, num_feat, 3, 1, 1, bias=True)
+        self.conv2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1, bias=True)
+        self.relu = nn.ReLU(inplace=True)
+
+        if not pytorch_init:
+            default_init_weights([self.conv1, self.conv2], 0.1)
+
+    def forward(self, x):
+        identity = x
+        out = self.conv2(self.relu(self.conv1(x)))
+        return identity + out * self.res_scale
+
+
+class Upsample(nn.Sequential):
+    """Upsample module.
+
+    Args:
+        scale (int): Scale factor. Supported scales: 2^n and 3.
+        num_feat (int): Channel number of intermediate features.
+    """
+
+    def __init__(self, scale, num_feat):
+        m = []
+        if (scale & (scale - 1)) == 0:  # scale = 2^n
+            for _ in range(int(math.log(scale, 2))):
+                m.append(nn.Conv2d(num_feat, 4 * num_feat, 3, 1, 1))
+                m.append(nn.PixelShuffle(2))
+        elif scale == 3:
+            m.append(nn.Conv2d(num_feat, 9 * num_feat, 3, 1, 1))
+            m.append(nn.PixelShuffle(3))
+        else:
+            raise ValueError(f'scale {scale} is not supported. Supported scales: 2^n and 3.')
+        super(Upsample, self).__init__(*m)
+
+
+def flow_warp(x, flow, interp_mode='bilinear', padding_mode='zeros', align_corners=True):
+    """Warp an image or feature map with optical flow.
+
+    Args:
+        x (Tensor): Tensor with size (n, c, h, w).
+        flow (Tensor): Tensor with size (n, h, w, 2), normal value.
+        interp_mode (str): 'nearest' or 'bilinear'. Default: 'bilinear'.
+        padding_mode (str): 'zeros' or 'border' or 'reflection'.
+            Default: 'zeros'.
+        align_corners (bool): Before pytorch 1.3, the default value is
+            align_corners=True. After pytorch 1.3, the default value is
+            align_corners=False. Here, we use the True as default.
+
+    Returns:
+        Tensor: Warped image or feature map.
+    """
+    assert x.size()[-2:] == flow.size()[1:3]
+    _, _, h, w = x.size()
+    # create mesh grid
+    grid_y, grid_x = torch.meshgrid(torch.arange(0, h).type_as(x), torch.arange(0, w).type_as(x))
+    grid = torch.stack((grid_x, grid_y), 2).float()  # W(x), H(y), 2
+    grid.requires_grad = False
+
+    vgrid = grid + flow
+    # scale grid to [-1,1]
+    vgrid_x = 2.0 * vgrid[:, :, :, 0] / max(w - 1, 1) - 1.0
+    vgrid_y = 2.0 * vgrid[:, :, :, 1] / max(h - 1, 1) - 1.0
+    vgrid_scaled = torch.stack((vgrid_x, vgrid_y), dim=3)
+    output = F.grid_sample(x, vgrid_scaled, mode=interp_mode, padding_mode=padding_mode, align_corners=align_corners)
+
+    # TODO, what if align_corners=False
+    return output
+
+
+def resize_flow(flow, size_type, sizes, interp_mode='bilinear', align_corners=False):
+    """Resize a flow according to ratio or shape.
+
+    Args:
+        flow (Tensor): Precomputed flow. shape [N, 2, H, W].
+        size_type (str): 'ratio' or 'shape'.
+        sizes (list[int | float]): the ratio for resizing or the final output
+            shape.
+            1) The order of ratio should be [ratio_h, ratio_w]. For
+            downsampling, the ratio should be smaller than 1.0 (i.e., ratio
+            < 1.0). For upsampling, the ratio should be larger than 1.0 (i.e.,
+            ratio > 1.0).
+            2) The order of output_size should be [out_h, out_w].
+        interp_mode (str): The mode of interpolation for resizing.
+            Default: 'bilinear'.
+        align_corners (bool): Whether align corners. Default: False.
+
+    Returns:
+        Tensor: Resized flow.
+    """
+    _, _, flow_h, flow_w = flow.size()
+    if size_type == 'ratio':
+        output_h, output_w = int(flow_h * sizes[0]), int(flow_w * sizes[1])
+    elif size_type == 'shape':
+        output_h, output_w = sizes[0], sizes[1]
+    else:
+        raise ValueError(f'Size type should be ratio or shape, but got type {size_type}.')
+
+    input_flow = flow.clone()
+    ratio_h = output_h / flow_h
+    ratio_w = output_w / flow_w
+    input_flow[:, 0, :, :] *= ratio_w
+    input_flow[:, 1, :, :] *= ratio_h
+    resized_flow = F.interpolate(
+        input=input_flow, size=(output_h, output_w), mode=interp_mode, align_corners=align_corners)
+    return resized_flow
+
+
+# TODO: may write a cpp file
+def pixel_unshuffle(x, scale):
+    """ Pixel unshuffle.
+
+    Args:
+        x (Tensor): Input feature with shape (b, c, hh, hw).
+        scale (int): Downsample ratio.
+
+    Returns:
+        Tensor: the pixel unshuffled feature.
+    """
+    b, c, hh, hw = x.size()
+    out_channel = c * (scale**2)
+    assert hh % scale == 0 and hw % scale == 0
+    h = hh // scale
+    w = hw // scale
+    x_view = x.view(b, c, h, scale, w, scale)
+    return x_view.permute(0, 1, 3, 5, 2, 4).reshape(b, out_channel, h, w)
+
+
+class DCNv2Pack(ModulatedDeformConvPack):
+    """Modulated deformable conv for deformable alignment.
+
+    Different from the official DCNv2Pack, which generates offsets and masks
+    from the preceding features, this DCNv2Pack takes another different
+    features to generate offsets and masks.
+
+    ``Paper: Delving Deep into Deformable Alignment in Video Super-Resolution``
+    """
+
+    def forward(self, x, feat):
+        out = self.conv_offset(feat)
+        o1, o2, mask = torch.chunk(out, 3, dim=1)
+        offset = torch.cat((o1, o2), dim=1)
+        mask = torch.sigmoid(mask)
+
+        offset_absmean = torch.mean(torch.abs(offset))
+        if offset_absmean > 50:
+            logger = get_root_logger()
+            logger.warning(f'Offset abs mean is {offset_absmean}, larger than 50.')
+
+        if LooseVersion(torchvision.__version__) >= LooseVersion('0.9.0'):
+            return torchvision.ops.deform_conv2d(x, offset, self.weight, self.bias, self.stride, self.padding,
+                                                 self.dilation, mask)
+        else:
+            return modulated_deform_conv(x, offset, mask, self.weight, self.bias, self.stride, self.padding,
+                                         self.dilation, self.groups, self.deformable_groups)
+
+
+def _no_grad_trunc_normal_(tensor, mean, std, a, b):
+    # From: https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/weight_init.py
+    # Cut & paste from PyTorch official master until it's in a few official releases - RW
+    # Method based on https://people.sc.fsu.edu/~jburkardt/presentations/truncated_normal.pdf
+    def norm_cdf(x):
+        # Computes standard normal cumulative distribution function
+        return (1. + math.erf(x / math.sqrt(2.))) / 2.
+
+    if (mean < a - 2 * std) or (mean > b + 2 * std):
+        warnings.warn(
+            'mean is more than 2 std from [a, b] in nn.init.trunc_normal_. '
+            'The distribution of values may be incorrect.',
+            stacklevel=2)
+
+    with torch.no_grad():
+        # Values are generated by using a truncated uniform distribution and
+        # then using the inverse CDF for the normal distribution.
+        # Get upper and lower cdf values
+        low = norm_cdf((a - mean) / std)
+        up = norm_cdf((b - mean) / std)
+
+        # Uniformly fill tensor with values from [low, up], then translate to
+        # [2l-1, 2u-1].
+        tensor.uniform_(2 * low - 1, 2 * up - 1)
+
+        # Use inverse cdf transform for normal distribution to get truncated
+        # standard normal
+        tensor.erfinv_()
+
+        # Transform to proper mean, std
+        tensor.mul_(std * math.sqrt(2.))
+        tensor.add_(mean)
+
+        # Clamp to ensure it's in the proper range
+        tensor.clamp_(min=a, max=b)
+        return tensor
+
+
+def trunc_normal_(tensor, mean=0., std=1., a=-2., b=2.):
+    r"""Fills the input Tensor with values drawn from a truncated
+    normal distribution.
+
+    From: https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/weight_init.py
+
+    The values are effectively drawn from the
+    normal distribution :math:`\mathcal{N}(\text{mean}, \text{std}^2)`
+    with values outside :math:`[a, b]` redrawn until they are within
+    the bounds. The method used for generating the random values works
+    best when :math:`a \leq \text{mean} \leq b`.
+
+    Args:
+        tensor: an n-dimensional `torch.Tensor`
+        mean: the mean of the normal distribution
+        std: the standard deviation of the normal distribution
+        a: the minimum cutoff value
+        b: the maximum cutoff value
+
+    Examples:
+        >>> w = torch.empty(3, 5)
+        >>> nn.init.trunc_normal_(w)
+    """
+    return _no_grad_trunc_normal_(tensor, mean, std, a, b)
+
+
+# From PyTorch
+def _ntuple(n):
+
+    def parse(x):
+        if isinstance(x, collections.abc.Iterable):
+            return x
+        return tuple(repeat(x, n))
+
+    return parse
+
+
+to_1tuple = _ntuple(1)
+to_2tuple = _ntuple(2)
+to_3tuple = _ntuple(3)
+to_4tuple = _ntuple(4)
+to_ntuple = _ntuple

basicsr/archs/basicvsr_arch.py

@@ -0,0 +1,336 @@
+import torch
+from torch import nn as nn
+from torch.nn import functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .arch_util import ResidualBlockNoBN, flow_warp, make_layer
+from .edvr_arch import PCDAlignment, TSAFusion
+from .spynet_arch import SpyNet
+
+
+@ARCH_REGISTRY.register()
+class BasicVSR(nn.Module):
+    """A recurrent network for video SR. Now only x4 is supported.
+
+    Args:
+        num_feat (int): Number of channels. Default: 64.
+        num_block (int): Number of residual blocks for each branch. Default: 15
+        spynet_path (str): Path to the pretrained weights of SPyNet. Default: None.
+    """
+
+    def __init__(self, num_feat=64, num_block=15, spynet_path=None):
+        super().__init__()
+        self.num_feat = num_feat
+
+        # alignment
+        self.spynet = SpyNet(spynet_path)
+
+        # propagation
+        self.backward_trunk = ConvResidualBlocks(num_feat + 3, num_feat, num_block)
+        self.forward_trunk = ConvResidualBlocks(num_feat + 3, num_feat, num_block)
+
+        # reconstruction
+        self.fusion = nn.Conv2d(num_feat * 2, num_feat, 1, 1, 0, bias=True)
+        self.upconv1 = nn.Conv2d(num_feat, num_feat * 4, 3, 1, 1, bias=True)
+        self.upconv2 = nn.Conv2d(num_feat, 64 * 4, 3, 1, 1, bias=True)
+        self.conv_hr = nn.Conv2d(64, 64, 3, 1, 1)
+        self.conv_last = nn.Conv2d(64, 3, 3, 1, 1)
+
+        self.pixel_shuffle = nn.PixelShuffle(2)
+
+        # activation functions
+        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+
+    def get_flow(self, x):
+        b, n, c, h, w = x.size()
+
+        x_1 = x[:, :-1, :, :, :].reshape(-1, c, h, w)
+        x_2 = x[:, 1:, :, :, :].reshape(-1, c, h, w)
+
+        flows_backward = self.spynet(x_1, x_2).view(b, n - 1, 2, h, w)
+        flows_forward = self.spynet(x_2, x_1).view(b, n - 1, 2, h, w)
+
+        return flows_forward, flows_backward
+
+    def forward(self, x):
+        """Forward function of BasicVSR.
+
+        Args:
+            x: Input frames with shape (b, n, c, h, w). n is the temporal dimension / number of frames.
+        """
+        flows_forward, flows_backward = self.get_flow(x)
+        b, n, _, h, w = x.size()
+
+        # backward branch
+        out_l = []
+        feat_prop = x.new_zeros(b, self.num_feat, h, w)
+        for i in range(n - 1, -1, -1):
+            x_i = x[:, i, :, :, :]
+            if i < n - 1:
+                flow = flows_backward[:, i, :, :, :]
+                feat_prop = flow_warp(feat_prop, flow.permute(0, 2, 3, 1))
+            feat_prop = torch.cat([x_i, feat_prop], dim=1)
+            feat_prop = self.backward_trunk(feat_prop)
+            out_l.insert(0, feat_prop)
+
+        # forward branch
+        feat_prop = torch.zeros_like(feat_prop)
+        for i in range(0, n):
+            x_i = x[:, i, :, :, :]
+            if i > 0:
+                flow = flows_forward[:, i - 1, :, :, :]
+                feat_prop = flow_warp(feat_prop, flow.permute(0, 2, 3, 1))
+
+            feat_prop = torch.cat([x_i, feat_prop], dim=1)
+            feat_prop = self.forward_trunk(feat_prop)
+
+            # upsample
+            out = torch.cat([out_l[i], feat_prop], dim=1)
+            out = self.lrelu(self.fusion(out))
+            out = self.lrelu(self.pixel_shuffle(self.upconv1(out)))
+            out = self.lrelu(self.pixel_shuffle(self.upconv2(out)))
+            out = self.lrelu(self.conv_hr(out))
+            out = self.conv_last(out)
+            base = F.interpolate(x_i, scale_factor=4, mode='bilinear', align_corners=False)
+            out += base
+            out_l[i] = out
+
+        return torch.stack(out_l, dim=1)
+
+
+class ConvResidualBlocks(nn.Module):
+    """Conv and residual block used in BasicVSR.
+
+    Args:
+        num_in_ch (int): Number of input channels. Default: 3.
+        num_out_ch (int): Number of output channels. Default: 64.
+        num_block (int): Number of residual blocks. Default: 15.
+    """
+
+    def __init__(self, num_in_ch=3, num_out_ch=64, num_block=15):
+        super().__init__()
+        self.main = nn.Sequential(
+            nn.Conv2d(num_in_ch, num_out_ch, 3, 1, 1, bias=True), nn.LeakyReLU(negative_slope=0.1, inplace=True),
+            make_layer(ResidualBlockNoBN, num_block, num_feat=num_out_ch))
+
+    def forward(self, fea):
+        return self.main(fea)
+
+
+@ARCH_REGISTRY.register()
+class IconVSR(nn.Module):
+    """IconVSR, proposed also in the BasicVSR paper.
+
+    Args:
+        num_feat (int): Number of channels. Default: 64.
+        num_block (int): Number of residual blocks for each branch. Default: 15.
+        keyframe_stride (int): Keyframe stride. Default: 5.
+        temporal_padding (int): Temporal padding. Default: 2.
+        spynet_path (str): Path to the pretrained weights of SPyNet. Default: None.
+        edvr_path (str): Path to the pretrained EDVR model. Default: None.
+    """
+
+    def __init__(self,
+                 num_feat=64,
+                 num_block=15,
+                 keyframe_stride=5,
+                 temporal_padding=2,
+                 spynet_path=None,
+                 edvr_path=None):
+        super().__init__()
+
+        self.num_feat = num_feat
+        self.temporal_padding = temporal_padding
+        self.keyframe_stride = keyframe_stride
+
+        # keyframe_branch
+        self.edvr = EDVRFeatureExtractor(temporal_padding * 2 + 1, num_feat, edvr_path)
+        # alignment
+        self.spynet = SpyNet(spynet_path)
+
+        # propagation
+        self.backward_fusion = nn.Conv2d(2 * num_feat, num_feat, 3, 1, 1, bias=True)
+        self.backward_trunk = ConvResidualBlocks(num_feat + 3, num_feat, num_block)
+
+        self.forward_fusion = nn.Conv2d(2 * num_feat, num_feat, 3, 1, 1, bias=True)
+        self.forward_trunk = ConvResidualBlocks(2 * num_feat + 3, num_feat, num_block)
+
+        # reconstruction
+        self.upconv1 = nn.Conv2d(num_feat, num_feat * 4, 3, 1, 1, bias=True)
+        self.upconv2 = nn.Conv2d(num_feat, 64 * 4, 3, 1, 1, bias=True)
+        self.conv_hr = nn.Conv2d(64, 64, 3, 1, 1)
+        self.conv_last = nn.Conv2d(64, 3, 3, 1, 1)
+
+        self.pixel_shuffle = nn.PixelShuffle(2)
+
+        # activation functions
+        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+
+    def pad_spatial(self, x):
+        """Apply padding spatially.
+
+        Since the PCD module in EDVR requires that the resolution is a multiple
+        of 4, we apply padding to the input LR images if their resolution is
+        not divisible by 4.
+
+        Args:
+            x (Tensor): Input LR sequence with shape (n, t, c, h, w).
+        Returns:
+            Tensor: Padded LR sequence with shape (n, t, c, h_pad, w_pad).
+        """
+        n, t, c, h, w = x.size()
+
+        pad_h = (4 - h % 4) % 4
+        pad_w = (4 - w % 4) % 4
+
+        # padding
+        x = x.view(-1, c, h, w)
+        x = F.pad(x, [0, pad_w, 0, pad_h], mode='reflect')
+
+        return x.view(n, t, c, h + pad_h, w + pad_w)
+
+    def get_flow(self, x):
+        b, n, c, h, w = x.size()
+
+        x_1 = x[:, :-1, :, :, :].reshape(-1, c, h, w)
+        x_2 = x[:, 1:, :, :, :].reshape(-1, c, h, w)
+
+        flows_backward = self.spynet(x_1, x_2).view(b, n - 1, 2, h, w)
+        flows_forward = self.spynet(x_2, x_1).view(b, n - 1, 2, h, w)
+
+        return flows_forward, flows_backward
+
+    def get_keyframe_feature(self, x, keyframe_idx):
+        if self.temporal_padding == 2:
+            x = [x[:, [4, 3]], x, x[:, [-4, -5]]]
+        elif self.temporal_padding == 3:
+            x = [x[:, [6, 5, 4]], x, x[:, [-5, -6, -7]]]
+        x = torch.cat(x, dim=1)
+
+        num_frames = 2 * self.temporal_padding + 1
+        feats_keyframe = {}
+        for i in keyframe_idx:
+            feats_keyframe[i] = self.edvr(x[:, i:i + num_frames].contiguous())
+        return feats_keyframe
+
+    def forward(self, x):
+        b, n, _, h_input, w_input = x.size()
+
+        x = self.pad_spatial(x)
+        h, w = x.shape[3:]
+
+        keyframe_idx = list(range(0, n, self.keyframe_stride))
+        if keyframe_idx[-1] != n - 1:
+            keyframe_idx.append(n - 1)  # last frame is a keyframe
+
+        # compute flow and keyframe features
+        flows_forward, flows_backward = self.get_flow(x)
+        feats_keyframe = self.get_keyframe_feature(x, keyframe_idx)
+
+        # backward branch
+        out_l = []
+        feat_prop = x.new_zeros(b, self.num_feat, h, w)
+        for i in range(n - 1, -1, -1):
+            x_i = x[:, i, :, :, :]
+            if i < n - 1:
+                flow = flows_backward[:, i, :, :, :]
+                feat_prop = flow_warp(feat_prop, flow.permute(0, 2, 3, 1))
+            if i in keyframe_idx:
+                feat_prop = torch.cat([feat_prop, feats_keyframe[i]], dim=1)
+                feat_prop = self.backward_fusion(feat_prop)
+            feat_prop = torch.cat([x_i, feat_prop], dim=1)
+            feat_prop = self.backward_trunk(feat_prop)
+            out_l.insert(0, feat_prop)
+
+        # forward branch
+        feat_prop = torch.zeros_like(feat_prop)
+        for i in range(0, n):
+            x_i = x[:, i, :, :, :]
+            if i > 0:
+                flow = flows_forward[:, i - 1, :, :, :]
+                feat_prop = flow_warp(feat_prop, flow.permute(0, 2, 3, 1))
+            if i in keyframe_idx:
+                feat_prop = torch.cat([feat_prop, feats_keyframe[i]], dim=1)
+                feat_prop = self.forward_fusion(feat_prop)
+
+            feat_prop = torch.cat([x_i, out_l[i], feat_prop], dim=1)
+            feat_prop = self.forward_trunk(feat_prop)
+
+            # upsample
+            out = self.lrelu(self.pixel_shuffle(self.upconv1(feat_prop)))
+            out = self.lrelu(self.pixel_shuffle(self.upconv2(out)))
+            out = self.lrelu(self.conv_hr(out))
+            out = self.conv_last(out)
+            base = F.interpolate(x_i, scale_factor=4, mode='bilinear', align_corners=False)
+            out += base
+            out_l[i] = out
+
+        return torch.stack(out_l, dim=1)[..., :4 * h_input, :4 * w_input]
+
+
+class EDVRFeatureExtractor(nn.Module):
+    """EDVR feature extractor used in IconVSR.
+
+    Args:
+        num_input_frame (int): Number of input frames.
+        num_feat (int): Number of feature channels
+        load_path (str): Path to the pretrained weights of EDVR. Default: None.
+    """
+
+    def __init__(self, num_input_frame, num_feat, load_path):
+
+        super(EDVRFeatureExtractor, self).__init__()
+
+        self.center_frame_idx = num_input_frame // 2
+
+        # extract pyramid features
+        self.conv_first = nn.Conv2d(3, num_feat, 3, 1, 1)
+        self.feature_extraction = make_layer(ResidualBlockNoBN, 5, num_feat=num_feat)
+        self.conv_l2_1 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
+        self.conv_l2_2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.conv_l3_1 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
+        self.conv_l3_2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+
+        # pcd and tsa module
+        self.pcd_align = PCDAlignment(num_feat=num_feat, deformable_groups=8)
+        self.fusion = TSAFusion(num_feat=num_feat, num_frame=num_input_frame, center_frame_idx=self.center_frame_idx)
+
+        # activation function
+        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+
+        if load_path:
+            self.load_state_dict(torch.load(load_path, map_location=lambda storage, loc: storage)['params'])
+
+    def forward(self, x):
+        b, n, c, h, w = x.size()
+
+        # extract features for each frame
+        # L1
+        feat_l1 = self.lrelu(self.conv_first(x.view(-1, c, h, w)))
+        feat_l1 = self.feature_extraction(feat_l1)
+        # L2
+        feat_l2 = self.lrelu(self.conv_l2_1(feat_l1))
+        feat_l2 = self.lrelu(self.conv_l2_2(feat_l2))
+        # L3
+        feat_l3 = self.lrelu(self.conv_l3_1(feat_l2))
+        feat_l3 = self.lrelu(self.conv_l3_2(feat_l3))
+
+        feat_l1 = feat_l1.view(b, n, -1, h, w)
+        feat_l2 = feat_l2.view(b, n, -1, h // 2, w // 2)
+        feat_l3 = feat_l3.view(b, n, -1, h // 4, w // 4)
+
+        # PCD alignment
+        ref_feat_l = [  # reference feature list
+            feat_l1[:, self.center_frame_idx, :, :, :].clone(), feat_l2[:, self.center_frame_idx, :, :, :].clone(),
+            feat_l3[:, self.center_frame_idx, :, :, :].clone()
+        ]
+        aligned_feat = []
+        for i in range(n):
+            nbr_feat_l = [  # neighboring feature list
+                feat_l1[:, i, :, :, :].clone(), feat_l2[:, i, :, :, :].clone(), feat_l3[:, i, :, :, :].clone()
+            ]
+            aligned_feat.append(self.pcd_align(nbr_feat_l, ref_feat_l))
+        aligned_feat = torch.stack(aligned_feat, dim=1)  # (b, t, c, h, w)
+
+        # TSA fusion
+        return self.fusion(aligned_feat)

basicsr/archs/basicvsrpp_arch.py

@@ -0,0 +1,417 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torchvision
+import warnings
+
+from basicsr.archs.arch_util import flow_warp
+from basicsr.archs.basicvsr_arch import ConvResidualBlocks
+from basicsr.archs.spynet_arch import SpyNet
+from basicsr.ops.dcn import ModulatedDeformConvPack
+from basicsr.utils.registry import ARCH_REGISTRY
+
+
+@ARCH_REGISTRY.register()
+class BasicVSRPlusPlus(nn.Module):
+    """BasicVSR++ network structure.
+
+    Support either x4 upsampling or same size output. Since DCN is used in this
+    model, it can only be used with CUDA enabled. If CUDA is not enabled,
+    feature alignment will be skipped. Besides, we adopt the official DCN
+    implementation and the version of torch need to be higher than 1.9.
+
+    ``Paper: BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment``
+
+    Args:
+        mid_channels (int, optional): Channel number of the intermediate
+            features. Default: 64.
+        num_blocks (int, optional): The number of residual blocks in each
+            propagation branch. Default: 7.
+        max_residue_magnitude (int): The maximum magnitude of the offset
+            residue (Eq. 6 in paper). Default: 10.
+        is_low_res_input (bool, optional): Whether the input is low-resolution
+            or not. If False, the output resolution is equal to the input
+            resolution. Default: True.
+        spynet_path (str): Path to the pretrained weights of SPyNet. Default: None.
+        cpu_cache_length (int, optional): When the length of sequence is larger
+            than this value, the intermediate features are sent to CPU. This
+            saves GPU memory, but slows down the inference speed. You can
+            increase this number if you have a GPU with large memory.
+            Default: 100.
+    """
+
+    def __init__(self,
+                 mid_channels=64,
+                 num_blocks=7,
+                 max_residue_magnitude=10,
+                 is_low_res_input=True,
+                 spynet_path=None,
+                 cpu_cache_length=100):
+
+        super().__init__()
+        self.mid_channels = mid_channels
+        self.is_low_res_input = is_low_res_input
+        self.cpu_cache_length = cpu_cache_length
+
+        # optical flow
+        self.spynet = SpyNet(spynet_path)
+
+        # feature extraction module
+        if is_low_res_input:
+            self.feat_extract = ConvResidualBlocks(3, mid_channels, 5)
+        else:
+            self.feat_extract = nn.Sequential(
+                nn.Conv2d(3, mid_channels, 3, 2, 1), nn.LeakyReLU(negative_slope=0.1, inplace=True),
+                nn.Conv2d(mid_channels, mid_channels, 3, 2, 1), nn.LeakyReLU(negative_slope=0.1, inplace=True),
+                ConvResidualBlocks(mid_channels, mid_channels, 5))
+
+        # propagation branches
+        self.deform_align = nn.ModuleDict()
+        self.backbone = nn.ModuleDict()
+        modules = ['backward_1', 'forward_1', 'backward_2', 'forward_2']
+        for i, module in enumerate(modules):
+            if torch.cuda.is_available():
+                self.deform_align[module] = SecondOrderDeformableAlignment(
+                    2 * mid_channels,
+                    mid_channels,
+                    3,
+                    padding=1,
+                    deformable_groups=16,
+                    max_residue_magnitude=max_residue_magnitude)
+            self.backbone[module] = ConvResidualBlocks((2 + i) * mid_channels, mid_channels, num_blocks)
+
+        # upsampling module
+        self.reconstruction = ConvResidualBlocks(5 * mid_channels, mid_channels, 5)
+
+        self.upconv1 = nn.Conv2d(mid_channels, mid_channels * 4, 3, 1, 1, bias=True)
+        self.upconv2 = nn.Conv2d(mid_channels, 64 * 4, 3, 1, 1, bias=True)
+
+        self.pixel_shuffle = nn.PixelShuffle(2)
+
+        self.conv_hr = nn.Conv2d(64, 64, 3, 1, 1)
+        self.conv_last = nn.Conv2d(64, 3, 3, 1, 1)
+        self.img_upsample = nn.Upsample(scale_factor=4, mode='bilinear', align_corners=False)
+
+        # activation function
+        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+
+        # check if the sequence is augmented by flipping
+        self.is_mirror_extended = False
+
+        if len(self.deform_align) > 0:
+            self.is_with_alignment = True
+        else:
+            self.is_with_alignment = False
+            warnings.warn('Deformable alignment module is not added. '
+                          'Probably your CUDA is not configured correctly. DCN can only '
+                          'be used with CUDA enabled. Alignment is skipped now.')
+
+    def check_if_mirror_extended(self, lqs):
+        """Check whether the input is a mirror-extended sequence.
+
+        If mirror-extended, the i-th (i=0, ..., t-1) frame is equal to the (t-1-i)-th frame.
+
+        Args:
+            lqs (tensor): Input low quality (LQ) sequence with shape (n, t, c, h, w).
+        """
+
+        if lqs.size(1) % 2 == 0:
+            lqs_1, lqs_2 = torch.chunk(lqs, 2, dim=1)
+            if torch.norm(lqs_1 - lqs_2.flip(1)) == 0:
+                self.is_mirror_extended = True
+
+    def compute_flow(self, lqs):
+        """Compute optical flow using SPyNet for feature alignment.
+
+        Note that if the input is an mirror-extended sequence, 'flows_forward'
+        is not needed, since it is equal to 'flows_backward.flip(1)'.
+
+        Args:
+            lqs (tensor): Input low quality (LQ) sequence with
+                shape (n, t, c, h, w).
+
+        Return:
+            tuple(Tensor): Optical flow. 'flows_forward' corresponds to the flows used for forward-time propagation \
+                (current to previous). 'flows_backward' corresponds to the flows used for backward-time \
+                propagation (current to next).
+        """
+
+        n, t, c, h, w = lqs.size()
+        lqs_1 = lqs[:, :-1, :, :, :].reshape(-1, c, h, w)
+        lqs_2 = lqs[:, 1:, :, :, :].reshape(-1, c, h, w)
+
+        flows_backward = self.spynet(lqs_1, lqs_2).view(n, t - 1, 2, h, w)
+
+        if self.is_mirror_extended:  # flows_forward = flows_backward.flip(1)
+            flows_forward = flows_backward.flip(1)
+        else:
+            flows_forward = self.spynet(lqs_2, lqs_1).view(n, t - 1, 2, h, w)
+
+        if self.cpu_cache:
+            flows_backward = flows_backward.cpu()
+            flows_forward = flows_forward.cpu()
+
+        return flows_forward, flows_backward
+
+    def propagate(self, feats, flows, module_name):
+        """Propagate the latent features throughout the sequence.
+
+        Args:
+            feats dict(list[tensor]): Features from previous branches. Each
+                component is a list of tensors with shape (n, c, h, w).
+            flows (tensor): Optical flows with shape (n, t - 1, 2, h, w).
+            module_name (str): The name of the propgation branches. Can either
+                be 'backward_1', 'forward_1', 'backward_2', 'forward_2'.
+
+        Return:
+            dict(list[tensor]): A dictionary containing all the propagated \
+                features. Each key in the dictionary corresponds to a \
+                propagation branch, which is represented by a list of tensors.
+        """
+
+        n, t, _, h, w = flows.size()
+
+        frame_idx = range(0, t + 1)
+        flow_idx = range(-1, t)
+        mapping_idx = list(range(0, len(feats['spatial'])))
+        mapping_idx += mapping_idx[::-1]
+
+        if 'backward' in module_name:
+            frame_idx = frame_idx[::-1]
+            flow_idx = frame_idx
+
+        feat_prop = flows.new_zeros(n, self.mid_channels, h, w)
+        for i, idx in enumerate(frame_idx):
+            feat_current = feats['spatial'][mapping_idx[idx]]
+            if self.cpu_cache:
+                feat_current = feat_current.cuda()
+                feat_prop = feat_prop.cuda()
+            # second-order deformable alignment
+            if i > 0 and self.is_with_alignment:
+                flow_n1 = flows[:, flow_idx[i], :, :, :]
+                if self.cpu_cache:
+                    flow_n1 = flow_n1.cuda()
+
+                cond_n1 = flow_warp(feat_prop, flow_n1.permute(0, 2, 3, 1))
+
+                # initialize second-order features
+                feat_n2 = torch.zeros_like(feat_prop)
+                flow_n2 = torch.zeros_like(flow_n1)
+                cond_n2 = torch.zeros_like(cond_n1)
+
+                if i > 1:  # second-order features
+                    feat_n2 = feats[module_name][-2]
+                    if self.cpu_cache:
+                        feat_n2 = feat_n2.cuda()
+
+                    flow_n2 = flows[:, flow_idx[i - 1], :, :, :]
+                    if self.cpu_cache:
+                        flow_n2 = flow_n2.cuda()
+
+                    flow_n2 = flow_n1 + flow_warp(flow_n2, flow_n1.permute(0, 2, 3, 1))
+                    cond_n2 = flow_warp(feat_n2, flow_n2.permute(0, 2, 3, 1))
+
+                # flow-guided deformable convolution
+                cond = torch.cat([cond_n1, feat_current, cond_n2], dim=1)
+                feat_prop = torch.cat([feat_prop, feat_n2], dim=1)
+                feat_prop = self.deform_align[module_name](feat_prop, cond, flow_n1, flow_n2)
+
+            # concatenate and residual blocks
+            feat = [feat_current] + [feats[k][idx] for k in feats if k not in ['spatial', module_name]] + [feat_prop]
+            if self.cpu_cache:
+                feat = [f.cuda() for f in feat]
+
+            feat = torch.cat(feat, dim=1)
+            feat_prop = feat_prop + self.backbone[module_name](feat)
+            feats[module_name].append(feat_prop)
+
+            if self.cpu_cache:
+                feats[module_name][-1] = feats[module_name][-1].cpu()
+                torch.cuda.empty_cache()
+
+        if 'backward' in module_name:
+            feats[module_name] = feats[module_name][::-1]
+
+        return feats
+
+    def upsample(self, lqs, feats):
+        """Compute the output image given the features.
+
+        Args:
+            lqs (tensor): Input low quality (LQ) sequence with
+                shape (n, t, c, h, w).
+            feats (dict): The features from the propagation branches.
+
+        Returns:
+            Tensor: Output HR sequence with shape (n, t, c, 4h, 4w).
+        """
+
+        outputs = []
+        num_outputs = len(feats['spatial'])
+
+        mapping_idx = list(range(0, num_outputs))
+        mapping_idx += mapping_idx[::-1]
+
+        for i in range(0, lqs.size(1)):
+            hr = [feats[k].pop(0) for k in feats if k != 'spatial']
+            hr.insert(0, feats['spatial'][mapping_idx[i]])
+            hr = torch.cat(hr, dim=1)
+            if self.cpu_cache:
+                hr = hr.cuda()
+
+            hr = self.reconstruction(hr)
+            hr = self.lrelu(self.pixel_shuffle(self.upconv1(hr)))
+            hr = self.lrelu(self.pixel_shuffle(self.upconv2(hr)))
+            hr = self.lrelu(self.conv_hr(hr))
+            hr = self.conv_last(hr)
+            if self.is_low_res_input:
+                hr += self.img_upsample(lqs[:, i, :, :, :])
+            else:
+                hr += lqs[:, i, :, :, :]
+
+            if self.cpu_cache:
+                hr = hr.cpu()
+                torch.cuda.empty_cache()
+
+            outputs.append(hr)
+
+        return torch.stack(outputs, dim=1)
+
+    def forward(self, lqs):
+        """Forward function for BasicVSR++.
+
+        Args:
+            lqs (tensor): Input low quality (LQ) sequence with
+                shape (n, t, c, h, w).
+
+        Returns:
+            Tensor: Output HR sequence with shape (n, t, c, 4h, 4w).
+        """
+
+        n, t, c, h, w = lqs.size()
+
+        # whether to cache the features in CPU
+        self.cpu_cache = True if t > self.cpu_cache_length else False
+
+        if self.is_low_res_input:
+            lqs_downsample = lqs.clone()
+        else:
+            lqs_downsample = F.interpolate(
+                lqs.view(-1, c, h, w), scale_factor=0.25, mode='bicubic').view(n, t, c, h // 4, w // 4)
+
+        # check whether the input is an extended sequence
+        self.check_if_mirror_extended(lqs)
+
+        feats = {}
+        # compute spatial features
+        if self.cpu_cache:
+            feats['spatial'] = []
+            for i in range(0, t):
+                feat = self.feat_extract(lqs[:, i, :, :, :]).cpu()
+                feats['spatial'].append(feat)
+                torch.cuda.empty_cache()
+        else:
+            feats_ = self.feat_extract(lqs.view(-1, c, h, w))
+            h, w = feats_.shape[2:]
+            feats_ = feats_.view(n, t, -1, h, w)
+            feats['spatial'] = [feats_[:, i, :, :, :] for i in range(0, t)]
+
+        # compute optical flow using the low-res inputs
+        assert lqs_downsample.size(3) >= 64 and lqs_downsample.size(4) >= 64, (
+            'The height and width of low-res inputs must be at least 64, '
+            f'but got {h} and {w}.')
+        flows_forward, flows_backward = self.compute_flow(lqs_downsample)
+
+        # feature propgation
+        for iter_ in [1, 2]:
+            for direction in ['backward', 'forward']:
+                module = f'{direction}_{iter_}'
+
+                feats[module] = []
+
+                if direction == 'backward':
+                    flows = flows_backward
+                elif flows_forward is not None:
+                    flows = flows_forward
+                else:
+                    flows = flows_backward.flip(1)
+
+                feats = self.propagate(feats, flows, module)
+                if self.cpu_cache:
+                    del flows
+                    torch.cuda.empty_cache()
+
+        return self.upsample(lqs, feats)
+
+
+class SecondOrderDeformableAlignment(ModulatedDeformConvPack):
+    """Second-order deformable alignment module.
+
+    Args:
+        in_channels (int): Same as nn.Conv2d.
+        out_channels (int): Same as nn.Conv2d.
+        kernel_size (int or tuple[int]): Same as nn.Conv2d.
+        stride (int or tuple[int]): Same as nn.Conv2d.
+        padding (int or tuple[int]): Same as nn.Conv2d.
+        dilation (int or tuple[int]): Same as nn.Conv2d.
+        groups (int): Same as nn.Conv2d.
+        bias (bool or str): If specified as `auto`, it will be decided by the
+            norm_cfg. Bias will be set as True if norm_cfg is None, otherwise
+            False.
+        max_residue_magnitude (int): The maximum magnitude of the offset
+            residue (Eq. 6 in paper). Default: 10.
+    """
+
+    def __init__(self, *args, **kwargs):
+        self.max_residue_magnitude = kwargs.pop('max_residue_magnitude', 10)
+
+        super(SecondOrderDeformableAlignment, self).__init__(*args, **kwargs)
+
+        self.conv_offset = nn.Sequential(
+            nn.Conv2d(3 * self.out_channels + 4, self.out_channels, 3, 1, 1),
+            nn.LeakyReLU(negative_slope=0.1, inplace=True),
+            nn.Conv2d(self.out_channels, self.out_channels, 3, 1, 1),
+            nn.LeakyReLU(negative_slope=0.1, inplace=True),
+            nn.Conv2d(self.out_channels, self.out_channels, 3, 1, 1),
+            nn.LeakyReLU(negative_slope=0.1, inplace=True),
+            nn.Conv2d(self.out_channels, 27 * self.deformable_groups, 3, 1, 1),
+        )
+
+        self.init_offset()
+
+    def init_offset(self):
+
+        def _constant_init(module, val, bias=0):
+            if hasattr(module, 'weight') and module.weight is not None:
+                nn.init.constant_(module.weight, val)
+            if hasattr(module, 'bias') and module.bias is not None:
+                nn.init.constant_(module.bias, bias)
+
+        _constant_init(self.conv_offset[-1], val=0, bias=0)
+
+    def forward(self, x, extra_feat, flow_1, flow_2):
+        extra_feat = torch.cat([extra_feat, flow_1, flow_2], dim=1)
+        out = self.conv_offset(extra_feat)
+        o1, o2, mask = torch.chunk(out, 3, dim=1)
+
+        # offset
+        offset = self.max_residue_magnitude * torch.tanh(torch.cat((o1, o2), dim=1))
+        offset_1, offset_2 = torch.chunk(offset, 2, dim=1)
+        offset_1 = offset_1 + flow_1.flip(1).repeat(1, offset_1.size(1) // 2, 1, 1)
+        offset_2 = offset_2 + flow_2.flip(1).repeat(1, offset_2.size(1) // 2, 1, 1)
+        offset = torch.cat([offset_1, offset_2], dim=1)
+
+        # mask
+        mask = torch.sigmoid(mask)
+
+        return torchvision.ops.deform_conv2d(x, offset, self.weight, self.bias, self.stride, self.padding,
+                                             self.dilation, mask)
+
+
+# if __name__ == '__main__':
+#     spynet_path = 'experiments/pretrained_models/flownet/spynet_sintel_final-3d2a1287.pth'
+#     model = BasicVSRPlusPlus(spynet_path=spynet_path).cuda()
+#     input = torch.rand(1, 2, 3, 64, 64).cuda()
+#     output = model(input)
+#     print('===================')
+#     print(output.shape)

basicsr/archs/dfdnet_arch.py

@@ -0,0 +1,169 @@
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn.utils.spectral_norm import spectral_norm
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .dfdnet_util import AttentionBlock, Blur, MSDilationBlock, UpResBlock, adaptive_instance_normalization
+from .vgg_arch import VGGFeatureExtractor
+
+
+class SFTUpBlock(nn.Module):
+    """Spatial feature transform (SFT) with upsampling block.
+
+    Args:
+        in_channel (int): Number of input channels.
+        out_channel (int): Number of output channels.
+        kernel_size (int): Kernel size in convolutions. Default: 3.
+        padding (int): Padding in convolutions. Default: 1.
+    """
+
+    def __init__(self, in_channel, out_channel, kernel_size=3, padding=1):
+        super(SFTUpBlock, self).__init__()
+        self.conv1 = nn.Sequential(
+            Blur(in_channel),
+            spectral_norm(nn.Conv2d(in_channel, out_channel, kernel_size, padding=padding)),
+            nn.LeakyReLU(0.04, True),
+            # The official codes use two LeakyReLU here, so 0.04 for equivalent
+        )
+        self.convup = nn.Sequential(
+            nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False),
+            spectral_norm(nn.Conv2d(out_channel, out_channel, kernel_size, padding=padding)),
+            nn.LeakyReLU(0.2, True),
+        )
+
+        # for SFT scale and shift
+        self.scale_block = nn.Sequential(
+            spectral_norm(nn.Conv2d(in_channel, out_channel, 3, 1, 1)), nn.LeakyReLU(0.2, True),
+            spectral_norm(nn.Conv2d(out_channel, out_channel, 3, 1, 1)))
+        self.shift_block = nn.Sequential(
+            spectral_norm(nn.Conv2d(in_channel, out_channel, 3, 1, 1)), nn.LeakyReLU(0.2, True),
+            spectral_norm(nn.Conv2d(out_channel, out_channel, 3, 1, 1)), nn.Sigmoid())
+        # The official codes use sigmoid for shift block, do not know why
+
+    def forward(self, x, updated_feat):
+        out = self.conv1(x)
+        # SFT
+        scale = self.scale_block(updated_feat)
+        shift = self.shift_block(updated_feat)
+        out = out * scale + shift
+        # upsample
+        out = self.convup(out)
+        return out
+
+
+@ARCH_REGISTRY.register()
+class DFDNet(nn.Module):
+    """DFDNet: Deep Face Dictionary Network.
+
+    It only processes faces with 512x512 size.
+
+    Args:
+        num_feat (int): Number of feature channels.
+        dict_path (str): Path to the facial component dictionary.
+    """
+
+    def __init__(self, num_feat, dict_path):
+        super().__init__()
+        self.parts = ['left_eye', 'right_eye', 'nose', 'mouth']
+        # part_sizes: [80, 80, 50, 110]
+        channel_sizes = [128, 256, 512, 512]
+        self.feature_sizes = np.array([256, 128, 64, 32])
+        self.vgg_layers = ['relu2_2', 'relu3_4', 'relu4_4', 'conv5_4']
+        self.flag_dict_device = False
+
+        # dict
+        self.dict = torch.load(dict_path)
+
+        # vgg face extractor
+        self.vgg_extractor = VGGFeatureExtractor(
+            layer_name_list=self.vgg_layers,
+            vgg_type='vgg19',
+            use_input_norm=True,
+            range_norm=True,
+            requires_grad=False)
+
+        # attention block for fusing dictionary features and input features
+        self.attn_blocks = nn.ModuleDict()
+        for idx, feat_size in enumerate(self.feature_sizes):
+            for name in self.parts:
+                self.attn_blocks[f'{name}_{feat_size}'] = AttentionBlock(channel_sizes[idx])
+
+        # multi scale dilation block
+        self.multi_scale_dilation = MSDilationBlock(num_feat * 8, dilation=[4, 3, 2, 1])
+
+        # upsampling and reconstruction
+        self.upsample0 = SFTUpBlock(num_feat * 8, num_feat * 8)
+        self.upsample1 = SFTUpBlock(num_feat * 8, num_feat * 4)
+        self.upsample2 = SFTUpBlock(num_feat * 4, num_feat * 2)
+        self.upsample3 = SFTUpBlock(num_feat * 2, num_feat)
+        self.upsample4 = nn.Sequential(
+            spectral_norm(nn.Conv2d(num_feat, num_feat, 3, 1, 1)), nn.LeakyReLU(0.2, True), UpResBlock(num_feat),
+            UpResBlock(num_feat), nn.Conv2d(num_feat, 3, kernel_size=3, stride=1, padding=1), nn.Tanh())
+
+    def swap_feat(self, vgg_feat, updated_feat, dict_feat, location, part_name, f_size):
+        """swap the features from the dictionary."""
+        # get the original vgg features
+        part_feat = vgg_feat[:, :, location[1]:location[3], location[0]:location[2]].clone()
+        # resize original vgg features
+        part_resize_feat = F.interpolate(part_feat, dict_feat.size()[2:4], mode='bilinear', align_corners=False)
+        # use adaptive instance normalization to adjust color and illuminations
+        dict_feat = adaptive_instance_normalization(dict_feat, part_resize_feat)
+        # get similarity scores
+        similarity_score = F.conv2d(part_resize_feat, dict_feat)
+        similarity_score = F.softmax(similarity_score.view(-1), dim=0)
+        # select the most similar features in the dict (after norm)
+        select_idx = torch.argmax(similarity_score)
+        swap_feat = F.interpolate(dict_feat[select_idx:select_idx + 1], part_feat.size()[2:4])
+        # attention
+        attn = self.attn_blocks[f'{part_name}_' + str(f_size)](swap_feat - part_feat)
+        attn_feat = attn * swap_feat
+        # update features
+        updated_feat[:, :, location[1]:location[3], location[0]:location[2]] = attn_feat + part_feat
+        return updated_feat
+
+    def put_dict_to_device(self, x):
+        if self.flag_dict_device is False:
+            for k, v in self.dict.items():
+                for kk, vv in v.items():
+                    self.dict[k][kk] = vv.to(x)
+            self.flag_dict_device = True
+
+    def forward(self, x, part_locations):
+        """
+        Now only support testing with batch size = 0.
+
+        Args:
+            x (Tensor): Input faces with shape (b, c, 512, 512).
+            part_locations (list[Tensor]): Part locations.
+        """
+        self.put_dict_to_device(x)
+        # extract vggface features
+        vgg_features = self.vgg_extractor(x)
+        # update vggface features using the dictionary for each part
+        updated_vgg_features = []
+        batch = 0  # only supports testing with batch size = 0
+        for vgg_layer, f_size in zip(self.vgg_layers, self.feature_sizes):
+            dict_features = self.dict[f'{f_size}']
+            vgg_feat = vgg_features[vgg_layer]
+            updated_feat = vgg_feat.clone()
+
+            # swap features from dictionary
+            for part_idx, part_name in enumerate(self.parts):
+                location = (part_locations[part_idx][batch] // (512 / f_size)).int()
+                updated_feat = self.swap_feat(vgg_feat, updated_feat, dict_features[part_name], location, part_name,
+                                              f_size)
+
+            updated_vgg_features.append(updated_feat)
+
+        vgg_feat_dilation = self.multi_scale_dilation(vgg_features['conv5_4'])
+        # use updated vgg features to modulate the upsampled features with
+        # SFT (Spatial Feature Transform) scaling and shifting manner.
+        upsampled_feat = self.upsample0(vgg_feat_dilation, updated_vgg_features[3])
+        upsampled_feat = self.upsample1(upsampled_feat, updated_vgg_features[2])
+        upsampled_feat = self.upsample2(upsampled_feat, updated_vgg_features[1])
+        upsampled_feat = self.upsample3(upsampled_feat, updated_vgg_features[0])
+        out = self.upsample4(upsampled_feat)
+
+        return out

basicsr/archs/dfdnet_util.py

@@ -0,0 +1,162 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.autograd import Function
+from torch.nn.utils.spectral_norm import spectral_norm
+
+
+class BlurFunctionBackward(Function):
+
+    @staticmethod
+    def forward(ctx, grad_output, kernel, kernel_flip):
+        ctx.save_for_backward(kernel, kernel_flip)
+        grad_input = F.conv2d(grad_output, kernel_flip, padding=1, groups=grad_output.shape[1])
+        return grad_input
+
+    @staticmethod
+    def backward(ctx, gradgrad_output):
+        kernel, _ = ctx.saved_tensors
+        grad_input = F.conv2d(gradgrad_output, kernel, padding=1, groups=gradgrad_output.shape[1])
+        return grad_input, None, None
+
+
+class BlurFunction(Function):
+
+    @staticmethod
+    def forward(ctx, x, kernel, kernel_flip):
+        ctx.save_for_backward(kernel, kernel_flip)
+        output = F.conv2d(x, kernel, padding=1, groups=x.shape[1])
+        return output
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        kernel, kernel_flip = ctx.saved_tensors
+        grad_input = BlurFunctionBackward.apply(grad_output, kernel, kernel_flip)
+        return grad_input, None, None
+
+
+blur = BlurFunction.apply
+
+
+class Blur(nn.Module):
+
+    def __init__(self, channel):
+        super().__init__()
+        kernel = torch.tensor([[1, 2, 1], [2, 4, 2], [1, 2, 1]], dtype=torch.float32)
+        kernel = kernel.view(1, 1, 3, 3)
+        kernel = kernel / kernel.sum()
+        kernel_flip = torch.flip(kernel, [2, 3])
+
+        self.kernel = kernel.repeat(channel, 1, 1, 1)
+        self.kernel_flip = kernel_flip.repeat(channel, 1, 1, 1)
+
+    def forward(self, x):
+        return blur(x, self.kernel.type_as(x), self.kernel_flip.type_as(x))
+
+
+def calc_mean_std(feat, eps=1e-5):
+    """Calculate mean and std for adaptive_instance_normalization.
+
+    Args:
+        feat (Tensor): 4D tensor.
+        eps (float): A small value added to the variance to avoid
+            divide-by-zero. Default: 1e-5.
+    """
+    size = feat.size()
+    assert len(size) == 4, 'The input feature should be 4D tensor.'
+    n, c = size[:2]
+    feat_var = feat.view(n, c, -1).var(dim=2) + eps
+    feat_std = feat_var.sqrt().view(n, c, 1, 1)
+    feat_mean = feat.view(n, c, -1).mean(dim=2).view(n, c, 1, 1)
+    return feat_mean, feat_std
+
+
+def adaptive_instance_normalization(content_feat, style_feat):
+    """Adaptive instance normalization.
+
+    Adjust the reference features to have the similar color and illuminations
+    as those in the degradate features.
+
+    Args:
+        content_feat (Tensor): The reference feature.
+        style_feat (Tensor): The degradate features.
+    """
+    size = content_feat.size()
+    style_mean, style_std = calc_mean_std(style_feat)
+    content_mean, content_std = calc_mean_std(content_feat)
+    normalized_feat = (content_feat - content_mean.expand(size)) / content_std.expand(size)
+    return normalized_feat * style_std.expand(size) + style_mean.expand(size)
+
+
+def AttentionBlock(in_channel):
+    return nn.Sequential(
+        spectral_norm(nn.Conv2d(in_channel, in_channel, 3, 1, 1)), nn.LeakyReLU(0.2, True),
+        spectral_norm(nn.Conv2d(in_channel, in_channel, 3, 1, 1)))
+
+
+def conv_block(in_channels, out_channels, kernel_size=3, stride=1, dilation=1, bias=True):
+    """Conv block used in MSDilationBlock."""
+
+    return nn.Sequential(
+        spectral_norm(
+            nn.Conv2d(
+                in_channels,
+                out_channels,
+                kernel_size=kernel_size,
+                stride=stride,
+                dilation=dilation,
+                padding=((kernel_size - 1) // 2) * dilation,
+                bias=bias)),
+        nn.LeakyReLU(0.2),
+        spectral_norm(
+            nn.Conv2d(
+                out_channels,
+                out_channels,
+                kernel_size=kernel_size,
+                stride=stride,
+                dilation=dilation,
+                padding=((kernel_size - 1) // 2) * dilation,
+                bias=bias)),
+    )
+
+
+class MSDilationBlock(nn.Module):
+    """Multi-scale dilation block."""
+
+    def __init__(self, in_channels, kernel_size=3, dilation=(1, 1, 1, 1), bias=True):
+        super(MSDilationBlock, self).__init__()
+
+        self.conv_blocks = nn.ModuleList()
+        for i in range(4):
+            self.conv_blocks.append(conv_block(in_channels, in_channels, kernel_size, dilation=dilation[i], bias=bias))
+        self.conv_fusion = spectral_norm(
+            nn.Conv2d(
+                in_channels * 4,
+                in_channels,
+                kernel_size=kernel_size,
+                stride=1,
+                padding=(kernel_size - 1) // 2,
+                bias=bias))
+
+    def forward(self, x):
+        out = []
+        for i in range(4):
+            out.append(self.conv_blocks[i](x))
+        out = torch.cat(out, 1)
+        out = self.conv_fusion(out) + x
+        return out
+
+
+class UpResBlock(nn.Module):
+
+    def __init__(self, in_channel):
+        super(UpResBlock, self).__init__()
+        self.body = nn.Sequential(
+            nn.Conv2d(in_channel, in_channel, 3, 1, 1),
+            nn.LeakyReLU(0.2, True),
+            nn.Conv2d(in_channel, in_channel, 3, 1, 1),
+        )
+
+    def forward(self, x):
+        out = x + self.body(x)
+        return out

basicsr/archs/discriminator_arch.py

@@ -0,0 +1,150 @@
+from torch import nn as nn
+from torch.nn import functional as F
+from torch.nn.utils import spectral_norm
+
+from basicsr.utils.registry import ARCH_REGISTRY
+
+
+@ARCH_REGISTRY.register()
+class VGGStyleDiscriminator(nn.Module):
+    """VGG style discriminator with input size 128 x 128 or 256 x 256.
+
+    It is used to train SRGAN, ESRGAN, and VideoGAN.
+
+    Args:
+        num_in_ch (int): Channel number of inputs. Default: 3.
+        num_feat (int): Channel number of base intermediate features.Default: 64.
+    """
+
+    def __init__(self, num_in_ch, num_feat, input_size=128):
+        super(VGGStyleDiscriminator, self).__init__()
+        self.input_size = input_size
+        assert self.input_size == 128 or self.input_size == 256, (
+            f'input size must be 128 or 256, but received {input_size}')
+
+        self.conv0_0 = nn.Conv2d(num_in_ch, num_feat, 3, 1, 1, bias=True)
+        self.conv0_1 = nn.Conv2d(num_feat, num_feat, 4, 2, 1, bias=False)
+        self.bn0_1 = nn.BatchNorm2d(num_feat, affine=True)
+
+        self.conv1_0 = nn.Conv2d(num_feat, num_feat * 2, 3, 1, 1, bias=False)
+        self.bn1_0 = nn.BatchNorm2d(num_feat * 2, affine=True)
+        self.conv1_1 = nn.Conv2d(num_feat * 2, num_feat * 2, 4, 2, 1, bias=False)
+        self.bn1_1 = nn.BatchNorm2d(num_feat * 2, affine=True)
+
+        self.conv2_0 = nn.Conv2d(num_feat * 2, num_feat * 4, 3, 1, 1, bias=False)
+        self.bn2_0 = nn.BatchNorm2d(num_feat * 4, affine=True)
+        self.conv2_1 = nn.Conv2d(num_feat * 4, num_feat * 4, 4, 2, 1, bias=False)
+        self.bn2_1 = nn.BatchNorm2d(num_feat * 4, affine=True)
+
+        self.conv3_0 = nn.Conv2d(num_feat * 4, num_feat * 8, 3, 1, 1, bias=False)
+        self.bn3_0 = nn.BatchNorm2d(num_feat * 8, affine=True)
+        self.conv3_1 = nn.Conv2d(num_feat * 8, num_feat * 8, 4, 2, 1, bias=False)
+        self.bn3_1 = nn.BatchNorm2d(num_feat * 8, affine=True)
+
+        self.conv4_0 = nn.Conv2d(num_feat * 8, num_feat * 8, 3, 1, 1, bias=False)
+        self.bn4_0 = nn.BatchNorm2d(num_feat * 8, affine=True)
+        self.conv4_1 = nn.Conv2d(num_feat * 8, num_feat * 8, 4, 2, 1, bias=False)
+        self.bn4_1 = nn.BatchNorm2d(num_feat * 8, affine=True)
+
+        if self.input_size == 256:
+            self.conv5_0 = nn.Conv2d(num_feat * 8, num_feat * 8, 3, 1, 1, bias=False)
+            self.bn5_0 = nn.BatchNorm2d(num_feat * 8, affine=True)
+            self.conv5_1 = nn.Conv2d(num_feat * 8, num_feat * 8, 4, 2, 1, bias=False)
+            self.bn5_1 = nn.BatchNorm2d(num_feat * 8, affine=True)
+
+        self.linear1 = nn.Linear(num_feat * 8 * 4 * 4, 100)
+        self.linear2 = nn.Linear(100, 1)
+
+        # activation function
+        self.lrelu = nn.LeakyReLU(negative_slope=0.2, inplace=True)
+
+    def forward(self, x):
+        assert x.size(2) == self.input_size, (f'Input size must be identical to input_size, but received {x.size()}.')
+
+        feat = self.lrelu(self.conv0_0(x))
+        feat = self.lrelu(self.bn0_1(self.conv0_1(feat)))  # output spatial size: /2
+
+        feat = self.lrelu(self.bn1_0(self.conv1_0(feat)))
+        feat = self.lrelu(self.bn1_1(self.conv1_1(feat)))  # output spatial size: /4
+
+        feat = self.lrelu(self.bn2_0(self.conv2_0(feat)))
+        feat = self.lrelu(self.bn2_1(self.conv2_1(feat)))  # output spatial size: /8
+
+        feat = self.lrelu(self.bn3_0(self.conv3_0(feat)))
+        feat = self.lrelu(self.bn3_1(self.conv3_1(feat)))  # output spatial size: /16
+
+        feat = self.lrelu(self.bn4_0(self.conv4_0(feat)))
+        feat = self.lrelu(self.bn4_1(self.conv4_1(feat)))  # output spatial size: /32
+
+        if self.input_size == 256:
+            feat = self.lrelu(self.bn5_0(self.conv5_0(feat)))
+            feat = self.lrelu(self.bn5_1(self.conv5_1(feat)))  # output spatial size: / 64
+
+        # spatial size: (4, 4)
+        feat = feat.view(feat.size(0), -1)
+        feat = self.lrelu(self.linear1(feat))
+        out = self.linear2(feat)
+        return out
+
+
+@ARCH_REGISTRY.register(suffix='basicsr')
+class UNetDiscriminatorSN(nn.Module):
+    """Defines a U-Net discriminator with spectral normalization (SN)
+
+    It is used in Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data.
+
+    Arg:
+        num_in_ch (int): Channel number of inputs. Default: 3.
+        num_feat (int): Channel number of base intermediate features. Default: 64.
+        skip_connection (bool): Whether to use skip connections between U-Net. Default: True.
+    """
+
+    def __init__(self, num_in_ch, num_feat=64, skip_connection=True):
+        super(UNetDiscriminatorSN, self).__init__()
+        self.skip_connection = skip_connection
+        norm = spectral_norm
+        # the first convolution
+        self.conv0 = nn.Conv2d(num_in_ch, num_feat, kernel_size=3, stride=1, padding=1)
+        # downsample
+        self.conv1 = norm(nn.Conv2d(num_feat, num_feat * 2, 4, 2, 1, bias=False))
+        self.conv2 = norm(nn.Conv2d(num_feat * 2, num_feat * 4, 4, 2, 1, bias=False))
+        self.conv3 = norm(nn.Conv2d(num_feat * 4, num_feat * 8, 4, 2, 1, bias=False))
+        # upsample
+        self.conv4 = norm(nn.Conv2d(num_feat * 8, num_feat * 4, 3, 1, 1, bias=False))
+        self.conv5 = norm(nn.Conv2d(num_feat * 4, num_feat * 2, 3, 1, 1, bias=False))
+        self.conv6 = norm(nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1, bias=False))
+        # extra convolutions
+        self.conv7 = norm(nn.Conv2d(num_feat, num_feat, 3, 1, 1, bias=False))
+        self.conv8 = norm(nn.Conv2d(num_feat, num_feat, 3, 1, 1, bias=False))
+        self.conv9 = nn.Conv2d(num_feat, 1, 3, 1, 1)
+
+    def forward(self, x):
+        # downsample
+        x0 = F.leaky_relu(self.conv0(x), negative_slope=0.2, inplace=True)
+        x1 = F.leaky_relu(self.conv1(x0), negative_slope=0.2, inplace=True)
+        x2 = F.leaky_relu(self.conv2(x1), negative_slope=0.2, inplace=True)
+        x3 = F.leaky_relu(self.conv3(x2), negative_slope=0.2, inplace=True)
+
+        # upsample
+        x3 = F.interpolate(x3, scale_factor=2, mode='bilinear', align_corners=False)
+        x4 = F.leaky_relu(self.conv4(x3), negative_slope=0.2, inplace=True)
+
+        if self.skip_connection:
+            x4 = x4 + x2
+        x4 = F.interpolate(x4, scale_factor=2, mode='bilinear', align_corners=False)
+        x5 = F.leaky_relu(self.conv5(x4), negative_slope=0.2, inplace=True)
+
+        if self.skip_connection:
+            x5 = x5 + x1
+        x5 = F.interpolate(x5, scale_factor=2, mode='bilinear', align_corners=False)
+        x6 = F.leaky_relu(self.conv6(x5), negative_slope=0.2, inplace=True)
+
+        if self.skip_connection:
+            x6 = x6 + x0
+
+        # extra convolutions
+        out = F.leaky_relu(self.conv7(x6), negative_slope=0.2, inplace=True)
+        out = F.leaky_relu(self.conv8(out), negative_slope=0.2, inplace=True)
+        out = self.conv9(out)
+
+        return out

basicsr/archs/duf_arch.py

@@ -0,0 +1,276 @@
+import numpy as np
+import torch
+from torch import nn as nn
+from torch.nn import functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+
+
+class DenseBlocksTemporalReduce(nn.Module):
+    """A concatenation of 3 dense blocks with reduction in temporal dimension.
+
+    Note that the output temporal dimension is 6 fewer the input temporal dimension, since there are 3 blocks.
+
+    Args:
+        num_feat (int): Number of channels in the blocks. Default: 64.
+        num_grow_ch (int): Growing factor of the dense blocks. Default: 32
+        adapt_official_weights (bool): Whether to adapt the weights translated from the official implementation.
+            Set to false if you want to train from scratch. Default: False.
+    """
+
+    def __init__(self, num_feat=64, num_grow_ch=32, adapt_official_weights=False):
+        super(DenseBlocksTemporalReduce, self).__init__()
+        if adapt_official_weights:
+            eps = 1e-3
+            momentum = 1e-3
+        else:  # pytorch default values
+            eps = 1e-05
+            momentum = 0.1
+
+        self.temporal_reduce1 = nn.Sequential(
+            nn.BatchNorm3d(num_feat, eps=eps, momentum=momentum), nn.ReLU(inplace=True),
+            nn.Conv3d(num_feat, num_feat, (1, 1, 1), stride=(1, 1, 1), padding=(0, 0, 0), bias=True),
+            nn.BatchNorm3d(num_feat, eps=eps, momentum=momentum), nn.ReLU(inplace=True),
+            nn.Conv3d(num_feat, num_grow_ch, (3, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=True))
+
+        self.temporal_reduce2 = nn.Sequential(
+            nn.BatchNorm3d(num_feat + num_grow_ch, eps=eps, momentum=momentum), nn.ReLU(inplace=True),
+            nn.Conv3d(
+                num_feat + num_grow_ch,
+                num_feat + num_grow_ch, (1, 1, 1),
+                stride=(1, 1, 1),
+                padding=(0, 0, 0),
+                bias=True), nn.BatchNorm3d(num_feat + num_grow_ch, eps=eps, momentum=momentum), nn.ReLU(inplace=True),
+            nn.Conv3d(num_feat + num_grow_ch, num_grow_ch, (3, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=True))
+
+        self.temporal_reduce3 = nn.Sequential(
+            nn.BatchNorm3d(num_feat + 2 * num_grow_ch, eps=eps, momentum=momentum), nn.ReLU(inplace=True),
+            nn.Conv3d(
+                num_feat + 2 * num_grow_ch,
+                num_feat + 2 * num_grow_ch, (1, 1, 1),
+                stride=(1, 1, 1),
+                padding=(0, 0, 0),
+                bias=True), nn.BatchNorm3d(num_feat + 2 * num_grow_ch, eps=eps, momentum=momentum),
+            nn.ReLU(inplace=True),
+            nn.Conv3d(
+                num_feat + 2 * num_grow_ch, num_grow_ch, (3, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=True))
+
+    def forward(self, x):
+        """
+        Args:
+            x (Tensor): Input tensor with shape (b, num_feat, t, h, w).
+
+        Returns:
+            Tensor: Output with shape (b, num_feat + num_grow_ch * 3, 1, h, w).
+        """
+        x1 = self.temporal_reduce1(x)
+        x1 = torch.cat((x[:, :, 1:-1, :, :], x1), 1)
+
+        x2 = self.temporal_reduce2(x1)
+        x2 = torch.cat((x1[:, :, 1:-1, :, :], x2), 1)
+
+        x3 = self.temporal_reduce3(x2)
+        x3 = torch.cat((x2[:, :, 1:-1, :, :], x3), 1)
+
+        return x3
+
+
+class DenseBlocks(nn.Module):
+    """ A concatenation of N dense blocks.
+
+    Args:
+        num_feat (int): Number of channels in the blocks. Default: 64.
+        num_grow_ch (int): Growing factor of the dense blocks. Default: 32.
+        num_block (int): Number of dense blocks. The values are:
+            DUF-S (16 layers): 3
+            DUF-M (18 layers): 9
+            DUF-L (52 layers): 21
+        adapt_official_weights (bool): Whether to adapt the weights translated from the official implementation.
+            Set to false if you want to train from scratch. Default: False.
+    """
+
+    def __init__(self, num_block, num_feat=64, num_grow_ch=16, adapt_official_weights=False):
+        super(DenseBlocks, self).__init__()
+        if adapt_official_weights:
+            eps = 1e-3
+            momentum = 1e-3
+        else:  # pytorch default values
+            eps = 1e-05
+            momentum = 0.1
+
+        self.dense_blocks = nn.ModuleList()
+        for i in range(0, num_block):
+            self.dense_blocks.append(
+                nn.Sequential(
+                    nn.BatchNorm3d(num_feat + i * num_grow_ch, eps=eps, momentum=momentum), nn.ReLU(inplace=True),
+                    nn.Conv3d(
+                        num_feat + i * num_grow_ch,
+                        num_feat + i * num_grow_ch, (1, 1, 1),
+                        stride=(1, 1, 1),
+                        padding=(0, 0, 0),
+                        bias=True), nn.BatchNorm3d(num_feat + i * num_grow_ch, eps=eps, momentum=momentum),
+                    nn.ReLU(inplace=True),
+                    nn.Conv3d(
+                        num_feat + i * num_grow_ch,
+                        num_grow_ch, (3, 3, 3),
+                        stride=(1, 1, 1),
+                        padding=(1, 1, 1),
+                        bias=True)))
+
+    def forward(self, x):
+        """
+        Args:
+            x (Tensor): Input tensor with shape (b, num_feat, t, h, w).
+
+        Returns:
+            Tensor: Output with shape (b, num_feat + num_block * num_grow_ch, t, h, w).
+        """
+        for i in range(0, len(self.dense_blocks)):
+            y = self.dense_blocks[i](x)
+            x = torch.cat((x, y), 1)
+        return x
+
+
+class DynamicUpsamplingFilter(nn.Module):
+    """Dynamic upsampling filter used in DUF.
+
+    Reference: https://github.com/yhjo09/VSR-DUF
+
+    It only supports input with 3 channels. And it applies the same filters to 3 channels.
+
+    Args:
+        filter_size (tuple): Filter size of generated filters. The shape is (kh, kw). Default: (5, 5).
+    """
+
+    def __init__(self, filter_size=(5, 5)):
+        super(DynamicUpsamplingFilter, self).__init__()
+        if not isinstance(filter_size, tuple):
+            raise TypeError(f'The type of filter_size must be tuple, but got type{filter_size}')
+        if len(filter_size) != 2:
+            raise ValueError(f'The length of filter size must be 2, but got {len(filter_size)}.')
+        # generate a local expansion filter, similar to im2col
+        self.filter_size = filter_size
+        filter_prod = np.prod(filter_size)
+        expansion_filter = torch.eye(int(filter_prod)).view(filter_prod, 1, *filter_size)  # (kh*kw, 1, kh, kw)
+        self.expansion_filter = expansion_filter.repeat(3, 1, 1, 1)  # repeat for all the 3 channels
+
+    def forward(self, x, filters):
+        """Forward function for DynamicUpsamplingFilter.
+
+        Args:
+            x (Tensor): Input image with 3 channels. The shape is (n, 3, h, w).
+            filters (Tensor): Generated dynamic filters. The shape is (n, filter_prod, upsampling_square, h, w).
+                filter_prod: prod of filter kernel size, e.g., 1*5*5=25.
+                upsampling_square: similar to pixel shuffle, upsampling_square = upsampling * upsampling.
+                e.g., for x 4 upsampling, upsampling_square= 4*4 = 16
+
+        Returns:
+            Tensor: Filtered image with shape (n, 3*upsampling_square, h, w)
+        """
+        n, filter_prod, upsampling_square, h, w = filters.size()
+        kh, kw = self.filter_size
+        expanded_input = F.conv2d(
+            x, self.expansion_filter.to(x), padding=(kh // 2, kw // 2), groups=3)  # (n, 3*filter_prod, h, w)
+        expanded_input = expanded_input.view(n, 3, filter_prod, h, w).permute(0, 3, 4, 1,
+                                                                              2)  # (n, h, w, 3, filter_prod)
+        filters = filters.permute(0, 3, 4, 1, 2)  # (n, h, w, filter_prod, upsampling_square]
+        out = torch.matmul(expanded_input, filters)  # (n, h, w, 3, upsampling_square)
+        return out.permute(0, 3, 4, 1, 2).view(n, 3 * upsampling_square, h, w)
+
+
+@ARCH_REGISTRY.register()
+class DUF(nn.Module):
+    """Network architecture for DUF
+
+    ``Paper: Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation``
+
+    Reference: https://github.com/yhjo09/VSR-DUF
+
+    For all the models below, 'adapt_official_weights' is only necessary when
+    loading the weights converted from the official TensorFlow weights.
+    Please set it to False if you are training the model from scratch.
+
+    There are three models with different model size: DUF16Layers, DUF28Layers,
+    and DUF52Layers. This class is the base class for these models.
+
+    Args:
+        scale (int): The upsampling factor. Default: 4.
+        num_layer (int): The number of layers. Default: 52.
+        adapt_official_weights_weights (bool): Whether to adapt the weights
+            translated from the official implementation. Set to false if you
+            want to train from scratch. Default: False.
+    """
+
+    def __init__(self, scale=4, num_layer=52, adapt_official_weights=False):
+        super(DUF, self).__init__()
+        self.scale = scale
+        if adapt_official_weights:
+            eps = 1e-3
+            momentum = 1e-3
+        else:  # pytorch default values
+            eps = 1e-05
+            momentum = 0.1
+
+        self.conv3d1 = nn.Conv3d(3, 64, (1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=True)
+        self.dynamic_filter = DynamicUpsamplingFilter((5, 5))
+
+        if num_layer == 16:
+            num_block = 3
+            num_grow_ch = 32
+        elif num_layer == 28:
+            num_block = 9
+            num_grow_ch = 16
+        elif num_layer == 52:
+            num_block = 21
+            num_grow_ch = 16
+        else:
+            raise ValueError(f'Only supported (16, 28, 52) layers, but got {num_layer}.')
+
+        self.dense_block1 = DenseBlocks(
+            num_block=num_block, num_feat=64, num_grow_ch=num_grow_ch,
+            adapt_official_weights=adapt_official_weights)  # T = 7
+        self.dense_block2 = DenseBlocksTemporalReduce(
+            64 + num_grow_ch * num_block, num_grow_ch, adapt_official_weights=adapt_official_weights)  # T = 1
+        channels = 64 + num_grow_ch * num_block + num_grow_ch * 3
+        self.bn3d2 = nn.BatchNorm3d(channels, eps=eps, momentum=momentum)
+        self.conv3d2 = nn.Conv3d(channels, 256, (1, 3, 3), stride=(1, 1, 1), padding=(0, 1, 1), bias=True)
+
+        self.conv3d_r1 = nn.Conv3d(256, 256, (1, 1, 1), stride=(1, 1, 1), padding=(0, 0, 0), bias=True)
+        self.conv3d_r2 = nn.Conv3d(256, 3 * (scale**2), (1, 1, 1), stride=(1, 1, 1), padding=(0, 0, 0), bias=True)
+
+        self.conv3d_f1 = nn.Conv3d(256, 512, (1, 1, 1), stride=(1, 1, 1), padding=(0, 0, 0), bias=True)
+        self.conv3d_f2 = nn.Conv3d(
+            512, 1 * 5 * 5 * (scale**2), (1, 1, 1), stride=(1, 1, 1), padding=(0, 0, 0), bias=True)
+
+    def forward(self, x):
+        """
+        Args:
+            x (Tensor): Input with shape (b, 7, c, h, w)
+
+        Returns:
+            Tensor: Output with shape (b, c, h * scale, w * scale)
+        """
+        num_batches, num_imgs, _, h, w = x.size()
+
+        x = x.permute(0, 2, 1, 3, 4)  # (b, c, 7, h, w) for Conv3D
+        x_center = x[:, :, num_imgs // 2, :, :]
+
+        x = self.conv3d1(x)
+        x = self.dense_block1(x)
+        x = self.dense_block2(x)
+        x = F.relu(self.bn3d2(x), inplace=True)
+        x = F.relu(self.conv3d2(x), inplace=True)
+
+        # residual image
+        res = self.conv3d_r2(F.relu(self.conv3d_r1(x), inplace=True))
+
+        # filter
+        filter_ = self.conv3d_f2(F.relu(self.conv3d_f1(x), inplace=True))
+        filter_ = F.softmax(filter_.view(num_batches, 25, self.scale**2, h, w), dim=1)
+
+        # dynamic filter
+        out = self.dynamic_filter(x_center, filter_)
+        out += res.squeeze_(2)
+        out = F.pixel_shuffle(out, self.scale)
+
+        return out

basicsr/archs/ecbsr_arch.py

@@ -0,0 +1,275 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+
+
+class SeqConv3x3(nn.Module):
+    """The re-parameterizable block used in the ECBSR architecture.
+
+    ``Paper: Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices``
+
+    Reference: https://github.com/xindongzhang/ECBSR
+
+    Args:
+        seq_type (str): Sequence type, option: conv1x1-conv3x3 | conv1x1-sobelx | conv1x1-sobely | conv1x1-laplacian.
+        in_channels (int): Channel number of input.
+        out_channels (int): Channel number of output.
+        depth_multiplier (int): Width multiplier in the expand-and-squeeze conv. Default: 1.
+    """
+
+    def __init__(self, seq_type, in_channels, out_channels, depth_multiplier=1):
+        super(SeqConv3x3, self).__init__()
+        self.seq_type = seq_type
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+
+        if self.seq_type == 'conv1x1-conv3x3':
+            self.mid_planes = int(out_channels * depth_multiplier)
+            conv0 = torch.nn.Conv2d(self.in_channels, self.mid_planes, kernel_size=1, padding=0)
+            self.k0 = conv0.weight
+            self.b0 = conv0.bias
+
+            conv1 = torch.nn.Conv2d(self.mid_planes, self.out_channels, kernel_size=3)
+            self.k1 = conv1.weight
+            self.b1 = conv1.bias
+
+        elif self.seq_type == 'conv1x1-sobelx':
+            conv0 = torch.nn.Conv2d(self.in_channels, self.out_channels, kernel_size=1, padding=0)
+            self.k0 = conv0.weight
+            self.b0 = conv0.bias
+
+            # init scale and bias
+            scale = torch.randn(size=(self.out_channels, 1, 1, 1)) * 1e-3
+            self.scale = nn.Parameter(scale)
+            bias = torch.randn(self.out_channels) * 1e-3
+            bias = torch.reshape(bias, (self.out_channels, ))
+            self.bias = nn.Parameter(bias)
+            # init mask
+            self.mask = torch.zeros((self.out_channels, 1, 3, 3), dtype=torch.float32)
+            for i in range(self.out_channels):
+                self.mask[i, 0, 0, 0] = 1.0
+                self.mask[i, 0, 1, 0] = 2.0
+                self.mask[i, 0, 2, 0] = 1.0
+                self.mask[i, 0, 0, 2] = -1.0
+                self.mask[i, 0, 1, 2] = -2.0
+                self.mask[i, 0, 2, 2] = -1.0
+            self.mask = nn.Parameter(data=self.mask, requires_grad=False)
+
+        elif self.seq_type == 'conv1x1-sobely':
+            conv0 = torch.nn.Conv2d(self.in_channels, self.out_channels, kernel_size=1, padding=0)
+            self.k0 = conv0.weight
+            self.b0 = conv0.bias
+
+            # init scale and bias
+            scale = torch.randn(size=(self.out_channels, 1, 1, 1)) * 1e-3
+            self.scale = nn.Parameter(torch.FloatTensor(scale))
+            bias = torch.randn(self.out_channels) * 1e-3
+            bias = torch.reshape(bias, (self.out_channels, ))
+            self.bias = nn.Parameter(torch.FloatTensor(bias))
+            # init mask
+            self.mask = torch.zeros((self.out_channels, 1, 3, 3), dtype=torch.float32)
+            for i in range(self.out_channels):
+                self.mask[i, 0, 0, 0] = 1.0
+                self.mask[i, 0, 0, 1] = 2.0
+                self.mask[i, 0, 0, 2] = 1.0
+                self.mask[i, 0, 2, 0] = -1.0
+                self.mask[i, 0, 2, 1] = -2.0
+                self.mask[i, 0, 2, 2] = -1.0
+            self.mask = nn.Parameter(data=self.mask, requires_grad=False)
+
+        elif self.seq_type == 'conv1x1-laplacian':
+            conv0 = torch.nn.Conv2d(self.in_channels, self.out_channels, kernel_size=1, padding=0)
+            self.k0 = conv0.weight
+            self.b0 = conv0.bias
+
+            # init scale and bias
+            scale = torch.randn(size=(self.out_channels, 1, 1, 1)) * 1e-3
+            self.scale = nn.Parameter(torch.FloatTensor(scale))
+            bias = torch.randn(self.out_channels) * 1e-3
+            bias = torch.reshape(bias, (self.out_channels, ))
+            self.bias = nn.Parameter(torch.FloatTensor(bias))
+            # init mask
+            self.mask = torch.zeros((self.out_channels, 1, 3, 3), dtype=torch.float32)
+            for i in range(self.out_channels):
+                self.mask[i, 0, 0, 1] = 1.0
+                self.mask[i, 0, 1, 0] = 1.0
+                self.mask[i, 0, 1, 2] = 1.0
+                self.mask[i, 0, 2, 1] = 1.0
+                self.mask[i, 0, 1, 1] = -4.0
+            self.mask = nn.Parameter(data=self.mask, requires_grad=False)
+        else:
+            raise ValueError('The type of seqconv is not supported!')
+
+    def forward(self, x):
+        if self.seq_type == 'conv1x1-conv3x3':
+            # conv-1x1
+            y0 = F.conv2d(input=x, weight=self.k0, bias=self.b0, stride=1)
+            # explicitly padding with bias
+            y0 = F.pad(y0, (1, 1, 1, 1), 'constant', 0)
+            b0_pad = self.b0.view(1, -1, 1, 1)
+            y0[:, :, 0:1, :] = b0_pad
+            y0[:, :, -1:, :] = b0_pad
+            y0[:, :, :, 0:1] = b0_pad
+            y0[:, :, :, -1:] = b0_pad
+            # conv-3x3
+            y1 = F.conv2d(input=y0, weight=self.k1, bias=self.b1, stride=1)
+        else:
+            y0 = F.conv2d(input=x, weight=self.k0, bias=self.b0, stride=1)
+            # explicitly padding with bias
+            y0 = F.pad(y0, (1, 1, 1, 1), 'constant', 0)
+            b0_pad = self.b0.view(1, -1, 1, 1)
+            y0[:, :, 0:1, :] = b0_pad
+            y0[:, :, -1:, :] = b0_pad
+            y0[:, :, :, 0:1] = b0_pad
+            y0[:, :, :, -1:] = b0_pad
+            # conv-3x3
+            y1 = F.conv2d(input=y0, weight=self.scale * self.mask, bias=self.bias, stride=1, groups=self.out_channels)
+        return y1
+
+    def rep_params(self):
+        device = self.k0.get_device()
+        if device < 0:
+            device = None
+
+        if self.seq_type == 'conv1x1-conv3x3':
+            # re-param conv kernel
+            rep_weight = F.conv2d(input=self.k1, weight=self.k0.permute(1, 0, 2, 3))
+            # re-param conv bias
+            rep_bias = torch.ones(1, self.mid_planes, 3, 3, device=device) * self.b0.view(1, -1, 1, 1)
+            rep_bias = F.conv2d(input=rep_bias, weight=self.k1).view(-1, ) + self.b1
+        else:
+            tmp = self.scale * self.mask
+            k1 = torch.zeros((self.out_channels, self.out_channels, 3, 3), device=device)
+            for i in range(self.out_channels):
+                k1[i, i, :, :] = tmp[i, 0, :, :]
+            b1 = self.bias
+            # re-param conv kernel
+            rep_weight = F.conv2d(input=k1, weight=self.k0.permute(1, 0, 2, 3))
+            # re-param conv bias
+            rep_bias = torch.ones(1, self.out_channels, 3, 3, device=device) * self.b0.view(1, -1, 1, 1)
+            rep_bias = F.conv2d(input=rep_bias, weight=k1).view(-1, ) + b1
+        return rep_weight, rep_bias
+
+
+class ECB(nn.Module):
+    """The ECB block used in the ECBSR architecture.
+
+    Paper: Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices
+    Ref git repo: https://github.com/xindongzhang/ECBSR
+
+    Args:
+        in_channels (int): Channel number of input.
+        out_channels (int): Channel number of output.
+        depth_multiplier (int): Width multiplier in the expand-and-squeeze conv. Default: 1.
+        act_type (str): Activation type. Option: prelu | relu | rrelu | softplus | linear. Default: prelu.
+        with_idt (bool): Whether to use identity connection. Default: False.
+    """
+
+    def __init__(self, in_channels, out_channels, depth_multiplier, act_type='prelu', with_idt=False):
+        super(ECB, self).__init__()
+
+        self.depth_multiplier = depth_multiplier
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+        self.act_type = act_type
+
+        if with_idt and (self.in_channels == self.out_channels):
+            self.with_idt = True
+        else:
+            self.with_idt = False
+
+        self.conv3x3 = torch.nn.Conv2d(self.in_channels, self.out_channels, kernel_size=3, padding=1)
+        self.conv1x1_3x3 = SeqConv3x3('conv1x1-conv3x3', self.in_channels, self.out_channels, self.depth_multiplier)
+        self.conv1x1_sbx = SeqConv3x3('conv1x1-sobelx', self.in_channels, self.out_channels)
+        self.conv1x1_sby = SeqConv3x3('conv1x1-sobely', self.in_channels, self.out_channels)
+        self.conv1x1_lpl = SeqConv3x3('conv1x1-laplacian', self.in_channels, self.out_channels)
+
+        if self.act_type == 'prelu':
+            self.act = nn.PReLU(num_parameters=self.out_channels)
+        elif self.act_type == 'relu':
+            self.act = nn.ReLU(inplace=True)
+        elif self.act_type == 'rrelu':
+            self.act = nn.RReLU(lower=-0.05, upper=0.05)
+        elif self.act_type == 'softplus':
+            self.act = nn.Softplus()
+        elif self.act_type == 'linear':
+            pass
+        else:
+            raise ValueError('The type of activation if not support!')
+
+    def forward(self, x):
+        if self.training:
+            y = self.conv3x3(x) + self.conv1x1_3x3(x) + self.conv1x1_sbx(x) + self.conv1x1_sby(x) + self.conv1x1_lpl(x)
+            if self.with_idt:
+                y += x
+        else:
+            rep_weight, rep_bias = self.rep_params()
+            y = F.conv2d(input=x, weight=rep_weight, bias=rep_bias, stride=1, padding=1)
+        if self.act_type != 'linear':
+            y = self.act(y)
+        return y
+
+    def rep_params(self):
+        weight0, bias0 = self.conv3x3.weight, self.conv3x3.bias
+        weight1, bias1 = self.conv1x1_3x3.rep_params()
+        weight2, bias2 = self.conv1x1_sbx.rep_params()
+        weight3, bias3 = self.conv1x1_sby.rep_params()
+        weight4, bias4 = self.conv1x1_lpl.rep_params()
+        rep_weight, rep_bias = (weight0 + weight1 + weight2 + weight3 + weight4), (
+            bias0 + bias1 + bias2 + bias3 + bias4)
+
+        if self.with_idt:
+            device = rep_weight.get_device()
+            if device < 0:
+                device = None
+            weight_idt = torch.zeros(self.out_channels, self.out_channels, 3, 3, device=device)
+            for i in range(self.out_channels):
+                weight_idt[i, i, 1, 1] = 1.0
+            bias_idt = 0.0
+            rep_weight, rep_bias = rep_weight + weight_idt, rep_bias + bias_idt
+        return rep_weight, rep_bias
+
+
+@ARCH_REGISTRY.register()
+class ECBSR(nn.Module):
+    """ECBSR architecture.
+
+    Paper: Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices
+    Ref git repo: https://github.com/xindongzhang/ECBSR
+
+    Args:
+        num_in_ch (int): Channel number of inputs.
+        num_out_ch (int): Channel number of outputs.
+        num_block (int): Block number in the trunk network.
+        num_channel (int): Channel number.
+        with_idt (bool): Whether use identity in convolution layers.
+        act_type (str): Activation type.
+        scale (int): Upsampling factor.
+    """
+
+    def __init__(self, num_in_ch, num_out_ch, num_block, num_channel, with_idt, act_type, scale):
+        super(ECBSR, self).__init__()
+        self.num_in_ch = num_in_ch
+        self.scale = scale
+
+        backbone = []
+        backbone += [ECB(num_in_ch, num_channel, depth_multiplier=2.0, act_type=act_type, with_idt=with_idt)]
+        for _ in range(num_block):
+            backbone += [ECB(num_channel, num_channel, depth_multiplier=2.0, act_type=act_type, with_idt=with_idt)]
+        backbone += [
+            ECB(num_channel, num_out_ch * scale * scale, depth_multiplier=2.0, act_type='linear', with_idt=with_idt)
+        ]
+
+        self.backbone = nn.Sequential(*backbone)
+        self.upsampler = nn.PixelShuffle(scale)
+
+    def forward(self, x):
+        if self.num_in_ch > 1:
+            shortcut = torch.repeat_interleave(x, self.scale * self.scale, dim=1)
+        else:
+            shortcut = x  # will repeat the input in the channel dimension (repeat  scale * scale times)
+        y = self.backbone(x) + shortcut
+        y = self.upsampler(y)
+        return y

basicsr/archs/edsr_arch.py

@@ -0,0 +1,61 @@
+import torch
+from torch import nn as nn
+
+from basicsr.archs.arch_util import ResidualBlockNoBN, Upsample, make_layer
+from basicsr.utils.registry import ARCH_REGISTRY
+
+
+@ARCH_REGISTRY.register()
+class EDSR(nn.Module):
+    """EDSR network structure.
+
+    Paper: Enhanced Deep Residual Networks for Single Image Super-Resolution.
+    Ref git repo: https://github.com/thstkdgus35/EDSR-PyTorch
+
+    Args:
+        num_in_ch (int): Channel number of inputs.
+        num_out_ch (int): Channel number of outputs.
+        num_feat (int): Channel number of intermediate features.
+            Default: 64.
+        num_block (int): Block number in the trunk network. Default: 16.
+        upscale (int): Upsampling factor. Support 2^n and 3.
+            Default: 4.
+        res_scale (float): Used to scale the residual in residual block.
+            Default: 1.
+        img_range (float): Image range. Default: 255.
+        rgb_mean (tuple[float]): Image mean in RGB orders.
+            Default: (0.4488, 0.4371, 0.4040), calculated from DIV2K dataset.
+    """
+
+    def __init__(self,
+                 num_in_ch,
+                 num_out_ch,
+                 num_feat=64,
+                 num_block=16,
+                 upscale=4,
+                 res_scale=1,
+                 img_range=255.,
+                 rgb_mean=(0.4488, 0.4371, 0.4040)):
+        super(EDSR, self).__init__()
+
+        self.img_range = img_range
+        self.mean = torch.Tensor(rgb_mean).view(1, 3, 1, 1)
+
+        self.conv_first = nn.Conv2d(num_in_ch, num_feat, 3, 1, 1)
+        self.body = make_layer(ResidualBlockNoBN, num_block, num_feat=num_feat, res_scale=res_scale, pytorch_init=True)
+        self.conv_after_body = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.upsample = Upsample(upscale, num_feat)
+        self.conv_last = nn.Conv2d(num_feat, num_out_ch, 3, 1, 1)
+
+    def forward(self, x):
+        self.mean = self.mean.type_as(x)
+
+        x = (x - self.mean) * self.img_range
+        x = self.conv_first(x)
+        res = self.conv_after_body(self.body(x))
+        res += x
+
+        x = self.conv_last(self.upsample(res))
+        x = x / self.img_range + self.mean
+
+        return x

basicsr/archs/edvr_arch.py

@@ -0,0 +1,382 @@
+import torch
+from torch import nn as nn
+from torch.nn import functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .arch_util import DCNv2Pack, ResidualBlockNoBN, make_layer
+
+
+class PCDAlignment(nn.Module):
+    """Alignment module using Pyramid, Cascading and Deformable convolution
+    (PCD). It is used in EDVR.
+
+    ``Paper: EDVR: Video Restoration with Enhanced Deformable Convolutional Networks``
+
+    Args:
+        num_feat (int): Channel number of middle features. Default: 64.
+        deformable_groups (int): Deformable groups. Defaults: 8.
+    """
+
+    def __init__(self, num_feat=64, deformable_groups=8):
+        super(PCDAlignment, self).__init__()
+
+        # Pyramid has three levels:
+        # L3: level 3, 1/4 spatial size
+        # L2: level 2, 1/2 spatial size
+        # L1: level 1, original spatial size
+        self.offset_conv1 = nn.ModuleDict()
+        self.offset_conv2 = nn.ModuleDict()
+        self.offset_conv3 = nn.ModuleDict()
+        self.dcn_pack = nn.ModuleDict()
+        self.feat_conv = nn.ModuleDict()
+
+        # Pyramids
+        for i in range(3, 0, -1):
+            level = f'l{i}'
+            self.offset_conv1[level] = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
+            if i == 3:
+                self.offset_conv2[level] = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+            else:
+                self.offset_conv2[level] = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
+                self.offset_conv3[level] = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+            self.dcn_pack[level] = DCNv2Pack(num_feat, num_feat, 3, padding=1, deformable_groups=deformable_groups)
+
+            if i < 3:
+                self.feat_conv[level] = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
+
+        # Cascading dcn
+        self.cas_offset_conv1 = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
+        self.cas_offset_conv2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.cas_dcnpack = DCNv2Pack(num_feat, num_feat, 3, padding=1, deformable_groups=deformable_groups)
+
+        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False)
+        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+
+    def forward(self, nbr_feat_l, ref_feat_l):
+        """Align neighboring frame features to the reference frame features.
+
+        Args:
+            nbr_feat_l (list[Tensor]): Neighboring feature list. It
+                contains three pyramid levels (L1, L2, L3),
+                each with shape (b, c, h, w).
+            ref_feat_l (list[Tensor]): Reference feature list. It
+                contains three pyramid levels (L1, L2, L3),
+                each with shape (b, c, h, w).
+
+        Returns:
+            Tensor: Aligned features.
+        """
+        # Pyramids
+        upsampled_offset, upsampled_feat = None, None
+        for i in range(3, 0, -1):
+            level = f'l{i}'
+            offset = torch.cat([nbr_feat_l[i - 1], ref_feat_l[i - 1]], dim=1)
+            offset = self.lrelu(self.offset_conv1[level](offset))
+            if i == 3:
+                offset = self.lrelu(self.offset_conv2[level](offset))
+            else:
+                offset = self.lrelu(self.offset_conv2[level](torch.cat([offset, upsampled_offset], dim=1)))
+                offset = self.lrelu(self.offset_conv3[level](offset))
+
+            feat = self.dcn_pack[level](nbr_feat_l[i - 1], offset)
+            if i < 3:
+                feat = self.feat_conv[level](torch.cat([feat, upsampled_feat], dim=1))
+            if i > 1:
+                feat = self.lrelu(feat)
+
+            if i > 1:  # upsample offset and features
+                # x2: when we upsample the offset, we should also enlarge
+                # the magnitude.
+                upsampled_offset = self.upsample(offset) * 2
+                upsampled_feat = self.upsample(feat)
+
+        # Cascading
+        offset = torch.cat([feat, ref_feat_l[0]], dim=1)
+        offset = self.lrelu(self.cas_offset_conv2(self.lrelu(self.cas_offset_conv1(offset))))
+        feat = self.lrelu(self.cas_dcnpack(feat, offset))
+        return feat
+
+
+class TSAFusion(nn.Module):
+    """Temporal Spatial Attention (TSA) fusion module.
+
+    Temporal: Calculate the correlation between center frame and
+        neighboring frames;
+    Spatial: It has 3 pyramid levels, the attention is similar to SFT.
+        (SFT: Recovering realistic texture in image super-resolution by deep
+            spatial feature transform.)
+
+    Args:
+        num_feat (int): Channel number of middle features. Default: 64.
+        num_frame (int): Number of frames. Default: 5.
+        center_frame_idx (int): The index of center frame. Default: 2.
+    """
+
+    def __init__(self, num_feat=64, num_frame=5, center_frame_idx=2):
+        super(TSAFusion, self).__init__()
+        self.center_frame_idx = center_frame_idx
+        # temporal attention (before fusion conv)
+        self.temporal_attn1 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.temporal_attn2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.feat_fusion = nn.Conv2d(num_frame * num_feat, num_feat, 1, 1)
+
+        # spatial attention (after fusion conv)
+        self.max_pool = nn.MaxPool2d(3, stride=2, padding=1)
+        self.avg_pool = nn.AvgPool2d(3, stride=2, padding=1)
+        self.spatial_attn1 = nn.Conv2d(num_frame * num_feat, num_feat, 1)
+        self.spatial_attn2 = nn.Conv2d(num_feat * 2, num_feat, 1)
+        self.spatial_attn3 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.spatial_attn4 = nn.Conv2d(num_feat, num_feat, 1)
+        self.spatial_attn5 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.spatial_attn_l1 = nn.Conv2d(num_feat, num_feat, 1)
+        self.spatial_attn_l2 = nn.Conv2d(num_feat * 2, num_feat, 3, 1, 1)
+        self.spatial_attn_l3 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.spatial_attn_add1 = nn.Conv2d(num_feat, num_feat, 1)
+        self.spatial_attn_add2 = nn.Conv2d(num_feat, num_feat, 1)
+
+        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False)
+
+    def forward(self, aligned_feat):
+        """
+        Args:
+            aligned_feat (Tensor): Aligned features with shape (b, t, c, h, w).
+
+        Returns:
+            Tensor: Features after TSA with the shape (b, c, h, w).
+        """
+        b, t, c, h, w = aligned_feat.size()
+        # temporal attention
+        embedding_ref = self.temporal_attn1(aligned_feat[:, self.center_frame_idx, :, :, :].clone())
+        embedding = self.temporal_attn2(aligned_feat.view(-1, c, h, w))
+        embedding = embedding.view(b, t, -1, h, w)  # (b, t, c, h, w)
+
+        corr_l = []  # correlation list
+        for i in range(t):
+            emb_neighbor = embedding[:, i, :, :, :]
+            corr = torch.sum(emb_neighbor * embedding_ref, 1)  # (b, h, w)
+            corr_l.append(corr.unsqueeze(1))  # (b, 1, h, w)
+        corr_prob = torch.sigmoid(torch.cat(corr_l, dim=1))  # (b, t, h, w)
+        corr_prob = corr_prob.unsqueeze(2).expand(b, t, c, h, w)
+        corr_prob = corr_prob.contiguous().view(b, -1, h, w)  # (b, t*c, h, w)
+        aligned_feat = aligned_feat.view(b, -1, h, w) * corr_prob
+
+        # fusion
+        feat = self.lrelu(self.feat_fusion(aligned_feat))
+
+        # spatial attention
+        attn = self.lrelu(self.spatial_attn1(aligned_feat))
+        attn_max = self.max_pool(attn)
+        attn_avg = self.avg_pool(attn)
+        attn = self.lrelu(self.spatial_attn2(torch.cat([attn_max, attn_avg], dim=1)))
+        # pyramid levels
+        attn_level = self.lrelu(self.spatial_attn_l1(attn))
+        attn_max = self.max_pool(attn_level)
+        attn_avg = self.avg_pool(attn_level)
+        attn_level = self.lrelu(self.spatial_attn_l2(torch.cat([attn_max, attn_avg], dim=1)))
+        attn_level = self.lrelu(self.spatial_attn_l3(attn_level))
+        attn_level = self.upsample(attn_level)
+
+        attn = self.lrelu(self.spatial_attn3(attn)) + attn_level
+        attn = self.lrelu(self.spatial_attn4(attn))
+        attn = self.upsample(attn)
+        attn = self.spatial_attn5(attn)
+        attn_add = self.spatial_attn_add2(self.lrelu(self.spatial_attn_add1(attn)))
+        attn = torch.sigmoid(attn)
+
+        # after initialization, * 2 makes (attn * 2) to be close to 1.
+        feat = feat * attn * 2 + attn_add
+        return feat
+
+
+class PredeblurModule(nn.Module):
+    """Pre-dublur module.
+
+    Args:
+        num_in_ch (int): Channel number of input image. Default: 3.
+        num_feat (int): Channel number of intermediate features. Default: 64.
+        hr_in (bool): Whether the input has high resolution. Default: False.
+    """
+
+    def __init__(self, num_in_ch=3, num_feat=64, hr_in=False):
+        super(PredeblurModule, self).__init__()
+        self.hr_in = hr_in
+
+        self.conv_first = nn.Conv2d(num_in_ch, num_feat, 3, 1, 1)
+        if self.hr_in:
+            # downsample x4 by stride conv
+            self.stride_conv_hr1 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
+            self.stride_conv_hr2 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
+
+        # generate feature pyramid
+        self.stride_conv_l2 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
+        self.stride_conv_l3 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
+
+        self.resblock_l3 = ResidualBlockNoBN(num_feat=num_feat)
+        self.resblock_l2_1 = ResidualBlockNoBN(num_feat=num_feat)
+        self.resblock_l2_2 = ResidualBlockNoBN(num_feat=num_feat)
+        self.resblock_l1 = nn.ModuleList([ResidualBlockNoBN(num_feat=num_feat) for i in range(5)])
+
+        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=False)
+        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+
+    def forward(self, x):
+        feat_l1 = self.lrelu(self.conv_first(x))
+        if self.hr_in:
+            feat_l1 = self.lrelu(self.stride_conv_hr1(feat_l1))
+            feat_l1 = self.lrelu(self.stride_conv_hr2(feat_l1))
+
+        # generate feature pyramid
+        feat_l2 = self.lrelu(self.stride_conv_l2(feat_l1))
+        feat_l3 = self.lrelu(self.stride_conv_l3(feat_l2))
+
+        feat_l3 = self.upsample(self.resblock_l3(feat_l3))
+        feat_l2 = self.resblock_l2_1(feat_l2) + feat_l3
+        feat_l2 = self.upsample(self.resblock_l2_2(feat_l2))
+
+        for i in range(2):
+            feat_l1 = self.resblock_l1[i](feat_l1)
+        feat_l1 = feat_l1 + feat_l2
+        for i in range(2, 5):
+            feat_l1 = self.resblock_l1[i](feat_l1)
+        return feat_l1
+
+
+@ARCH_REGISTRY.register()
+class EDVR(nn.Module):
+    """EDVR network structure for video super-resolution.
+
+    Now only support X4 upsampling factor.
+
+    ``Paper: EDVR: Video Restoration with Enhanced Deformable Convolutional Networks``
+
+    Args:
+        num_in_ch (int): Channel number of input image. Default: 3.
+        num_out_ch (int): Channel number of output image. Default: 3.
+        num_feat (int): Channel number of intermediate features. Default: 64.
+        num_frame (int): Number of input frames. Default: 5.
+        deformable_groups (int): Deformable groups. Defaults: 8.
+        num_extract_block (int): Number of blocks for feature extraction.
+            Default: 5.
+        num_reconstruct_block (int): Number of blocks for reconstruction.
+            Default: 10.
+        center_frame_idx (int): The index of center frame. Frame counting from
+            0. Default: Middle of input frames.
+        hr_in (bool): Whether the input has high resolution. Default: False.
+        with_predeblur (bool): Whether has predeblur module.
+            Default: False.
+        with_tsa (bool): Whether has TSA module. Default: True.
+    """
+
+    def __init__(self,
+                 num_in_ch=3,
+                 num_out_ch=3,
+                 num_feat=64,
+                 num_frame=5,
+                 deformable_groups=8,
+                 num_extract_block=5,
+                 num_reconstruct_block=10,
+                 center_frame_idx=None,
+                 hr_in=False,
+                 with_predeblur=False,
+                 with_tsa=True):
+        super(EDVR, self).__init__()
+        if center_frame_idx is None:
+            self.center_frame_idx = num_frame // 2
+        else:
+            self.center_frame_idx = center_frame_idx
+        self.hr_in = hr_in
+        self.with_predeblur = with_predeblur
+        self.with_tsa = with_tsa
+
+        # extract features for each frame
+        if self.with_predeblur:
+            self.predeblur = PredeblurModule(num_feat=num_feat, hr_in=self.hr_in)
+            self.conv_1x1 = nn.Conv2d(num_feat, num_feat, 1, 1)
+        else:
+            self.conv_first = nn.Conv2d(num_in_ch, num_feat, 3, 1, 1)
+
+        # extract pyramid features
+        self.feature_extraction = make_layer(ResidualBlockNoBN, num_extract_block, num_feat=num_feat)
+        self.conv_l2_1 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
+        self.conv_l2_2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.conv_l3_1 = nn.Conv2d(num_feat, num_feat, 3, 2, 1)
+        self.conv_l3_2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+
+        # pcd and tsa module
+        self.pcd_align = PCDAlignment(num_feat=num_feat, deformable_groups=deformable_groups)
+        if self.with_tsa:
+            self.fusion = TSAFusion(num_feat=num_feat, num_frame=num_frame, center_frame_idx=self.center_frame_idx)
+        else:
+            self.fusion = nn.Conv2d(num_frame * num_feat, num_feat, 1, 1)
+
+        # reconstruction
+        self.reconstruction = make_layer(ResidualBlockNoBN, num_reconstruct_block, num_feat=num_feat)
+        # upsample
+        self.upconv1 = nn.Conv2d(num_feat, num_feat * 4, 3, 1, 1)
+        self.upconv2 = nn.Conv2d(num_feat, 64 * 4, 3, 1, 1)
+        self.pixel_shuffle = nn.PixelShuffle(2)
+        self.conv_hr = nn.Conv2d(64, 64, 3, 1, 1)
+        self.conv_last = nn.Conv2d(64, 3, 3, 1, 1)
+
+        # activation function
+        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+
+    def forward(self, x):
+        b, t, c, h, w = x.size()
+        if self.hr_in:
+            assert h % 16 == 0 and w % 16 == 0, ('The height and width must be multiple of 16.')
+        else:
+            assert h % 4 == 0 and w % 4 == 0, ('The height and width must be multiple of 4.')
+
+        x_center = x[:, self.center_frame_idx, :, :, :].contiguous()
+
+        # extract features for each frame
+        # L1
+        if self.with_predeblur:
+            feat_l1 = self.conv_1x1(self.predeblur(x.view(-1, c, h, w)))
+            if self.hr_in:
+                h, w = h // 4, w // 4
+        else:
+            feat_l1 = self.lrelu(self.conv_first(x.view(-1, c, h, w)))
+
+        feat_l1 = self.feature_extraction(feat_l1)
+        # L2
+        feat_l2 = self.lrelu(self.conv_l2_1(feat_l1))
+        feat_l2 = self.lrelu(self.conv_l2_2(feat_l2))
+        # L3
+        feat_l3 = self.lrelu(self.conv_l3_1(feat_l2))
+        feat_l3 = self.lrelu(self.conv_l3_2(feat_l3))
+
+        feat_l1 = feat_l1.view(b, t, -1, h, w)
+        feat_l2 = feat_l2.view(b, t, -1, h // 2, w // 2)
+        feat_l3 = feat_l3.view(b, t, -1, h // 4, w // 4)
+
+        # PCD alignment
+        ref_feat_l = [  # reference feature list
+            feat_l1[:, self.center_frame_idx, :, :, :].clone(), feat_l2[:, self.center_frame_idx, :, :, :].clone(),
+            feat_l3[:, self.center_frame_idx, :, :, :].clone()
+        ]
+        aligned_feat = []
+        for i in range(t):
+            nbr_feat_l = [  # neighboring feature list
+                feat_l1[:, i, :, :, :].clone(), feat_l2[:, i, :, :, :].clone(), feat_l3[:, i, :, :, :].clone()
+            ]
+            aligned_feat.append(self.pcd_align(nbr_feat_l, ref_feat_l))
+        aligned_feat = torch.stack(aligned_feat, dim=1)  # (b, t, c, h, w)
+
+        if not self.with_tsa:
+            aligned_feat = aligned_feat.view(b, -1, h, w)
+        feat = self.fusion(aligned_feat)
+
+        out = self.reconstruction(feat)
+        out = self.lrelu(self.pixel_shuffle(self.upconv1(out)))
+        out = self.lrelu(self.pixel_shuffle(self.upconv2(out)))
+        out = self.lrelu(self.conv_hr(out))
+        out = self.conv_last(out)
+        if self.hr_in:
+            base = x_center
+        else:
+            base = F.interpolate(x_center, scale_factor=4, mode='bilinear', align_corners=False)
+        out += base
+        return out

basicsr/archs/hifacegan_arch.py

@@ -0,0 +1,260 @@
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .hifacegan_util import BaseNetwork, LIPEncoder, SPADEResnetBlock, get_nonspade_norm_layer
+
+
+class SPADEGenerator(BaseNetwork):
+    """Generator with SPADEResBlock"""
+
+    def __init__(self,
+                 num_in_ch=3,
+                 num_feat=64,
+                 use_vae=False,
+                 z_dim=256,
+                 crop_size=512,
+                 norm_g='spectralspadesyncbatch3x3',
+                 is_train=True,
+                 init_train_phase=3):  # progressive training disabled
+        super().__init__()
+        self.nf = num_feat
+        self.input_nc = num_in_ch
+        self.is_train = is_train
+        self.train_phase = init_train_phase
+
+        self.scale_ratio = 5  # hardcoded now
+        self.sw = crop_size // (2**self.scale_ratio)
+        self.sh = self.sw  # 20210519: By default use square image, aspect_ratio = 1.0
+
+        if use_vae:
+            # In case of VAE, we will sample from random z vector
+            self.fc = nn.Linear(z_dim, 16 * self.nf * self.sw * self.sh)
+        else:
+            # Otherwise, we make the network deterministic by starting with
+            # downsampled segmentation map instead of random z
+            self.fc = nn.Conv2d(num_in_ch, 16 * self.nf, 3, padding=1)
+
+        self.head_0 = SPADEResnetBlock(16 * self.nf, 16 * self.nf, norm_g)
+
+        self.g_middle_0 = SPADEResnetBlock(16 * self.nf, 16 * self.nf, norm_g)
+        self.g_middle_1 = SPADEResnetBlock(16 * self.nf, 16 * self.nf, norm_g)
+
+        self.ups = nn.ModuleList([
+            SPADEResnetBlock(16 * self.nf, 8 * self.nf, norm_g),
+            SPADEResnetBlock(8 * self.nf, 4 * self.nf, norm_g),
+            SPADEResnetBlock(4 * self.nf, 2 * self.nf, norm_g),
+            SPADEResnetBlock(2 * self.nf, 1 * self.nf, norm_g)
+        ])
+
+        self.to_rgbs = nn.ModuleList([
+            nn.Conv2d(8 * self.nf, 3, 3, padding=1),
+            nn.Conv2d(4 * self.nf, 3, 3, padding=1),
+            nn.Conv2d(2 * self.nf, 3, 3, padding=1),
+            nn.Conv2d(1 * self.nf, 3, 3, padding=1)
+        ])
+
+        self.up = nn.Upsample(scale_factor=2)
+
+    def encode(self, input_tensor):
+        """
+        Encode input_tensor into feature maps, can be overridden in derived classes
+        Default: nearest downsampling of 2**5 = 32 times
+        """
+        h, w = input_tensor.size()[-2:]
+        sh, sw = h // 2**self.scale_ratio, w // 2**self.scale_ratio
+        x = F.interpolate(input_tensor, size=(sh, sw))
+        return self.fc(x)
+
+    def forward(self, x):
+        # In oroginal SPADE, seg means a segmentation map, but here we use x instead.
+        seg = x
+
+        x = self.encode(x)
+        x = self.head_0(x, seg)
+
+        x = self.up(x)
+        x = self.g_middle_0(x, seg)
+        x = self.g_middle_1(x, seg)
+
+        if self.is_train:
+            phase = self.train_phase + 1
+        else:
+            phase = len(self.to_rgbs)
+
+        for i in range(phase):
+            x = self.up(x)
+            x = self.ups[i](x, seg)
+
+        x = self.to_rgbs[phase - 1](F.leaky_relu(x, 2e-1))
+        x = torch.tanh(x)
+
+        return x
+
+    def mixed_guidance_forward(self, input_x, seg=None, n=0, mode='progressive'):
+        """
+        A helper class for subspace visualization. Input and seg are different images.
+        For the first n levels (including encoder) we use input, for the rest we use seg.
+
+        If mode = 'progressive', the output's like: AAABBB
+        If mode = 'one_plug', the output's like:    AAABAA
+        If mode = 'one_ablate', the output's like:  BBBABB
+        """
+
+        if seg is None:
+            return self.forward(input_x)
+
+        if self.is_train:
+            phase = self.train_phase + 1
+        else:
+            phase = len(self.to_rgbs)
+
+        if mode == 'progressive':
+            n = max(min(n, 4 + phase), 0)
+            guide_list = [input_x] * n + [seg] * (4 + phase - n)
+        elif mode == 'one_plug':
+            n = max(min(n, 4 + phase - 1), 0)
+            guide_list = [seg] * (4 + phase)
+            guide_list[n] = input_x
+        elif mode == 'one_ablate':
+            if n > 3 + phase:
+                return self.forward(input_x)
+            guide_list = [input_x] * (4 + phase)
+            guide_list[n] = seg
+
+        x = self.encode(guide_list[0])
+        x = self.head_0(x, guide_list[1])
+
+        x = self.up(x)
+        x = self.g_middle_0(x, guide_list[2])
+        x = self.g_middle_1(x, guide_list[3])
+
+        for i in range(phase):
+            x = self.up(x)
+            x = self.ups[i](x, guide_list[4 + i])
+
+        x = self.to_rgbs[phase - 1](F.leaky_relu(x, 2e-1))
+        x = torch.tanh(x)
+
+        return x
+
+
+@ARCH_REGISTRY.register()
+class HiFaceGAN(SPADEGenerator):
+    """
+    HiFaceGAN: SPADEGenerator with a learnable feature encoder
+    Current encoder design: LIPEncoder
+    """
+
+    def __init__(self,
+                 num_in_ch=3,
+                 num_feat=64,
+                 use_vae=False,
+                 z_dim=256,
+                 crop_size=512,
+                 norm_g='spectralspadesyncbatch3x3',
+                 is_train=True,
+                 init_train_phase=3):
+        super().__init__(num_in_ch, num_feat, use_vae, z_dim, crop_size, norm_g, is_train, init_train_phase)
+        self.lip_encoder = LIPEncoder(num_in_ch, num_feat, self.sw, self.sh, self.scale_ratio)
+
+    def encode(self, input_tensor):
+        return self.lip_encoder(input_tensor)
+
+
+@ARCH_REGISTRY.register()
+class HiFaceGANDiscriminator(BaseNetwork):
+    """
+    Inspired by pix2pixHD multiscale discriminator.
+
+    Args:
+        num_in_ch (int): Channel number of inputs. Default: 3.
+        num_out_ch (int): Channel number of outputs. Default: 3.
+        conditional_d (bool): Whether use conditional discriminator.
+            Default: True.
+        num_d (int): Number of Multiscale discriminators. Default: 3.
+        n_layers_d (int): Number of downsample layers in each D. Default: 4.
+        num_feat (int): Channel number of base intermediate features.
+            Default: 64.
+        norm_d (str): String to determine normalization layers in D.
+            Choices: [spectral][instance/batch/syncbatch]
+            Default: 'spectralinstance'.
+        keep_features (bool): Keep intermediate features for matching loss, etc.
+            Default: True.
+    """
+
+    def __init__(self,
+                 num_in_ch=3,
+                 num_out_ch=3,
+                 conditional_d=True,
+                 num_d=2,
+                 n_layers_d=4,
+                 num_feat=64,
+                 norm_d='spectralinstance',
+                 keep_features=True):
+        super().__init__()
+        self.num_d = num_d
+
+        input_nc = num_in_ch
+        if conditional_d:
+            input_nc += num_out_ch
+
+        for i in range(num_d):
+            subnet_d = NLayerDiscriminator(input_nc, n_layers_d, num_feat, norm_d, keep_features)
+            self.add_module(f'discriminator_{i}', subnet_d)
+
+    def downsample(self, x):
+        return F.avg_pool2d(x, kernel_size=3, stride=2, padding=[1, 1], count_include_pad=False)
+
+    # Returns list of lists of discriminator outputs.
+    # The final result is of size opt.num_d x opt.n_layers_D
+    def forward(self, x):
+        result = []
+        for _, _net_d in self.named_children():
+            out = _net_d(x)
+            result.append(out)
+            x = self.downsample(x)
+
+        return result
+
+
+class NLayerDiscriminator(BaseNetwork):
+    """Defines the PatchGAN discriminator with the specified arguments."""
+
+    def __init__(self, input_nc, n_layers_d, num_feat, norm_d, keep_features):
+        super().__init__()
+        kw = 4
+        padw = int(np.ceil((kw - 1.0) / 2))
+        nf = num_feat
+        self.keep_features = keep_features
+
+        norm_layer = get_nonspade_norm_layer(norm_d)
+        sequence = [[nn.Conv2d(input_nc, nf, kernel_size=kw, stride=2, padding=padw), nn.LeakyReLU(0.2, False)]]
+
+        for n in range(1, n_layers_d):
+            nf_prev = nf
+            nf = min(nf * 2, 512)
+            stride = 1 if n == n_layers_d - 1 else 2
+            sequence += [[
+                norm_layer(nn.Conv2d(nf_prev, nf, kernel_size=kw, stride=stride, padding=padw)),
+                nn.LeakyReLU(0.2, False)
+            ]]
+
+        sequence += [[nn.Conv2d(nf, 1, kernel_size=kw, stride=1, padding=padw)]]
+
+        # We divide the layers into groups to extract intermediate layer outputs
+        for n in range(len(sequence)):
+            self.add_module('model' + str(n), nn.Sequential(*sequence[n]))
+
+    def forward(self, x):
+        results = [x]
+        for submodel in self.children():
+            intermediate_output = submodel(results[-1])
+            results.append(intermediate_output)
+
+        if self.keep_features:
+            return results[1:]
+        else:
+            return results[-1]

basicsr/archs/hifacegan_util.py

@@ -0,0 +1,255 @@
+import re
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn import init
+# Warning: spectral norm could be buggy
+# under eval mode and multi-GPU inference
+# A workaround is sticking to single-GPU inference and train mode
+from torch.nn.utils import spectral_norm
+
+
+class SPADE(nn.Module):
+
+    def __init__(self, config_text, norm_nc, label_nc):
+        super().__init__()
+
+        assert config_text.startswith('spade')
+        parsed = re.search('spade(\\D+)(\\d)x\\d', config_text)
+        param_free_norm_type = str(parsed.group(1))
+        ks = int(parsed.group(2))
+
+        if param_free_norm_type == 'instance':
+            self.param_free_norm = nn.InstanceNorm2d(norm_nc)
+        elif param_free_norm_type == 'syncbatch':
+            print('SyncBatchNorm is currently not supported under single-GPU mode, switch to "instance" instead')
+            self.param_free_norm = nn.InstanceNorm2d(norm_nc)
+        elif param_free_norm_type == 'batch':
+            self.param_free_norm = nn.BatchNorm2d(norm_nc, affine=False)
+        else:
+            raise ValueError(f'{param_free_norm_type} is not a recognized param-free norm type in SPADE')
+
+        # The dimension of the intermediate embedding space. Yes, hardcoded.
+        nhidden = 128 if norm_nc > 128 else norm_nc
+
+        pw = ks // 2
+        self.mlp_shared = nn.Sequential(nn.Conv2d(label_nc, nhidden, kernel_size=ks, padding=pw), nn.ReLU())
+        self.mlp_gamma = nn.Conv2d(nhidden, norm_nc, kernel_size=ks, padding=pw, bias=False)
+        self.mlp_beta = nn.Conv2d(nhidden, norm_nc, kernel_size=ks, padding=pw, bias=False)
+
+    def forward(self, x, segmap):
+
+        # Part 1. generate parameter-free normalized activations
+        normalized = self.param_free_norm(x)
+
+        # Part 2. produce scaling and bias conditioned on semantic map
+        segmap = F.interpolate(segmap, size=x.size()[2:], mode='nearest')
+        actv = self.mlp_shared(segmap)
+        gamma = self.mlp_gamma(actv)
+        beta = self.mlp_beta(actv)
+
+        # apply scale and bias
+        out = normalized * gamma + beta
+
+        return out
+
+
+class SPADEResnetBlock(nn.Module):
+    """
+    ResNet block that uses SPADE. It differs from the ResNet block of pix2pixHD in that
+    it takes in the segmentation map as input, learns the skip connection if necessary,
+    and applies normalization first and then convolution.
+    This architecture seemed like a standard architecture for unconditional or
+    class-conditional GAN architecture using residual block.
+    The code was inspired from https://github.com/LMescheder/GAN_stability.
+    """
+
+    def __init__(self, fin, fout, norm_g='spectralspadesyncbatch3x3', semantic_nc=3):
+        super().__init__()
+        # Attributes
+        self.learned_shortcut = (fin != fout)
+        fmiddle = min(fin, fout)
+
+        # create conv layers
+        self.conv_0 = nn.Conv2d(fin, fmiddle, kernel_size=3, padding=1)
+        self.conv_1 = nn.Conv2d(fmiddle, fout, kernel_size=3, padding=1)
+        if self.learned_shortcut:
+            self.conv_s = nn.Conv2d(fin, fout, kernel_size=1, bias=False)
+
+        # apply spectral norm if specified
+        if 'spectral' in norm_g:
+            self.conv_0 = spectral_norm(self.conv_0)
+            self.conv_1 = spectral_norm(self.conv_1)
+            if self.learned_shortcut:
+                self.conv_s = spectral_norm(self.conv_s)
+
+        # define normalization layers
+        spade_config_str = norm_g.replace('spectral', '')
+        self.norm_0 = SPADE(spade_config_str, fin, semantic_nc)
+        self.norm_1 = SPADE(spade_config_str, fmiddle, semantic_nc)
+        if self.learned_shortcut:
+            self.norm_s = SPADE(spade_config_str, fin, semantic_nc)
+
+    # note the resnet block with SPADE also takes in |seg|,
+    # the semantic segmentation map as input
+    def forward(self, x, seg):
+        x_s = self.shortcut(x, seg)
+        dx = self.conv_0(self.act(self.norm_0(x, seg)))
+        dx = self.conv_1(self.act(self.norm_1(dx, seg)))
+        out = x_s + dx
+        return out
+
+    def shortcut(self, x, seg):
+        if self.learned_shortcut:
+            x_s = self.conv_s(self.norm_s(x, seg))
+        else:
+            x_s = x
+        return x_s
+
+    def act(self, x):
+        return F.leaky_relu(x, 2e-1)
+
+
+class BaseNetwork(nn.Module):
+    """ A basis for hifacegan archs with custom initialization """
+
+    def init_weights(self, init_type='normal', gain=0.02):
+
+        def init_func(m):
+            classname = m.__class__.__name__
+            if classname.find('BatchNorm2d') != -1:
+                if hasattr(m, 'weight') and m.weight is not None:
+                    init.normal_(m.weight.data, 1.0, gain)
+                if hasattr(m, 'bias') and m.bias is not None:
+                    init.constant_(m.bias.data, 0.0)
+            elif hasattr(m, 'weight') and (classname.find('Conv') != -1 or classname.find('Linear') != -1):
+                if init_type == 'normal':
+                    init.normal_(m.weight.data, 0.0, gain)
+                elif init_type == 'xavier':
+                    init.xavier_normal_(m.weight.data, gain=gain)
+                elif init_type == 'xavier_uniform':
+                    init.xavier_uniform_(m.weight.data, gain=1.0)
+                elif init_type == 'kaiming':
+                    init.kaiming_normal_(m.weight.data, a=0, mode='fan_in')
+                elif init_type == 'orthogonal':
+                    init.orthogonal_(m.weight.data, gain=gain)
+                elif init_type == 'none':  # uses pytorch's default init method
+                    m.reset_parameters()
+                else:
+                    raise NotImplementedError(f'initialization method [{init_type}] is not implemented')
+                if hasattr(m, 'bias') and m.bias is not None:
+                    init.constant_(m.bias.data, 0.0)
+
+        self.apply(init_func)
+
+        # propagate to children
+        for m in self.children():
+            if hasattr(m, 'init_weights'):
+                m.init_weights(init_type, gain)
+
+    def forward(self, x):
+        pass
+
+
+def lip2d(x, logit, kernel=3, stride=2, padding=1):
+    weight = logit.exp()
+    return F.avg_pool2d(x * weight, kernel, stride, padding) / F.avg_pool2d(weight, kernel, stride, padding)
+
+
+class SoftGate(nn.Module):
+    COEFF = 12.0
+
+    def forward(self, x):
+        return torch.sigmoid(x).mul(self.COEFF)
+
+
+class SimplifiedLIP(nn.Module):
+
+    def __init__(self, channels):
+        super(SimplifiedLIP, self).__init__()
+        self.logit = nn.Sequential(
+            nn.Conv2d(channels, channels, 3, padding=1, bias=False), nn.InstanceNorm2d(channels, affine=True),
+            SoftGate())
+
+    def init_layer(self):
+        self.logit[0].weight.data.fill_(0.0)
+
+    def forward(self, x):
+        frac = lip2d(x, self.logit(x))
+        return frac
+
+
+class LIPEncoder(BaseNetwork):
+    """Local Importance-based Pooling (Ziteng Gao et.al.,ICCV 2019)"""
+
+    def __init__(self, input_nc, ngf, sw, sh, n_2xdown, norm_layer=nn.InstanceNorm2d):
+        super().__init__()
+        self.sw = sw
+        self.sh = sh
+        self.max_ratio = 16
+        # 20200310: Several Convolution (stride 1) + LIP blocks, 4 fold
+        kw = 3
+        pw = (kw - 1) // 2
+
+        model = [
+            nn.Conv2d(input_nc, ngf, kw, stride=1, padding=pw, bias=False),
+            norm_layer(ngf),
+            nn.ReLU(),
+        ]
+        cur_ratio = 1
+        for i in range(n_2xdown):
+            next_ratio = min(cur_ratio * 2, self.max_ratio)
+            model += [
+                SimplifiedLIP(ngf * cur_ratio),
+                nn.Conv2d(ngf * cur_ratio, ngf * next_ratio, kw, stride=1, padding=pw),
+                norm_layer(ngf * next_ratio),
+            ]
+            cur_ratio = next_ratio
+            if i < n_2xdown - 1:
+                model += [nn.ReLU(inplace=True)]
+
+        self.model = nn.Sequential(*model)
+
+    def forward(self, x):
+        return self.model(x)
+
+
+def get_nonspade_norm_layer(norm_type='instance'):
+    # helper function to get # output channels of the previous layer
+    def get_out_channel(layer):
+        if hasattr(layer, 'out_channels'):
+            return getattr(layer, 'out_channels')
+        return layer.weight.size(0)
+
+    # this function will be returned
+    def add_norm_layer(layer):
+        nonlocal norm_type
+        if norm_type.startswith('spectral'):
+            layer = spectral_norm(layer)
+            subnorm_type = norm_type[len('spectral'):]
+
+        if subnorm_type == 'none' or len(subnorm_type) == 0:
+            return layer
+
+        # remove bias in the previous layer, which is meaningless
+        # since it has no effect after normalization
+        if getattr(layer, 'bias', None) is not None:
+            delattr(layer, 'bias')
+            layer.register_parameter('bias', None)
+
+        if subnorm_type == 'batch':
+            norm_layer = nn.BatchNorm2d(get_out_channel(layer), affine=True)
+        elif subnorm_type == 'sync_batch':
+            print('SyncBatchNorm is currently not supported under single-GPU mode, switch to "instance" instead')
+            # norm_layer = SynchronizedBatchNorm2d(
+            #    get_out_channel(layer), affine=True)
+            norm_layer = nn.InstanceNorm2d(get_out_channel(layer), affine=False)
+        elif subnorm_type == 'instance':
+            norm_layer = nn.InstanceNorm2d(get_out_channel(layer), affine=False)
+        else:
+            raise ValueError(f'normalization layer {subnorm_type} is not recognized')
+
+        return nn.Sequential(layer, norm_layer)
+
+    print('This is a legacy from nvlabs/SPADE, and will be removed in future versions.')
+    return add_norm_layer

basicsr/archs/inception.py

@@ -0,0 +1,307 @@
+# Modified from https://github.com/mseitzer/pytorch-fid/blob/master/pytorch_fid/inception.py # noqa: E501
+# For FID metric
+
+import os
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.utils.model_zoo import load_url
+from torchvision import models
+
+# Inception weights ported to Pytorch from
+# http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz
+FID_WEIGHTS_URL = 'https://github.com/mseitzer/pytorch-fid/releases/download/fid_weights/pt_inception-2015-12-05-6726825d.pth'  # noqa: E501
+LOCAL_FID_WEIGHTS = 'experiments/pretrained_models/pt_inception-2015-12-05-6726825d.pth'  # noqa: E501
+
+
+class InceptionV3(nn.Module):
+    """Pretrained InceptionV3 network returning feature maps"""
+
+    # Index of default block of inception to return,
+    # corresponds to output of final average pooling
+    DEFAULT_BLOCK_INDEX = 3
+
+    # Maps feature dimensionality to their output blocks indices
+    BLOCK_INDEX_BY_DIM = {
+        64: 0,  # First max pooling features
+        192: 1,  # Second max pooling features
+        768: 2,  # Pre-aux classifier features
+        2048: 3  # Final average pooling features
+    }
+
+    def __init__(self,
+                 output_blocks=(DEFAULT_BLOCK_INDEX),
+                 resize_input=True,
+                 normalize_input=True,
+                 requires_grad=False,
+                 use_fid_inception=True):
+        """Build pretrained InceptionV3.
+
+        Args:
+            output_blocks (list[int]): Indices of blocks to return features of.
+                Possible values are:
+                - 0: corresponds to output of first max pooling
+                - 1: corresponds to output of second max pooling
+                - 2: corresponds to output which is fed to aux classifier
+                - 3: corresponds to output of final average pooling
+            resize_input (bool): If true, bilinearly resizes input to width and
+                height 299 before feeding input to model. As the network
+                without fully connected layers is fully convolutional, it
+                should be able to handle inputs of arbitrary size, so resizing
+                might not be strictly needed. Default: True.
+            normalize_input (bool): If true, scales the input from range (0, 1)
+                to the range the pretrained Inception network expects,
+                namely (-1, 1). Default: True.
+            requires_grad (bool): If true, parameters of the model require
+                gradients. Possibly useful for finetuning the network.
+                Default: False.
+            use_fid_inception (bool): If true, uses the pretrained Inception
+                model used in Tensorflow's FID implementation.
+                If false, uses the pretrained Inception model available in
+                torchvision. The FID Inception model has different weights
+                and a slightly different structure from torchvision's
+                Inception model. If you want to compute FID scores, you are
+                strongly advised to set this parameter to true to get
+                comparable results. Default: True.
+        """
+        super(InceptionV3, self).__init__()
+
+        self.resize_input = resize_input
+        self.normalize_input = normalize_input
+        self.output_blocks = sorted(output_blocks)
+        self.last_needed_block = max(output_blocks)
+
+        assert self.last_needed_block <= 3, ('Last possible output block index is 3')
+
+        self.blocks = nn.ModuleList()
+
+        if use_fid_inception:
+            inception = fid_inception_v3()
+        else:
+            try:
+                inception = models.inception_v3(pretrained=True, init_weights=False)
+            except TypeError:
+                # pytorch < 1.5 does not have init_weights for inception_v3
+                inception = models.inception_v3(pretrained=True)
+
+        # Block 0: input to maxpool1
+        block0 = [
+            inception.Conv2d_1a_3x3, inception.Conv2d_2a_3x3, inception.Conv2d_2b_3x3,
+            nn.MaxPool2d(kernel_size=3, stride=2)
+        ]
+        self.blocks.append(nn.Sequential(*block0))
+
+        # Block 1: maxpool1 to maxpool2
+        if self.last_needed_block >= 1:
+            block1 = [inception.Conv2d_3b_1x1, inception.Conv2d_4a_3x3, nn.MaxPool2d(kernel_size=3, stride=2)]
+            self.blocks.append(nn.Sequential(*block1))
+
+        # Block 2: maxpool2 to aux classifier
+        if self.last_needed_block >= 2:
+            block2 = [
+                inception.Mixed_5b,
+                inception.Mixed_5c,
+                inception.Mixed_5d,
+                inception.Mixed_6a,
+                inception.Mixed_6b,
+                inception.Mixed_6c,
+                inception.Mixed_6d,
+                inception.Mixed_6e,
+            ]
+            self.blocks.append(nn.Sequential(*block2))
+
+        # Block 3: aux classifier to final avgpool
+        if self.last_needed_block >= 3:
+            block3 = [
+                inception.Mixed_7a, inception.Mixed_7b, inception.Mixed_7c,
+                nn.AdaptiveAvgPool2d(output_size=(1, 1))
+            ]
+            self.blocks.append(nn.Sequential(*block3))
+
+        for param in self.parameters():
+            param.requires_grad = requires_grad
+
+    def forward(self, x):
+        """Get Inception feature maps.
+
+        Args:
+            x (Tensor): Input tensor of shape (b, 3, h, w).
+                Values are expected to be in range (-1, 1). You can also input
+                (0, 1) with setting normalize_input = True.
+
+        Returns:
+            list[Tensor]: Corresponding to the selected output block, sorted
+            ascending by index.
+        """
+        output = []
+
+        if self.resize_input:
+            x = F.interpolate(x, size=(299, 299), mode='bilinear', align_corners=False)
+
+        if self.normalize_input:
+            x = 2 * x - 1  # Scale from range (0, 1) to range (-1, 1)
+
+        for idx, block in enumerate(self.blocks):
+            x = block(x)
+            if idx in self.output_blocks:
+                output.append(x)
+
+            if idx == self.last_needed_block:
+                break
+
+        return output
+
+
+def fid_inception_v3():
+    """Build pretrained Inception model for FID computation.
+
+    The Inception model for FID computation uses a different set of weights
+    and has a slightly different structure than torchvision's Inception.
+
+    This method first constructs torchvision's Inception and then patches the
+    necessary parts that are different in the FID Inception model.
+    """
+    try:
+        inception = models.inception_v3(num_classes=1008, aux_logits=False, pretrained=False, init_weights=False)
+    except TypeError:
+        # pytorch < 1.5 does not have init_weights for inception_v3
+        inception = models.inception_v3(num_classes=1008, aux_logits=False, pretrained=False)
+
+    inception.Mixed_5b = FIDInceptionA(192, pool_features=32)
+    inception.Mixed_5c = FIDInceptionA(256, pool_features=64)
+    inception.Mixed_5d = FIDInceptionA(288, pool_features=64)
+    inception.Mixed_6b = FIDInceptionC(768, channels_7x7=128)
+    inception.Mixed_6c = FIDInceptionC(768, channels_7x7=160)
+    inception.Mixed_6d = FIDInceptionC(768, channels_7x7=160)
+    inception.Mixed_6e = FIDInceptionC(768, channels_7x7=192)
+    inception.Mixed_7b = FIDInceptionE_1(1280)
+    inception.Mixed_7c = FIDInceptionE_2(2048)
+
+    if os.path.exists(LOCAL_FID_WEIGHTS):
+        state_dict = torch.load(LOCAL_FID_WEIGHTS, map_location=lambda storage, loc: storage)
+    else:
+        state_dict = load_url(FID_WEIGHTS_URL, progress=True)
+
+    inception.load_state_dict(state_dict)
+    return inception
+
+
+class FIDInceptionA(models.inception.InceptionA):
+    """InceptionA block patched for FID computation"""
+
+    def __init__(self, in_channels, pool_features):
+        super(FIDInceptionA, self).__init__(in_channels, pool_features)
+
+    def forward(self, x):
+        branch1x1 = self.branch1x1(x)
+
+        branch5x5 = self.branch5x5_1(x)
+        branch5x5 = self.branch5x5_2(branch5x5)
+
+        branch3x3dbl = self.branch3x3dbl_1(x)
+        branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl)
+        branch3x3dbl = self.branch3x3dbl_3(branch3x3dbl)
+
+        # Patch: Tensorflow's average pool does not use the padded zero's in
+        # its average calculation
+        branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1, count_include_pad=False)
+        branch_pool = self.branch_pool(branch_pool)
+
+        outputs = [branch1x1, branch5x5, branch3x3dbl, branch_pool]
+        return torch.cat(outputs, 1)
+
+
+class FIDInceptionC(models.inception.InceptionC):
+    """InceptionC block patched for FID computation"""
+
+    def __init__(self, in_channels, channels_7x7):
+        super(FIDInceptionC, self).__init__(in_channels, channels_7x7)
+
+    def forward(self, x):
+        branch1x1 = self.branch1x1(x)
+
+        branch7x7 = self.branch7x7_1(x)
+        branch7x7 = self.branch7x7_2(branch7x7)
+        branch7x7 = self.branch7x7_3(branch7x7)
+
+        branch7x7dbl = self.branch7x7dbl_1(x)
+        branch7x7dbl = self.branch7x7dbl_2(branch7x7dbl)
+        branch7x7dbl = self.branch7x7dbl_3(branch7x7dbl)
+        branch7x7dbl = self.branch7x7dbl_4(branch7x7dbl)
+        branch7x7dbl = self.branch7x7dbl_5(branch7x7dbl)
+
+        # Patch: Tensorflow's average pool does not use the padded zero's in
+        # its average calculation
+        branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1, count_include_pad=False)
+        branch_pool = self.branch_pool(branch_pool)
+
+        outputs = [branch1x1, branch7x7, branch7x7dbl, branch_pool]
+        return torch.cat(outputs, 1)
+
+
+class FIDInceptionE_1(models.inception.InceptionE):
+    """First InceptionE block patched for FID computation"""
+
+    def __init__(self, in_channels):
+        super(FIDInceptionE_1, self).__init__(in_channels)
+
+    def forward(self, x):
+        branch1x1 = self.branch1x1(x)
+
+        branch3x3 = self.branch3x3_1(x)
+        branch3x3 = [
+            self.branch3x3_2a(branch3x3),
+            self.branch3x3_2b(branch3x3),
+        ]
+        branch3x3 = torch.cat(branch3x3, 1)
+
+        branch3x3dbl = self.branch3x3dbl_1(x)
+        branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl)
+        branch3x3dbl = [
+            self.branch3x3dbl_3a(branch3x3dbl),
+            self.branch3x3dbl_3b(branch3x3dbl),
+        ]
+        branch3x3dbl = torch.cat(branch3x3dbl, 1)
+
+        # Patch: Tensorflow's average pool does not use the padded zero's in
+        # its average calculation
+        branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1, count_include_pad=False)
+        branch_pool = self.branch_pool(branch_pool)
+
+        outputs = [branch1x1, branch3x3, branch3x3dbl, branch_pool]
+        return torch.cat(outputs, 1)
+
+
+class FIDInceptionE_2(models.inception.InceptionE):
+    """Second InceptionE block patched for FID computation"""
+
+    def __init__(self, in_channels):
+        super(FIDInceptionE_2, self).__init__(in_channels)
+
+    def forward(self, x):
+        branch1x1 = self.branch1x1(x)
+
+        branch3x3 = self.branch3x3_1(x)
+        branch3x3 = [
+            self.branch3x3_2a(branch3x3),
+            self.branch3x3_2b(branch3x3),
+        ]
+        branch3x3 = torch.cat(branch3x3, 1)
+
+        branch3x3dbl = self.branch3x3dbl_1(x)
+        branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl)
+        branch3x3dbl = [
+            self.branch3x3dbl_3a(branch3x3dbl),
+            self.branch3x3dbl_3b(branch3x3dbl),
+        ]
+        branch3x3dbl = torch.cat(branch3x3dbl, 1)
+
+        # Patch: The FID Inception model uses max pooling instead of average
+        # pooling. This is likely an error in this specific Inception
+        # implementation, as other Inception models use average pooling here
+        # (which matches the description in the paper).
+        branch_pool = F.max_pool2d(x, kernel_size=3, stride=1, padding=1)
+        branch_pool = self.branch_pool(branch_pool)
+
+        outputs = [branch1x1, branch3x3, branch3x3dbl, branch_pool]
+        return torch.cat(outputs, 1)

basicsr/archs/rcan_arch.py

@@ -0,0 +1,135 @@
+import torch
+from torch import nn as nn
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .arch_util import Upsample, make_layer
+
+
+class ChannelAttention(nn.Module):
+    """Channel attention used in RCAN.
+
+    Args:
+        num_feat (int): Channel number of intermediate features.
+        squeeze_factor (int): Channel squeeze factor. Default: 16.
+    """
+
+    def __init__(self, num_feat, squeeze_factor=16):
+        super(ChannelAttention, self).__init__()
+        self.attention = nn.Sequential(
+            nn.AdaptiveAvgPool2d(1), nn.Conv2d(num_feat, num_feat // squeeze_factor, 1, padding=0),
+            nn.ReLU(inplace=True), nn.Conv2d(num_feat // squeeze_factor, num_feat, 1, padding=0), nn.Sigmoid())
+
+    def forward(self, x):
+        y = self.attention(x)
+        return x * y
+
+
+class RCAB(nn.Module):
+    """Residual Channel Attention Block (RCAB) used in RCAN.
+
+    Args:
+        num_feat (int): Channel number of intermediate features.
+        squeeze_factor (int): Channel squeeze factor. Default: 16.
+        res_scale (float): Scale the residual. Default: 1.
+    """
+
+    def __init__(self, num_feat, squeeze_factor=16, res_scale=1):
+        super(RCAB, self).__init__()
+        self.res_scale = res_scale
+
+        self.rcab = nn.Sequential(
+            nn.Conv2d(num_feat, num_feat, 3, 1, 1), nn.ReLU(True), nn.Conv2d(num_feat, num_feat, 3, 1, 1),
+            ChannelAttention(num_feat, squeeze_factor))
+
+    def forward(self, x):
+        res = self.rcab(x) * self.res_scale
+        return res + x
+
+
+class ResidualGroup(nn.Module):
+    """Residual Group of RCAB.
+
+    Args:
+        num_feat (int): Channel number of intermediate features.
+        num_block (int): Block number in the body network.
+        squeeze_factor (int): Channel squeeze factor. Default: 16.
+        res_scale (float): Scale the residual. Default: 1.
+    """
+
+    def __init__(self, num_feat, num_block, squeeze_factor=16, res_scale=1):
+        super(ResidualGroup, self).__init__()
+
+        self.residual_group = make_layer(
+            RCAB, num_block, num_feat=num_feat, squeeze_factor=squeeze_factor, res_scale=res_scale)
+        self.conv = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+
+    def forward(self, x):
+        res = self.conv(self.residual_group(x))
+        return res + x
+
+
+@ARCH_REGISTRY.register()
+class RCAN(nn.Module):
+    """Residual Channel Attention Networks.
+
+    ``Paper: Image Super-Resolution Using Very Deep Residual Channel Attention Networks``
+
+    Reference: https://github.com/yulunzhang/RCAN
+
+    Args:
+        num_in_ch (int): Channel number of inputs.
+        num_out_ch (int): Channel number of outputs.
+        num_feat (int): Channel number of intermediate features.
+            Default: 64.
+        num_group (int): Number of ResidualGroup. Default: 10.
+        num_block (int): Number of RCAB in ResidualGroup. Default: 16.
+        squeeze_factor (int): Channel squeeze factor. Default: 16.
+        upscale (int): Upsampling factor. Support 2^n and 3.
+            Default: 4.
+        res_scale (float): Used to scale the residual in residual block.
+            Default: 1.
+        img_range (float): Image range. Default: 255.
+        rgb_mean (tuple[float]): Image mean in RGB orders.
+            Default: (0.4488, 0.4371, 0.4040), calculated from DIV2K dataset.
+    """
+
+    def __init__(self,
+                 num_in_ch,
+                 num_out_ch,
+                 num_feat=64,
+                 num_group=10,
+                 num_block=16,
+                 squeeze_factor=16,
+                 upscale=4,
+                 res_scale=1,
+                 img_range=255.,
+                 rgb_mean=(0.4488, 0.4371, 0.4040)):
+        super(RCAN, self).__init__()
+
+        self.img_range = img_range
+        self.mean = torch.Tensor(rgb_mean).view(1, 3, 1, 1)
+
+        self.conv_first = nn.Conv2d(num_in_ch, num_feat, 3, 1, 1)
+        self.body = make_layer(
+            ResidualGroup,
+            num_group,
+            num_feat=num_feat,
+            num_block=num_block,
+            squeeze_factor=squeeze_factor,
+            res_scale=res_scale)
+        self.conv_after_body = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.upsample = Upsample(upscale, num_feat)
+        self.conv_last = nn.Conv2d(num_feat, num_out_ch, 3, 1, 1)
+
+    def forward(self, x):
+        self.mean = self.mean.type_as(x)
+
+        x = (x - self.mean) * self.img_range
+        x = self.conv_first(x)
+        res = self.conv_after_body(self.body(x))
+        res += x
+
+        x = self.conv_last(self.upsample(res))
+        x = x / self.img_range + self.mean
+
+        return x

basicsr/archs/ridnet_arch.py

@@ -0,0 +1,180 @@
+import torch
+import torch.nn as nn
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .arch_util import ResidualBlockNoBN, make_layer
+
+
+class MeanShift(nn.Conv2d):
+    """ Data normalization with mean and std.
+
+    Args:
+        rgb_range (int): Maximum value of RGB.
+        rgb_mean (list[float]): Mean for RGB channels.
+        rgb_std (list[float]): Std for RGB channels.
+        sign (int): For subtraction, sign is -1, for addition, sign is 1.
+            Default: -1.
+        requires_grad (bool): Whether to update the self.weight and self.bias.
+            Default: True.
+    """
+
+    def __init__(self, rgb_range, rgb_mean, rgb_std, sign=-1, requires_grad=True):
+        super(MeanShift, self).__init__(3, 3, kernel_size=1)
+        std = torch.Tensor(rgb_std)
+        self.weight.data = torch.eye(3).view(3, 3, 1, 1)
+        self.weight.data.div_(std.view(3, 1, 1, 1))
+        self.bias.data = sign * rgb_range * torch.Tensor(rgb_mean)
+        self.bias.data.div_(std)
+        self.requires_grad = requires_grad
+
+
+class EResidualBlockNoBN(nn.Module):
+    """Enhanced Residual block without BN.
+
+    There are three convolution layers in residual branch.
+    """
+
+    def __init__(self, in_channels, out_channels):
+        super(EResidualBlockNoBN, self).__init__()
+
+        self.body = nn.Sequential(
+            nn.Conv2d(in_channels, out_channels, 3, 1, 1),
+            nn.ReLU(inplace=True),
+            nn.Conv2d(out_channels, out_channels, 3, 1, 1),
+            nn.ReLU(inplace=True),
+            nn.Conv2d(out_channels, out_channels, 1, 1, 0),
+        )
+        self.relu = nn.ReLU(inplace=True)
+
+    def forward(self, x):
+        out = self.body(x)
+        out = self.relu(out + x)
+        return out
+
+
+class MergeRun(nn.Module):
+    """ Merge-and-run unit.
+
+    This unit contains two branches with different dilated convolutions,
+    followed by a convolution to process the concatenated features.
+
+    Paper: Real Image Denoising with Feature Attention
+    Ref git repo: https://github.com/saeed-anwar/RIDNet
+    """
+
+    def __init__(self, in_channels, out_channels, kernel_size=3, stride=1, padding=1):
+        super(MergeRun, self).__init__()
+
+        self.dilation1 = nn.Sequential(
+            nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding), nn.ReLU(inplace=True),
+            nn.Conv2d(out_channels, out_channels, kernel_size, stride, 2, 2), nn.ReLU(inplace=True))
+        self.dilation2 = nn.Sequential(
+            nn.Conv2d(in_channels, out_channels, kernel_size, stride, 3, 3), nn.ReLU(inplace=True),
+            nn.Conv2d(out_channels, out_channels, kernel_size, stride, 4, 4), nn.ReLU(inplace=True))
+
+        self.aggregation = nn.Sequential(
+            nn.Conv2d(out_channels * 2, out_channels, kernel_size, stride, padding), nn.ReLU(inplace=True))
+
+    def forward(self, x):
+        dilation1 = self.dilation1(x)
+        dilation2 = self.dilation2(x)
+        out = torch.cat([dilation1, dilation2], dim=1)
+        out = self.aggregation(out)
+        out = out + x
+        return out
+
+
+class ChannelAttention(nn.Module):
+    """Channel attention.
+
+    Args:
+        num_feat (int): Channel number of intermediate features.
+        squeeze_factor (int): Channel squeeze factor. Default:
+    """
+
+    def __init__(self, mid_channels, squeeze_factor=16):
+        super(ChannelAttention, self).__init__()
+        self.attention = nn.Sequential(
+            nn.AdaptiveAvgPool2d(1), nn.Conv2d(mid_channels, mid_channels // squeeze_factor, 1, padding=0),
+            nn.ReLU(inplace=True), nn.Conv2d(mid_channels // squeeze_factor, mid_channels, 1, padding=0), nn.Sigmoid())
+
+    def forward(self, x):
+        y = self.attention(x)
+        return x * y
+
+
+class EAM(nn.Module):
+    """Enhancement attention modules (EAM) in RIDNet.
+
+    This module contains a merge-and-run unit, a residual block,
+    an enhanced residual block and a feature attention unit.
+
+    Attributes:
+        merge: The merge-and-run unit.
+        block1: The residual block.
+        block2: The enhanced residual block.
+        ca: The feature/channel attention unit.
+    """
+
+    def __init__(self, in_channels, mid_channels, out_channels):
+        super(EAM, self).__init__()
+
+        self.merge = MergeRun(in_channels, mid_channels)
+        self.block1 = ResidualBlockNoBN(mid_channels)
+        self.block2 = EResidualBlockNoBN(mid_channels, out_channels)
+        self.ca = ChannelAttention(out_channels)
+        # The residual block in the paper contains a relu after addition.
+        self.relu = nn.ReLU(inplace=True)
+
+    def forward(self, x):
+        out = self.merge(x)
+        out = self.relu(self.block1(out))
+        out = self.block2(out)
+        out = self.ca(out)
+        return out
+
+
+@ARCH_REGISTRY.register()
+class RIDNet(nn.Module):
+    """RIDNet: Real Image Denoising with Feature Attention.
+
+    Ref git repo: https://github.com/saeed-anwar/RIDNet
+
+    Args:
+        in_channels (int): Channel number of inputs.
+        mid_channels (int): Channel number of EAM modules.
+            Default: 64.
+        out_channels (int): Channel number of outputs.
+        num_block (int): Number of EAM. Default: 4.
+        img_range (float): Image range. Default: 255.
+        rgb_mean (tuple[float]): Image mean in RGB orders.
+            Default: (0.4488, 0.4371, 0.4040), calculated from DIV2K dataset.
+    """
+
+    def __init__(self,
+                 in_channels,
+                 mid_channels,
+                 out_channels,
+                 num_block=4,
+                 img_range=255.,
+                 rgb_mean=(0.4488, 0.4371, 0.4040),
+                 rgb_std=(1.0, 1.0, 1.0)):
+        super(RIDNet, self).__init__()
+
+        self.sub_mean = MeanShift(img_range, rgb_mean, rgb_std)
+        self.add_mean = MeanShift(img_range, rgb_mean, rgb_std, 1)
+
+        self.head = nn.Conv2d(in_channels, mid_channels, 3, 1, 1)
+        self.body = make_layer(
+            EAM, num_block, in_channels=mid_channels, mid_channels=mid_channels, out_channels=mid_channels)
+        self.tail = nn.Conv2d(mid_channels, out_channels, 3, 1, 1)
+
+        self.relu = nn.ReLU(inplace=True)
+
+    def forward(self, x):
+        res = self.sub_mean(x)
+        res = self.tail(self.body(self.relu(self.head(res))))
+        res = self.add_mean(res)
+
+        out = x + res
+        return out

basicsr/archs/rrdbnet_arch.py

@@ -0,0 +1,119 @@
+import torch
+from torch import nn as nn
+from torch.nn import functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .arch_util import default_init_weights, make_layer, pixel_unshuffle
+
+
+class ResidualDenseBlock(nn.Module):
+    """Residual Dense Block.
+
+    Used in RRDB block in ESRGAN.
+
+    Args:
+        num_feat (int): Channel number of intermediate features.
+        num_grow_ch (int): Channels for each growth.
+    """
+
+    def __init__(self, num_feat=64, num_grow_ch=32):
+        super(ResidualDenseBlock, self).__init__()
+        self.conv1 = nn.Conv2d(num_feat, num_grow_ch, 3, 1, 1)
+        self.conv2 = nn.Conv2d(num_feat + num_grow_ch, num_grow_ch, 3, 1, 1)
+        self.conv3 = nn.Conv2d(num_feat + 2 * num_grow_ch, num_grow_ch, 3, 1, 1)
+        self.conv4 = nn.Conv2d(num_feat + 3 * num_grow_ch, num_grow_ch, 3, 1, 1)
+        self.conv5 = nn.Conv2d(num_feat + 4 * num_grow_ch, num_feat, 3, 1, 1)
+
+        self.lrelu = nn.LeakyReLU(negative_slope=0.2, inplace=True)
+
+        # initialization
+        default_init_weights([self.conv1, self.conv2, self.conv3, self.conv4, self.conv5], 0.1)
+
+    def forward(self, x):
+        x1 = self.lrelu(self.conv1(x))
+        x2 = self.lrelu(self.conv2(torch.cat((x, x1), 1)))
+        x3 = self.lrelu(self.conv3(torch.cat((x, x1, x2), 1)))
+        x4 = self.lrelu(self.conv4(torch.cat((x, x1, x2, x3), 1)))
+        x5 = self.conv5(torch.cat((x, x1, x2, x3, x4), 1))
+        # Empirically, we use 0.2 to scale the residual for better performance
+        return x5 * 0.2 + x
+
+
+class RRDB(nn.Module):
+    """Residual in Residual Dense Block.
+
+    Used in RRDB-Net in ESRGAN.
+
+    Args:
+        num_feat (int): Channel number of intermediate features.
+        num_grow_ch (int): Channels for each growth.
+    """
+
+    def __init__(self, num_feat, num_grow_ch=32):
+        super(RRDB, self).__init__()
+        self.rdb1 = ResidualDenseBlock(num_feat, num_grow_ch)
+        self.rdb2 = ResidualDenseBlock(num_feat, num_grow_ch)
+        self.rdb3 = ResidualDenseBlock(num_feat, num_grow_ch)
+
+    def forward(self, x):
+        out = self.rdb1(x)
+        out = self.rdb2(out)
+        out = self.rdb3(out)
+        # Empirically, we use 0.2 to scale the residual for better performance
+        return out * 0.2 + x
+
+
+@ARCH_REGISTRY.register()
+class RRDBNet(nn.Module):
+    """Networks consisting of Residual in Residual Dense Block, which is used
+    in ESRGAN.
+
+    ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks.
+
+    We extend ESRGAN for scale x2 and scale x1.
+    Note: This is one option for scale 1, scale 2 in RRDBNet.
+    We first employ the pixel-unshuffle (an inverse operation of pixelshuffle to reduce the spatial size
+    and enlarge the channel size before feeding inputs into the main ESRGAN architecture.
+
+    Args:
+        num_in_ch (int): Channel number of inputs.
+        num_out_ch (int): Channel number of outputs.
+        num_feat (int): Channel number of intermediate features.
+            Default: 64
+        num_block (int): Block number in the trunk network. Defaults: 23
+        num_grow_ch (int): Channels for each growth. Default: 32.
+    """
+
+    def __init__(self, num_in_ch, num_out_ch, scale=4, num_feat=64, num_block=23, num_grow_ch=32):
+        super(RRDBNet, self).__init__()
+        self.scale = scale
+        if scale == 2:
+            num_in_ch = num_in_ch * 4
+        elif scale == 1:
+            num_in_ch = num_in_ch * 16
+        self.conv_first = nn.Conv2d(num_in_ch, num_feat, 3, 1, 1)
+        self.body = make_layer(RRDB, num_block, num_feat=num_feat, num_grow_ch=num_grow_ch)
+        self.conv_body = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        # upsample
+        self.conv_up1 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.conv_up2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.conv_hr = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.conv_last = nn.Conv2d(num_feat, num_out_ch, 3, 1, 1)
+
+        self.lrelu = nn.LeakyReLU(negative_slope=0.2, inplace=True)
+
+    def forward(self, x):
+        if self.scale == 2:
+            feat = pixel_unshuffle(x, scale=2)
+        elif self.scale == 1:
+            feat = pixel_unshuffle(x, scale=4)
+        else:
+            feat = x
+        feat = self.conv_first(feat)
+        body_feat = self.conv_body(self.body(feat))
+        feat = feat + body_feat
+        # upsample
+        feat = self.lrelu(self.conv_up1(F.interpolate(feat, scale_factor=2, mode='nearest')))
+        feat = self.lrelu(self.conv_up2(F.interpolate(feat, scale_factor=2, mode='nearest')))
+        out = self.conv_last(self.lrelu(self.conv_hr(feat)))
+        return out

basicsr/archs/spynet_arch.py

@@ -0,0 +1,96 @@
+import math
+import torch
+from torch import nn as nn
+from torch.nn import functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .arch_util import flow_warp
+
+
+class BasicModule(nn.Module):
+    """Basic Module for SpyNet.
+    """
+
+    def __init__(self):
+        super(BasicModule, self).__init__()
+
+        self.basic_module = nn.Sequential(
+            nn.Conv2d(in_channels=8, out_channels=32, kernel_size=7, stride=1, padding=3), nn.ReLU(inplace=False),
+            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=7, stride=1, padding=3), nn.ReLU(inplace=False),
+            nn.Conv2d(in_channels=64, out_channels=32, kernel_size=7, stride=1, padding=3), nn.ReLU(inplace=False),
+            nn.Conv2d(in_channels=32, out_channels=16, kernel_size=7, stride=1, padding=3), nn.ReLU(inplace=False),
+            nn.Conv2d(in_channels=16, out_channels=2, kernel_size=7, stride=1, padding=3))
+
+    def forward(self, tensor_input):
+        return self.basic_module(tensor_input)
+
+
+@ARCH_REGISTRY.register()
+class SpyNet(nn.Module):
+    """SpyNet architecture.
+
+    Args:
+        load_path (str): path for pretrained SpyNet. Default: None.
+    """
+
+    def __init__(self, load_path=None):
+        super(SpyNet, self).__init__()
+        self.basic_module = nn.ModuleList([BasicModule() for _ in range(6)])
+        if load_path:
+            self.load_state_dict(torch.load(load_path, map_location=lambda storage, loc: storage)['params'])
+
+        self.register_buffer('mean', torch.Tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
+        self.register_buffer('std', torch.Tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
+
+    def preprocess(self, tensor_input):
+        tensor_output = (tensor_input - self.mean) / self.std
+        return tensor_output
+
+    def process(self, ref, supp):
+        flow = []
+
+        ref = [self.preprocess(ref)]
+        supp = [self.preprocess(supp)]
+
+        for level in range(5):
+            ref.insert(0, F.avg_pool2d(input=ref[0], kernel_size=2, stride=2, count_include_pad=False))
+            supp.insert(0, F.avg_pool2d(input=supp[0], kernel_size=2, stride=2, count_include_pad=False))
+
+        flow = ref[0].new_zeros(
+            [ref[0].size(0), 2,
+             int(math.floor(ref[0].size(2) / 2.0)),
+             int(math.floor(ref[0].size(3) / 2.0))])
+
+        for level in range(len(ref)):
+            upsampled_flow = F.interpolate(input=flow, scale_factor=2, mode='bilinear', align_corners=True) * 2.0
+
+            if upsampled_flow.size(2) != ref[level].size(2):
+                upsampled_flow = F.pad(input=upsampled_flow, pad=[0, 0, 0, 1], mode='replicate')
+            if upsampled_flow.size(3) != ref[level].size(3):
+                upsampled_flow = F.pad(input=upsampled_flow, pad=[0, 1, 0, 0], mode='replicate')
+
+            flow = self.basic_module[level](torch.cat([
+                ref[level],
+                flow_warp(
+                    supp[level], upsampled_flow.permute(0, 2, 3, 1), interp_mode='bilinear', padding_mode='border'),
+                upsampled_flow
+            ], 1)) + upsampled_flow
+
+        return flow
+
+    def forward(self, ref, supp):
+        assert ref.size() == supp.size()
+
+        h, w = ref.size(2), ref.size(3)
+        w_floor = math.floor(math.ceil(w / 32.0) * 32.0)
+        h_floor = math.floor(math.ceil(h / 32.0) * 32.0)
+
+        ref = F.interpolate(input=ref, size=(h_floor, w_floor), mode='bilinear', align_corners=False)
+        supp = F.interpolate(input=supp, size=(h_floor, w_floor), mode='bilinear', align_corners=False)
+
+        flow = F.interpolate(input=self.process(ref, supp), size=(h, w), mode='bilinear', align_corners=False)
+
+        flow[:, 0, :, :] *= float(w) / float(w_floor)
+        flow[:, 1, :, :] *= float(h) / float(h_floor)
+
+        return flow

basicsr/archs/srresnet_arch.py

@@ -0,0 +1,65 @@
+from torch import nn as nn
+from torch.nn import functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .arch_util import ResidualBlockNoBN, default_init_weights, make_layer
+
+
+@ARCH_REGISTRY.register()
+class MSRResNet(nn.Module):
+    """Modified SRResNet.
+
+    A compacted version modified from SRResNet in
+    "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network"
+    It uses residual blocks without BN, similar to EDSR.
+    Currently, it supports x2, x3 and x4 upsampling scale factor.
+
+    Args:
+        num_in_ch (int): Channel number of inputs. Default: 3.
+        num_out_ch (int): Channel number of outputs. Default: 3.
+        num_feat (int): Channel number of intermediate features. Default: 64.
+        num_block (int): Block number in the body network. Default: 16.
+        upscale (int): Upsampling factor. Support x2, x3 and x4. Default: 4.
+    """
+
+    def __init__(self, num_in_ch=3, num_out_ch=3, num_feat=64, num_block=16, upscale=4):
+        super(MSRResNet, self).__init__()
+        self.upscale = upscale
+
+        self.conv_first = nn.Conv2d(num_in_ch, num_feat, 3, 1, 1)
+        self.body = make_layer(ResidualBlockNoBN, num_block, num_feat=num_feat)
+
+        # upsampling
+        if self.upscale in [2, 3]:
+            self.upconv1 = nn.Conv2d(num_feat, num_feat * self.upscale * self.upscale, 3, 1, 1)
+            self.pixel_shuffle = nn.PixelShuffle(self.upscale)
+        elif self.upscale == 4:
+            self.upconv1 = nn.Conv2d(num_feat, num_feat * 4, 3, 1, 1)
+            self.upconv2 = nn.Conv2d(num_feat, num_feat * 4, 3, 1, 1)
+            self.pixel_shuffle = nn.PixelShuffle(2)
+
+        self.conv_hr = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+        self.conv_last = nn.Conv2d(num_feat, num_out_ch, 3, 1, 1)
+
+        # activation function
+        self.lrelu = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+
+        # initialization
+        default_init_weights([self.conv_first, self.upconv1, self.conv_hr, self.conv_last], 0.1)
+        if self.upscale == 4:
+            default_init_weights(self.upconv2, 0.1)
+
+    def forward(self, x):
+        feat = self.lrelu(self.conv_first(x))
+        out = self.body(feat)
+
+        if self.upscale == 4:
+            out = self.lrelu(self.pixel_shuffle(self.upconv1(out)))
+            out = self.lrelu(self.pixel_shuffle(self.upconv2(out)))
+        elif self.upscale in [2, 3]:
+            out = self.lrelu(self.pixel_shuffle(self.upconv1(out)))
+
+        out = self.conv_last(self.lrelu(self.conv_hr(out)))
+        base = F.interpolate(x, scale_factor=self.upscale, mode='bilinear', align_corners=False)
+        out += base
+        return out

basicsr/archs/srvgg_arch.py

@@ -0,0 +1,70 @@
+from torch import nn as nn
+from torch.nn import functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+
+
+@ARCH_REGISTRY.register(suffix='basicsr')
+class SRVGGNetCompact(nn.Module):
+    """A compact VGG-style network structure for super-resolution.
+
+    It is a compact network structure, which performs upsampling in the last layer and no convolution is
+    conducted on the HR feature space.
+
+    Args:
+        num_in_ch (int): Channel number of inputs. Default: 3.
+        num_out_ch (int): Channel number of outputs. Default: 3.
+        num_feat (int): Channel number of intermediate features. Default: 64.
+        num_conv (int): Number of convolution layers in the body network. Default: 16.
+        upscale (int): Upsampling factor. Default: 4.
+        act_type (str): Activation type, options: 'relu', 'prelu', 'leakyrelu'. Default: prelu.
+    """
+
+    def __init__(self, num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=16, upscale=4, act_type='prelu'):
+        super(SRVGGNetCompact, self).__init__()
+        self.num_in_ch = num_in_ch
+        self.num_out_ch = num_out_ch
+        self.num_feat = num_feat
+        self.num_conv = num_conv
+        self.upscale = upscale
+        self.act_type = act_type
+
+        self.body = nn.ModuleList()
+        # the first conv
+        self.body.append(nn.Conv2d(num_in_ch, num_feat, 3, 1, 1))
+        # the first activation
+        if act_type == 'relu':
+            activation = nn.ReLU(inplace=True)
+        elif act_type == 'prelu':
+            activation = nn.PReLU(num_parameters=num_feat)
+        elif act_type == 'leakyrelu':
+            activation = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+        self.body.append(activation)
+
+        # the body structure
+        for _ in range(num_conv):
+            self.body.append(nn.Conv2d(num_feat, num_feat, 3, 1, 1))
+            # activation
+            if act_type == 'relu':
+                activation = nn.ReLU(inplace=True)
+            elif act_type == 'prelu':
+                activation = nn.PReLU(num_parameters=num_feat)
+            elif act_type == 'leakyrelu':
+                activation = nn.LeakyReLU(negative_slope=0.1, inplace=True)
+            self.body.append(activation)
+
+        # the last conv
+        self.body.append(nn.Conv2d(num_feat, num_out_ch * upscale * upscale, 3, 1, 1))
+        # upsample
+        self.upsampler = nn.PixelShuffle(upscale)
+
+    def forward(self, x):
+        out = x
+        for i in range(0, len(self.body)):
+            out = self.body[i](out)
+
+        out = self.upsampler(out)
+        # add the nearest upsampled image, so that the network learns the residual
+        base = F.interpolate(x, scale_factor=self.upscale, mode='nearest')
+        out += base
+        return out

basicsr/archs/stylegan2_arch.py

@@ -0,0 +1,799 @@
+import math
+import random
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+from basicsr.ops.fused_act import FusedLeakyReLU, fused_leaky_relu
+from basicsr.ops.upfirdn2d import upfirdn2d
+from basicsr.utils.registry import ARCH_REGISTRY
+
+
+class NormStyleCode(nn.Module):
+
+    def forward(self, x):
+        """Normalize the style codes.
+
+        Args:
+            x (Tensor): Style codes with shape (b, c).
+
+        Returns:
+            Tensor: Normalized tensor.
+        """
+        return x * torch.rsqrt(torch.mean(x**2, dim=1, keepdim=True) + 1e-8)
+
+
+def make_resample_kernel(k):
+    """Make resampling kernel for UpFirDn.
+
+    Args:
+        k (list[int]): A list indicating the 1D resample kernel magnitude.
+
+    Returns:
+        Tensor: 2D resampled kernel.
+    """
+    k = torch.tensor(k, dtype=torch.float32)
+    if k.ndim == 1:
+        k = k[None, :] * k[:, None]  # to 2D kernel, outer product
+    # normalize
+    k /= k.sum()
+    return k
+
+
+class UpFirDnUpsample(nn.Module):
+    """Upsample, FIR filter, and downsample (upsampole version).
+
+    References:
+    1. https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.upfirdn.html  # noqa: E501
+    2. http://www.ece.northwestern.edu/local-apps/matlabhelp/toolbox/signal/upfirdn.html  # noqa: E501
+
+    Args:
+        resample_kernel (list[int]): A list indicating the 1D resample kernel
+            magnitude.
+        factor (int): Upsampling scale factor. Default: 2.
+    """
+
+    def __init__(self, resample_kernel, factor=2):
+        super(UpFirDnUpsample, self).__init__()
+        self.kernel = make_resample_kernel(resample_kernel) * (factor**2)
+        self.factor = factor
+
+        pad = self.kernel.shape[0] - factor
+        self.pad = ((pad + 1) // 2 + factor - 1, pad // 2)
+
+    def forward(self, x):
+        out = upfirdn2d(x, self.kernel.type_as(x), up=self.factor, down=1, pad=self.pad)
+        return out
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}(factor={self.factor})')
+
+
+class UpFirDnDownsample(nn.Module):
+    """Upsample, FIR filter, and downsample (downsampole version).
+
+    Args:
+        resample_kernel (list[int]): A list indicating the 1D resample kernel
+            magnitude.
+        factor (int): Downsampling scale factor. Default: 2.
+    """
+
+    def __init__(self, resample_kernel, factor=2):
+        super(UpFirDnDownsample, self).__init__()
+        self.kernel = make_resample_kernel(resample_kernel)
+        self.factor = factor
+
+        pad = self.kernel.shape[0] - factor
+        self.pad = ((pad + 1) // 2, pad // 2)
+
+    def forward(self, x):
+        out = upfirdn2d(x, self.kernel.type_as(x), up=1, down=self.factor, pad=self.pad)
+        return out
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}(factor={self.factor})')
+
+
+class UpFirDnSmooth(nn.Module):
+    """Upsample, FIR filter, and downsample (smooth version).
+
+    Args:
+        resample_kernel (list[int]): A list indicating the 1D resample kernel
+            magnitude.
+        upsample_factor (int): Upsampling scale factor. Default: 1.
+        downsample_factor (int): Downsampling scale factor. Default: 1.
+        kernel_size (int): Kernel size: Default: 1.
+    """
+
+    def __init__(self, resample_kernel, upsample_factor=1, downsample_factor=1, kernel_size=1):
+        super(UpFirDnSmooth, self).__init__()
+        self.upsample_factor = upsample_factor
+        self.downsample_factor = downsample_factor
+        self.kernel = make_resample_kernel(resample_kernel)
+        if upsample_factor > 1:
+            self.kernel = self.kernel * (upsample_factor**2)
+
+        if upsample_factor > 1:
+            pad = (self.kernel.shape[0] - upsample_factor) - (kernel_size - 1)
+            self.pad = ((pad + 1) // 2 + upsample_factor - 1, pad // 2 + 1)
+        elif downsample_factor > 1:
+            pad = (self.kernel.shape[0] - downsample_factor) + (kernel_size - 1)
+            self.pad = ((pad + 1) // 2, pad // 2)
+        else:
+            raise NotImplementedError
+
+    def forward(self, x):
+        out = upfirdn2d(x, self.kernel.type_as(x), up=1, down=1, pad=self.pad)
+        return out
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}(upsample_factor={self.upsample_factor}'
+                f', downsample_factor={self.downsample_factor})')
+
+
+class EqualLinear(nn.Module):
+    """Equalized Linear as StyleGAN2.
+
+    Args:
+        in_channels (int): Size of each sample.
+        out_channels (int): Size of each output sample.
+        bias (bool): If set to ``False``, the layer will not learn an additive
+            bias. Default: ``True``.
+        bias_init_val (float): Bias initialized value. Default: 0.
+        lr_mul (float): Learning rate multiplier. Default: 1.
+        activation (None | str): The activation after ``linear`` operation.
+            Supported: 'fused_lrelu', None. Default: None.
+    """
+
+    def __init__(self, in_channels, out_channels, bias=True, bias_init_val=0, lr_mul=1, activation=None):
+        super(EqualLinear, self).__init__()
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+        self.lr_mul = lr_mul
+        self.activation = activation
+        if self.activation not in ['fused_lrelu', None]:
+            raise ValueError(f'Wrong activation value in EqualLinear: {activation}'
+                             "Supported ones are: ['fused_lrelu', None].")
+        self.scale = (1 / math.sqrt(in_channels)) * lr_mul
+
+        self.weight = nn.Parameter(torch.randn(out_channels, in_channels).div_(lr_mul))
+        if bias:
+            self.bias = nn.Parameter(torch.zeros(out_channels).fill_(bias_init_val))
+        else:
+            self.register_parameter('bias', None)
+
+    def forward(self, x):
+        if self.bias is None:
+            bias = None
+        else:
+            bias = self.bias * self.lr_mul
+        if self.activation == 'fused_lrelu':
+            out = F.linear(x, self.weight * self.scale)
+            out = fused_leaky_relu(out, bias)
+        else:
+            out = F.linear(x, self.weight * self.scale, bias=bias)
+        return out
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}(in_channels={self.in_channels}, '
+                f'out_channels={self.out_channels}, bias={self.bias is not None})')
+
+
+class ModulatedConv2d(nn.Module):
+    """Modulated Conv2d used in StyleGAN2.
+
+    There is no bias in ModulatedConv2d.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+        kernel_size (int): Size of the convolving kernel.
+        num_style_feat (int): Channel number of style features.
+        demodulate (bool): Whether to demodulate in the conv layer.
+            Default: True.
+        sample_mode (str | None): Indicating 'upsample', 'downsample' or None.
+            Default: None.
+        resample_kernel (list[int]): A list indicating the 1D resample kernel
+            magnitude. Default: (1, 3, 3, 1).
+        eps (float): A value added to the denominator for numerical stability.
+            Default: 1e-8.
+    """
+
+    def __init__(self,
+                 in_channels,
+                 out_channels,
+                 kernel_size,
+                 num_style_feat,
+                 demodulate=True,
+                 sample_mode=None,
+                 resample_kernel=(1, 3, 3, 1),
+                 eps=1e-8):
+        super(ModulatedConv2d, self).__init__()
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+        self.kernel_size = kernel_size
+        self.demodulate = demodulate
+        self.sample_mode = sample_mode
+        self.eps = eps
+
+        if self.sample_mode == 'upsample':
+            self.smooth = UpFirDnSmooth(
+                resample_kernel, upsample_factor=2, downsample_factor=1, kernel_size=kernel_size)
+        elif self.sample_mode == 'downsample':
+            self.smooth = UpFirDnSmooth(
+                resample_kernel, upsample_factor=1, downsample_factor=2, kernel_size=kernel_size)
+        elif self.sample_mode is None:
+            pass
+        else:
+            raise ValueError(f'Wrong sample mode {self.sample_mode}, '
+                             "supported ones are ['upsample', 'downsample', None].")
+
+        self.scale = 1 / math.sqrt(in_channels * kernel_size**2)
+        # modulation inside each modulated conv
+        self.modulation = EqualLinear(
+            num_style_feat, in_channels, bias=True, bias_init_val=1, lr_mul=1, activation=None)
+
+        self.weight = nn.Parameter(torch.randn(1, out_channels, in_channels, kernel_size, kernel_size))
+        self.padding = kernel_size // 2
+
+    def forward(self, x, style):
+        """Forward function.
+
+        Args:
+            x (Tensor): Tensor with shape (b, c, h, w).
+            style (Tensor): Tensor with shape (b, num_style_feat).
+
+        Returns:
+            Tensor: Modulated tensor after convolution.
+        """
+        b, c, h, w = x.shape  # c = c_in
+        # weight modulation
+        style = self.modulation(style).view(b, 1, c, 1, 1)
+        # self.weight: (1, c_out, c_in, k, k); style: (b, 1, c, 1, 1)
+        weight = self.scale * self.weight * style  # (b, c_out, c_in, k, k)
+
+        if self.demodulate:
+            demod = torch.rsqrt(weight.pow(2).sum([2, 3, 4]) + self.eps)
+            weight = weight * demod.view(b, self.out_channels, 1, 1, 1)
+
+        weight = weight.view(b * self.out_channels, c, self.kernel_size, self.kernel_size)
+
+        if self.sample_mode == 'upsample':
+            x = x.view(1, b * c, h, w)
+            weight = weight.view(b, self.out_channels, c, self.kernel_size, self.kernel_size)
+            weight = weight.transpose(1, 2).reshape(b * c, self.out_channels, self.kernel_size, self.kernel_size)
+            out = F.conv_transpose2d(x, weight, padding=0, stride=2, groups=b)
+            out = out.view(b, self.out_channels, *out.shape[2:4])
+            out = self.smooth(out)
+        elif self.sample_mode == 'downsample':
+            x = self.smooth(x)
+            x = x.view(1, b * c, *x.shape[2:4])
+            out = F.conv2d(x, weight, padding=0, stride=2, groups=b)
+            out = out.view(b, self.out_channels, *out.shape[2:4])
+        else:
+            x = x.view(1, b * c, h, w)
+            # weight: (b*c_out, c_in, k, k), groups=b
+            out = F.conv2d(x, weight, padding=self.padding, groups=b)
+            out = out.view(b, self.out_channels, *out.shape[2:4])
+
+        return out
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}(in_channels={self.in_channels}, '
+                f'out_channels={self.out_channels}, '
+                f'kernel_size={self.kernel_size}, '
+                f'demodulate={self.demodulate}, sample_mode={self.sample_mode})')
+
+
+class StyleConv(nn.Module):
+    """Style conv.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+        kernel_size (int): Size of the convolving kernel.
+        num_style_feat (int): Channel number of style features.
+        demodulate (bool): Whether demodulate in the conv layer. Default: True.
+        sample_mode (str | None): Indicating 'upsample', 'downsample' or None.
+            Default: None.
+        resample_kernel (list[int]): A list indicating the 1D resample kernel
+            magnitude. Default: (1, 3, 3, 1).
+    """
+
+    def __init__(self,
+                 in_channels,
+                 out_channels,
+                 kernel_size,
+                 num_style_feat,
+                 demodulate=True,
+                 sample_mode=None,
+                 resample_kernel=(1, 3, 3, 1)):
+        super(StyleConv, self).__init__()
+        self.modulated_conv = ModulatedConv2d(
+            in_channels,
+            out_channels,
+            kernel_size,
+            num_style_feat,
+            demodulate=demodulate,
+            sample_mode=sample_mode,
+            resample_kernel=resample_kernel)
+        self.weight = nn.Parameter(torch.zeros(1))  # for noise injection
+        self.activate = FusedLeakyReLU(out_channels)
+
+    def forward(self, x, style, noise=None):
+        # modulate
+        out = self.modulated_conv(x, style)
+        # noise injection
+        if noise is None:
+            b, _, h, w = out.shape
+            noise = out.new_empty(b, 1, h, w).normal_()
+        out = out + self.weight * noise
+        # activation (with bias)
+        out = self.activate(out)
+        return out
+
+
+class ToRGB(nn.Module):
+    """To RGB from features.
+
+    Args:
+        in_channels (int): Channel number of input.
+        num_style_feat (int): Channel number of style features.
+        upsample (bool): Whether to upsample. Default: True.
+        resample_kernel (list[int]): A list indicating the 1D resample kernel
+            magnitude. Default: (1, 3, 3, 1).
+    """
+
+    def __init__(self, in_channels, num_style_feat, upsample=True, resample_kernel=(1, 3, 3, 1)):
+        super(ToRGB, self).__init__()
+        if upsample:
+            self.upsample = UpFirDnUpsample(resample_kernel, factor=2)
+        else:
+            self.upsample = None
+        self.modulated_conv = ModulatedConv2d(
+            in_channels, 3, kernel_size=1, num_style_feat=num_style_feat, demodulate=False, sample_mode=None)
+        self.bias = nn.Parameter(torch.zeros(1, 3, 1, 1))
+
+    def forward(self, x, style, skip=None):
+        """Forward function.
+
+        Args:
+            x (Tensor): Feature tensor with shape (b, c, h, w).
+            style (Tensor): Tensor with shape (b, num_style_feat).
+            skip (Tensor): Base/skip tensor. Default: None.
+
+        Returns:
+            Tensor: RGB images.
+        """
+        out = self.modulated_conv(x, style)
+        out = out + self.bias
+        if skip is not None:
+            if self.upsample:
+                skip = self.upsample(skip)
+            out = out + skip
+        return out
+
+
+class ConstantInput(nn.Module):
+    """Constant input.
+
+    Args:
+        num_channel (int): Channel number of constant input.
+        size (int): Spatial size of constant input.
+    """
+
+    def __init__(self, num_channel, size):
+        super(ConstantInput, self).__init__()
+        self.weight = nn.Parameter(torch.randn(1, num_channel, size, size))
+
+    def forward(self, batch):
+        out = self.weight.repeat(batch, 1, 1, 1)
+        return out
+
+
+@ARCH_REGISTRY.register()
+class StyleGAN2Generator(nn.Module):
+    """StyleGAN2 Generator.
+
+    Args:
+        out_size (int): The spatial size of outputs.
+        num_style_feat (int): Channel number of style features. Default: 512.
+        num_mlp (int): Layer number of MLP style layers. Default: 8.
+        channel_multiplier (int): Channel multiplier for large networks of
+            StyleGAN2. Default: 2.
+        resample_kernel (list[int]): A list indicating the 1D resample kernel
+            magnitude. A cross production will be applied to extent 1D resample
+            kernel to 2D resample kernel. Default: (1, 3, 3, 1).
+        lr_mlp (float): Learning rate multiplier for mlp layers. Default: 0.01.
+        narrow (float): Narrow ratio for channels. Default: 1.0.
+    """
+
+    def __init__(self,
+                 out_size,
+                 num_style_feat=512,
+                 num_mlp=8,
+                 channel_multiplier=2,
+                 resample_kernel=(1, 3, 3, 1),
+                 lr_mlp=0.01,
+                 narrow=1):
+        super(StyleGAN2Generator, self).__init__()
+        # Style MLP layers
+        self.num_style_feat = num_style_feat
+        style_mlp_layers = [NormStyleCode()]
+        for i in range(num_mlp):
+            style_mlp_layers.append(
+                EqualLinear(
+                    num_style_feat, num_style_feat, bias=True, bias_init_val=0, lr_mul=lr_mlp,
+                    activation='fused_lrelu'))
+        self.style_mlp = nn.Sequential(*style_mlp_layers)
+
+        channels = {
+            '4': int(512 * narrow),
+            '8': int(512 * narrow),
+            '16': int(512 * narrow),
+            '32': int(512 * narrow),
+            '64': int(256 * channel_multiplier * narrow),
+            '128': int(128 * channel_multiplier * narrow),
+            '256': int(64 * channel_multiplier * narrow),
+            '512': int(32 * channel_multiplier * narrow),
+            '1024': int(16 * channel_multiplier * narrow)
+        }
+        self.channels = channels
+
+        self.constant_input = ConstantInput(channels['4'], size=4)
+        self.style_conv1 = StyleConv(
+            channels['4'],
+            channels['4'],
+            kernel_size=3,
+            num_style_feat=num_style_feat,
+            demodulate=True,
+            sample_mode=None,
+            resample_kernel=resample_kernel)
+        self.to_rgb1 = ToRGB(channels['4'], num_style_feat, upsample=False, resample_kernel=resample_kernel)
+
+        self.log_size = int(math.log(out_size, 2))
+        self.num_layers = (self.log_size - 2) * 2 + 1
+        self.num_latent = self.log_size * 2 - 2
+
+        self.style_convs = nn.ModuleList()
+        self.to_rgbs = nn.ModuleList()
+        self.noises = nn.Module()
+
+        in_channels = channels['4']
+        # noise
+        for layer_idx in range(self.num_layers):
+            resolution = 2**((layer_idx + 5) // 2)
+            shape = [1, 1, resolution, resolution]
+            self.noises.register_buffer(f'noise{layer_idx}', torch.randn(*shape))
+        # style convs and to_rgbs
+        for i in range(3, self.log_size + 1):
+            out_channels = channels[f'{2**i}']
+            self.style_convs.append(
+                StyleConv(
+                    in_channels,
+                    out_channels,
+                    kernel_size=3,
+                    num_style_feat=num_style_feat,
+                    demodulate=True,
+                    sample_mode='upsample',
+                    resample_kernel=resample_kernel,
+                ))
+            self.style_convs.append(
+                StyleConv(
+                    out_channels,
+                    out_channels,
+                    kernel_size=3,
+                    num_style_feat=num_style_feat,
+                    demodulate=True,
+                    sample_mode=None,
+                    resample_kernel=resample_kernel))
+            self.to_rgbs.append(ToRGB(out_channels, num_style_feat, upsample=True, resample_kernel=resample_kernel))
+            in_channels = out_channels
+
+    def make_noise(self):
+        """Make noise for noise injection."""
+        device = self.constant_input.weight.device
+        noises = [torch.randn(1, 1, 4, 4, device=device)]
+
+        for i in range(3, self.log_size + 1):
+            for _ in range(2):
+                noises.append(torch.randn(1, 1, 2**i, 2**i, device=device))
+
+        return noises
+
+    def get_latent(self, x):
+        return self.style_mlp(x)
+
+    def mean_latent(self, num_latent):
+        latent_in = torch.randn(num_latent, self.num_style_feat, device=self.constant_input.weight.device)
+        latent = self.style_mlp(latent_in).mean(0, keepdim=True)
+        return latent
+
+    def forward(self,
+                styles,
+                input_is_latent=False,
+                noise=None,
+                randomize_noise=True,
+                truncation=1,
+                truncation_latent=None,
+                inject_index=None,
+                return_latents=False):
+        """Forward function for StyleGAN2Generator.
+
+        Args:
+            styles (list[Tensor]): Sample codes of styles.
+            input_is_latent (bool): Whether input is latent style.
+                Default: False.
+            noise (Tensor | None): Input noise or None. Default: None.
+            randomize_noise (bool): Randomize noise, used when 'noise' is
+                False. Default: True.
+            truncation (float): TODO. Default: 1.
+            truncation_latent (Tensor | None): TODO. Default: None.
+            inject_index (int | None): The injection index for mixing noise.
+                Default: None.
+            return_latents (bool): Whether to return style latents.
+                Default: False.
+        """
+        # style codes -> latents with Style MLP layer
+        if not input_is_latent:
+            styles = [self.style_mlp(s) for s in styles]
+        # noises
+        if noise is None:
+            if randomize_noise:
+                noise = [None] * self.num_layers  # for each style conv layer
+            else:  # use the stored noise
+                noise = [getattr(self.noises, f'noise{i}') for i in range(self.num_layers)]
+        # style truncation
+        if truncation < 1:
+            style_truncation = []
+            for style in styles:
+                style_truncation.append(truncation_latent + truncation * (style - truncation_latent))
+            styles = style_truncation
+        # get style latent with injection
+        if len(styles) == 1:
+            inject_index = self.num_latent
+
+            if styles[0].ndim < 3:
+                # repeat latent code for all the layers
+                latent = styles[0].unsqueeze(1).repeat(1, inject_index, 1)
+            else:  # used for encoder with different latent code for each layer
+                latent = styles[0]
+        elif len(styles) == 2:  # mixing noises
+            if inject_index is None:
+                inject_index = random.randint(1, self.num_latent - 1)
+            latent1 = styles[0].unsqueeze(1).repeat(1, inject_index, 1)
+            latent2 = styles[1].unsqueeze(1).repeat(1, self.num_latent - inject_index, 1)
+            latent = torch.cat([latent1, latent2], 1)
+
+        # main generation
+        out = self.constant_input(latent.shape[0])
+        out = self.style_conv1(out, latent[:, 0], noise=noise[0])
+        skip = self.to_rgb1(out, latent[:, 1])
+
+        i = 1
+        for conv1, conv2, noise1, noise2, to_rgb in zip(self.style_convs[::2], self.style_convs[1::2], noise[1::2],
+                                                        noise[2::2], self.to_rgbs):
+            out = conv1(out, latent[:, i], noise=noise1)
+            out = conv2(out, latent[:, i + 1], noise=noise2)
+            skip = to_rgb(out, latent[:, i + 2], skip)
+            i += 2
+
+        image = skip
+
+        if return_latents:
+            return image, latent
+        else:
+            return image, None
+
+
+class ScaledLeakyReLU(nn.Module):
+    """Scaled LeakyReLU.
+
+    Args:
+        negative_slope (float): Negative slope. Default: 0.2.
+    """
+
+    def __init__(self, negative_slope=0.2):
+        super(ScaledLeakyReLU, self).__init__()
+        self.negative_slope = negative_slope
+
+    def forward(self, x):
+        out = F.leaky_relu(x, negative_slope=self.negative_slope)
+        return out * math.sqrt(2)
+
+
+class EqualConv2d(nn.Module):
+    """Equalized Linear as StyleGAN2.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+        kernel_size (int): Size of the convolving kernel.
+        stride (int): Stride of the convolution. Default: 1
+        padding (int): Zero-padding added to both sides of the input.
+            Default: 0.
+        bias (bool): If ``True``, adds a learnable bias to the output.
+            Default: ``True``.
+        bias_init_val (float): Bias initialized value. Default: 0.
+    """
+
+    def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, bias=True, bias_init_val=0):
+        super(EqualConv2d, self).__init__()
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+        self.kernel_size = kernel_size
+        self.stride = stride
+        self.padding = padding
+        self.scale = 1 / math.sqrt(in_channels * kernel_size**2)
+
+        self.weight = nn.Parameter(torch.randn(out_channels, in_channels, kernel_size, kernel_size))
+        if bias:
+            self.bias = nn.Parameter(torch.zeros(out_channels).fill_(bias_init_val))
+        else:
+            self.register_parameter('bias', None)
+
+    def forward(self, x):
+        out = F.conv2d(
+            x,
+            self.weight * self.scale,
+            bias=self.bias,
+            stride=self.stride,
+            padding=self.padding,
+        )
+
+        return out
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}(in_channels={self.in_channels}, '
+                f'out_channels={self.out_channels}, '
+                f'kernel_size={self.kernel_size},'
+                f' stride={self.stride}, padding={self.padding}, '
+                f'bias={self.bias is not None})')
+
+
+class ConvLayer(nn.Sequential):
+    """Conv Layer used in StyleGAN2 Discriminator.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+        kernel_size (int): Kernel size.
+        downsample (bool): Whether downsample by a factor of 2.
+            Default: False.
+        resample_kernel (list[int]): A list indicating the 1D resample
+            kernel magnitude. A cross production will be applied to
+            extent 1D resample kernel to 2D resample kernel.
+            Default: (1, 3, 3, 1).
+        bias (bool): Whether with bias. Default: True.
+        activate (bool): Whether use activateion. Default: True.
+    """
+
+    def __init__(self,
+                 in_channels,
+                 out_channels,
+                 kernel_size,
+                 downsample=False,
+                 resample_kernel=(1, 3, 3, 1),
+                 bias=True,
+                 activate=True):
+        layers = []
+        # downsample
+        if downsample:
+            layers.append(
+                UpFirDnSmooth(resample_kernel, upsample_factor=1, downsample_factor=2, kernel_size=kernel_size))
+            stride = 2
+            self.padding = 0
+        else:
+            stride = 1
+            self.padding = kernel_size // 2
+        # conv
+        layers.append(
+            EqualConv2d(
+                in_channels, out_channels, kernel_size, stride=stride, padding=self.padding, bias=bias
+                and not activate))
+        # activation
+        if activate:
+            if bias:
+                layers.append(FusedLeakyReLU(out_channels))
+            else:
+                layers.append(ScaledLeakyReLU(0.2))
+
+        super(ConvLayer, self).__init__(*layers)
+
+
+class ResBlock(nn.Module):
+    """Residual block used in StyleGAN2 Discriminator.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+        resample_kernel (list[int]): A list indicating the 1D resample
+            kernel magnitude. A cross production will be applied to
+            extent 1D resample kernel to 2D resample kernel.
+            Default: (1, 3, 3, 1).
+    """
+
+    def __init__(self, in_channels, out_channels, resample_kernel=(1, 3, 3, 1)):
+        super(ResBlock, self).__init__()
+
+        self.conv1 = ConvLayer(in_channels, in_channels, 3, bias=True, activate=True)
+        self.conv2 = ConvLayer(
+            in_channels, out_channels, 3, downsample=True, resample_kernel=resample_kernel, bias=True, activate=True)
+        self.skip = ConvLayer(
+            in_channels, out_channels, 1, downsample=True, resample_kernel=resample_kernel, bias=False, activate=False)
+
+    def forward(self, x):
+        out = self.conv1(x)
+        out = self.conv2(out)
+        skip = self.skip(x)
+        out = (out + skip) / math.sqrt(2)
+        return out
+
+
+@ARCH_REGISTRY.register()
+class StyleGAN2Discriminator(nn.Module):
+    """StyleGAN2 Discriminator.
+
+    Args:
+        out_size (int): The spatial size of outputs.
+        channel_multiplier (int): Channel multiplier for large networks of
+            StyleGAN2. Default: 2.
+        resample_kernel (list[int]): A list indicating the 1D resample kernel
+            magnitude. A cross production will be applied to extent 1D resample
+            kernel to 2D resample kernel. Default: (1, 3, 3, 1).
+        stddev_group (int): For group stddev statistics. Default: 4.
+        narrow (float): Narrow ratio for channels. Default: 1.0.
+    """
+
+    def __init__(self, out_size, channel_multiplier=2, resample_kernel=(1, 3, 3, 1), stddev_group=4, narrow=1):
+        super(StyleGAN2Discriminator, self).__init__()
+
+        channels = {
+            '4': int(512 * narrow),
+            '8': int(512 * narrow),
+            '16': int(512 * narrow),
+            '32': int(512 * narrow),
+            '64': int(256 * channel_multiplier * narrow),
+            '128': int(128 * channel_multiplier * narrow),
+            '256': int(64 * channel_multiplier * narrow),
+            '512': int(32 * channel_multiplier * narrow),
+            '1024': int(16 * channel_multiplier * narrow)
+        }
+
+        log_size = int(math.log(out_size, 2))
+
+        conv_body = [ConvLayer(3, channels[f'{out_size}'], 1, bias=True, activate=True)]
+
+        in_channels = channels[f'{out_size}']
+        for i in range(log_size, 2, -1):
+            out_channels = channels[f'{2**(i - 1)}']
+            conv_body.append(ResBlock(in_channels, out_channels, resample_kernel))
+            in_channels = out_channels
+        self.conv_body = nn.Sequential(*conv_body)
+
+        self.final_conv = ConvLayer(in_channels + 1, channels['4'], 3, bias=True, activate=True)
+        self.final_linear = nn.Sequential(
+            EqualLinear(
+                channels['4'] * 4 * 4, channels['4'], bias=True, bias_init_val=0, lr_mul=1, activation='fused_lrelu'),
+            EqualLinear(channels['4'], 1, bias=True, bias_init_val=0, lr_mul=1, activation=None),
+        )
+        self.stddev_group = stddev_group
+        self.stddev_feat = 1
+
+    def forward(self, x):
+        out = self.conv_body(x)
+
+        b, c, h, w = out.shape
+        # concatenate a group stddev statistics to out
+        group = min(b, self.stddev_group)  # Minibatch must be divisible by (or smaller than) group_size
+        stddev = out.view(group, -1, self.stddev_feat, c // self.stddev_feat, h, w)
+        stddev = torch.sqrt(stddev.var(0, unbiased=False) + 1e-8)
+        stddev = stddev.mean([2, 3, 4], keepdims=True).squeeze(2)
+        stddev = stddev.repeat(group, 1, h, w)
+        out = torch.cat([out, stddev], 1)
+
+        out = self.final_conv(out)
+        out = out.view(b, -1)
+        out = self.final_linear(out)
+
+        return out

basicsr/archs/stylegan2_bilinear_arch.py

@@ -0,0 +1,614 @@
+import math
+import random
+import torch
+from torch import nn
+from torch.nn import functional as F
+
+from basicsr.ops.fused_act import FusedLeakyReLU, fused_leaky_relu
+from basicsr.utils.registry import ARCH_REGISTRY
+
+
+class NormStyleCode(nn.Module):
+
+    def forward(self, x):
+        """Normalize the style codes.
+
+        Args:
+            x (Tensor): Style codes with shape (b, c).
+
+        Returns:
+            Tensor: Normalized tensor.
+        """
+        return x * torch.rsqrt(torch.mean(x**2, dim=1, keepdim=True) + 1e-8)
+
+
+class EqualLinear(nn.Module):
+    """Equalized Linear as StyleGAN2.
+
+    Args:
+        in_channels (int): Size of each sample.
+        out_channels (int): Size of each output sample.
+        bias (bool): If set to ``False``, the layer will not learn an additive
+            bias. Default: ``True``.
+        bias_init_val (float): Bias initialized value. Default: 0.
+        lr_mul (float): Learning rate multiplier. Default: 1.
+        activation (None | str): The activation after ``linear`` operation.
+            Supported: 'fused_lrelu', None. Default: None.
+    """
+
+    def __init__(self, in_channels, out_channels, bias=True, bias_init_val=0, lr_mul=1, activation=None):
+        super(EqualLinear, self).__init__()
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+        self.lr_mul = lr_mul
+        self.activation = activation
+        if self.activation not in ['fused_lrelu', None]:
+            raise ValueError(f'Wrong activation value in EqualLinear: {activation}'
+                             "Supported ones are: ['fused_lrelu', None].")
+        self.scale = (1 / math.sqrt(in_channels)) * lr_mul
+
+        self.weight = nn.Parameter(torch.randn(out_channels, in_channels).div_(lr_mul))
+        if bias:
+            self.bias = nn.Parameter(torch.zeros(out_channels).fill_(bias_init_val))
+        else:
+            self.register_parameter('bias', None)
+
+    def forward(self, x):
+        if self.bias is None:
+            bias = None
+        else:
+            bias = self.bias * self.lr_mul
+        if self.activation == 'fused_lrelu':
+            out = F.linear(x, self.weight * self.scale)
+            out = fused_leaky_relu(out, bias)
+        else:
+            out = F.linear(x, self.weight * self.scale, bias=bias)
+        return out
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}(in_channels={self.in_channels}, '
+                f'out_channels={self.out_channels}, bias={self.bias is not None})')
+
+
+class ModulatedConv2d(nn.Module):
+    """Modulated Conv2d used in StyleGAN2.
+
+    There is no bias in ModulatedConv2d.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+        kernel_size (int): Size of the convolving kernel.
+        num_style_feat (int): Channel number of style features.
+        demodulate (bool): Whether to demodulate in the conv layer.
+            Default: True.
+        sample_mode (str | None): Indicating 'upsample', 'downsample' or None.
+            Default: None.
+        eps (float): A value added to the denominator for numerical stability.
+            Default: 1e-8.
+    """
+
+    def __init__(self,
+                 in_channels,
+                 out_channels,
+                 kernel_size,
+                 num_style_feat,
+                 demodulate=True,
+                 sample_mode=None,
+                 eps=1e-8,
+                 interpolation_mode='bilinear'):
+        super(ModulatedConv2d, self).__init__()
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+        self.kernel_size = kernel_size
+        self.demodulate = demodulate
+        self.sample_mode = sample_mode
+        self.eps = eps
+        self.interpolation_mode = interpolation_mode
+        if self.interpolation_mode == 'nearest':
+            self.align_corners = None
+        else:
+            self.align_corners = False
+
+        self.scale = 1 / math.sqrt(in_channels * kernel_size**2)
+        # modulation inside each modulated conv
+        self.modulation = EqualLinear(
+            num_style_feat, in_channels, bias=True, bias_init_val=1, lr_mul=1, activation=None)
+
+        self.weight = nn.Parameter(torch.randn(1, out_channels, in_channels, kernel_size, kernel_size))
+        self.padding = kernel_size // 2
+
+    def forward(self, x, style):
+        """Forward function.
+
+        Args:
+            x (Tensor): Tensor with shape (b, c, h, w).
+            style (Tensor): Tensor with shape (b, num_style_feat).
+
+        Returns:
+            Tensor: Modulated tensor after convolution.
+        """
+        b, c, h, w = x.shape  # c = c_in
+        # weight modulation
+        style = self.modulation(style).view(b, 1, c, 1, 1)
+        # self.weight: (1, c_out, c_in, k, k); style: (b, 1, c, 1, 1)
+        weight = self.scale * self.weight * style  # (b, c_out, c_in, k, k)
+
+        if self.demodulate:
+            demod = torch.rsqrt(weight.pow(2).sum([2, 3, 4]) + self.eps)
+            weight = weight * demod.view(b, self.out_channels, 1, 1, 1)
+
+        weight = weight.view(b * self.out_channels, c, self.kernel_size, self.kernel_size)
+
+        if self.sample_mode == 'upsample':
+            x = F.interpolate(x, scale_factor=2, mode=self.interpolation_mode, align_corners=self.align_corners)
+        elif self.sample_mode == 'downsample':
+            x = F.interpolate(x, scale_factor=0.5, mode=self.interpolation_mode, align_corners=self.align_corners)
+
+        b, c, h, w = x.shape
+        x = x.view(1, b * c, h, w)
+        # weight: (b*c_out, c_in, k, k), groups=b
+        out = F.conv2d(x, weight, padding=self.padding, groups=b)
+        out = out.view(b, self.out_channels, *out.shape[2:4])
+
+        return out
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}(in_channels={self.in_channels}, '
+                f'out_channels={self.out_channels}, '
+                f'kernel_size={self.kernel_size}, '
+                f'demodulate={self.demodulate}, sample_mode={self.sample_mode})')
+
+
+class StyleConv(nn.Module):
+    """Style conv.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+        kernel_size (int): Size of the convolving kernel.
+        num_style_feat (int): Channel number of style features.
+        demodulate (bool): Whether demodulate in the conv layer. Default: True.
+        sample_mode (str | None): Indicating 'upsample', 'downsample' or None.
+            Default: None.
+    """
+
+    def __init__(self,
+                 in_channels,
+                 out_channels,
+                 kernel_size,
+                 num_style_feat,
+                 demodulate=True,
+                 sample_mode=None,
+                 interpolation_mode='bilinear'):
+        super(StyleConv, self).__init__()
+        self.modulated_conv = ModulatedConv2d(
+            in_channels,
+            out_channels,
+            kernel_size,
+            num_style_feat,
+            demodulate=demodulate,
+            sample_mode=sample_mode,
+            interpolation_mode=interpolation_mode)
+        self.weight = nn.Parameter(torch.zeros(1))  # for noise injection
+        self.activate = FusedLeakyReLU(out_channels)
+
+    def forward(self, x, style, noise=None):
+        # modulate
+        out = self.modulated_conv(x, style)
+        # noise injection
+        if noise is None:
+            b, _, h, w = out.shape
+            noise = out.new_empty(b, 1, h, w).normal_()
+        out = out + self.weight * noise
+        # activation (with bias)
+        out = self.activate(out)
+        return out
+
+
+class ToRGB(nn.Module):
+    """To RGB from features.
+
+    Args:
+        in_channels (int): Channel number of input.
+        num_style_feat (int): Channel number of style features.
+        upsample (bool): Whether to upsample. Default: True.
+    """
+
+    def __init__(self, in_channels, num_style_feat, upsample=True, interpolation_mode='bilinear'):
+        super(ToRGB, self).__init__()
+        self.upsample = upsample
+        self.interpolation_mode = interpolation_mode
+        if self.interpolation_mode == 'nearest':
+            self.align_corners = None
+        else:
+            self.align_corners = False
+        self.modulated_conv = ModulatedConv2d(
+            in_channels,
+            3,
+            kernel_size=1,
+            num_style_feat=num_style_feat,
+            demodulate=False,
+            sample_mode=None,
+            interpolation_mode=interpolation_mode)
+        self.bias = nn.Parameter(torch.zeros(1, 3, 1, 1))
+
+    def forward(self, x, style, skip=None):
+        """Forward function.
+
+        Args:
+            x (Tensor): Feature tensor with shape (b, c, h, w).
+            style (Tensor): Tensor with shape (b, num_style_feat).
+            skip (Tensor): Base/skip tensor. Default: None.
+
+        Returns:
+            Tensor: RGB images.
+        """
+        out = self.modulated_conv(x, style)
+        out = out + self.bias
+        if skip is not None:
+            if self.upsample:
+                skip = F.interpolate(
+                    skip, scale_factor=2, mode=self.interpolation_mode, align_corners=self.align_corners)
+            out = out + skip
+        return out
+
+
+class ConstantInput(nn.Module):
+    """Constant input.
+
+    Args:
+        num_channel (int): Channel number of constant input.
+        size (int): Spatial size of constant input.
+    """
+
+    def __init__(self, num_channel, size):
+        super(ConstantInput, self).__init__()
+        self.weight = nn.Parameter(torch.randn(1, num_channel, size, size))
+
+    def forward(self, batch):
+        out = self.weight.repeat(batch, 1, 1, 1)
+        return out
+
+
+@ARCH_REGISTRY.register(suffix='basicsr')
+class StyleGAN2GeneratorBilinear(nn.Module):
+    """StyleGAN2 Generator.
+
+    Args:
+        out_size (int): The spatial size of outputs.
+        num_style_feat (int): Channel number of style features. Default: 512.
+        num_mlp (int): Layer number of MLP style layers. Default: 8.
+        channel_multiplier (int): Channel multiplier for large networks of
+            StyleGAN2. Default: 2.
+        lr_mlp (float): Learning rate multiplier for mlp layers. Default: 0.01.
+        narrow (float): Narrow ratio for channels. Default: 1.0.
+    """
+
+    def __init__(self,
+                 out_size,
+                 num_style_feat=512,
+                 num_mlp=8,
+                 channel_multiplier=2,
+                 lr_mlp=0.01,
+                 narrow=1,
+                 interpolation_mode='bilinear'):
+        super(StyleGAN2GeneratorBilinear, self).__init__()
+        # Style MLP layers
+        self.num_style_feat = num_style_feat
+        style_mlp_layers = [NormStyleCode()]
+        for i in range(num_mlp):
+            style_mlp_layers.append(
+                EqualLinear(
+                    num_style_feat, num_style_feat, bias=True, bias_init_val=0, lr_mul=lr_mlp,
+                    activation='fused_lrelu'))
+        self.style_mlp = nn.Sequential(*style_mlp_layers)
+
+        channels = {
+            '4': int(512 * narrow),
+            '8': int(512 * narrow),
+            '16': int(512 * narrow),
+            '32': int(512 * narrow),
+            '64': int(256 * channel_multiplier * narrow),
+            '128': int(128 * channel_multiplier * narrow),
+            '256': int(64 * channel_multiplier * narrow),
+            '512': int(32 * channel_multiplier * narrow),
+            '1024': int(16 * channel_multiplier * narrow)
+        }
+        self.channels = channels
+
+        self.constant_input = ConstantInput(channels['4'], size=4)
+        self.style_conv1 = StyleConv(
+            channels['4'],
+            channels['4'],
+            kernel_size=3,
+            num_style_feat=num_style_feat,
+            demodulate=True,
+            sample_mode=None,
+            interpolation_mode=interpolation_mode)
+        self.to_rgb1 = ToRGB(channels['4'], num_style_feat, upsample=False, interpolation_mode=interpolation_mode)
+
+        self.log_size = int(math.log(out_size, 2))
+        self.num_layers = (self.log_size - 2) * 2 + 1
+        self.num_latent = self.log_size * 2 - 2
+
+        self.style_convs = nn.ModuleList()
+        self.to_rgbs = nn.ModuleList()
+        self.noises = nn.Module()
+
+        in_channels = channels['4']
+        # noise
+        for layer_idx in range(self.num_layers):
+            resolution = 2**((layer_idx + 5) // 2)
+            shape = [1, 1, resolution, resolution]
+            self.noises.register_buffer(f'noise{layer_idx}', torch.randn(*shape))
+        # style convs and to_rgbs
+        for i in range(3, self.log_size + 1):
+            out_channels = channels[f'{2**i}']
+            self.style_convs.append(
+                StyleConv(
+                    in_channels,
+                    out_channels,
+                    kernel_size=3,
+                    num_style_feat=num_style_feat,
+                    demodulate=True,
+                    sample_mode='upsample',
+                    interpolation_mode=interpolation_mode))
+            self.style_convs.append(
+                StyleConv(
+                    out_channels,
+                    out_channels,
+                    kernel_size=3,
+                    num_style_feat=num_style_feat,
+                    demodulate=True,
+                    sample_mode=None,
+                    interpolation_mode=interpolation_mode))
+            self.to_rgbs.append(
+                ToRGB(out_channels, num_style_feat, upsample=True, interpolation_mode=interpolation_mode))
+            in_channels = out_channels
+
+    def make_noise(self):
+        """Make noise for noise injection."""
+        device = self.constant_input.weight.device
+        noises = [torch.randn(1, 1, 4, 4, device=device)]
+
+        for i in range(3, self.log_size + 1):
+            for _ in range(2):
+                noises.append(torch.randn(1, 1, 2**i, 2**i, device=device))
+
+        return noises
+
+    def get_latent(self, x):
+        return self.style_mlp(x)
+
+    def mean_latent(self, num_latent):
+        latent_in = torch.randn(num_latent, self.num_style_feat, device=self.constant_input.weight.device)
+        latent = self.style_mlp(latent_in).mean(0, keepdim=True)
+        return latent
+
+    def forward(self,
+                styles,
+                input_is_latent=False,
+                noise=None,
+                randomize_noise=True,
+                truncation=1,
+                truncation_latent=None,
+                inject_index=None,
+                return_latents=False):
+        """Forward function for StyleGAN2Generator.
+
+        Args:
+            styles (list[Tensor]): Sample codes of styles.
+            input_is_latent (bool): Whether input is latent style.
+                Default: False.
+            noise (Tensor | None): Input noise or None. Default: None.
+            randomize_noise (bool): Randomize noise, used when 'noise' is
+                False. Default: True.
+            truncation (float): TODO. Default: 1.
+            truncation_latent (Tensor | None): TODO. Default: None.
+            inject_index (int | None): The injection index for mixing noise.
+                Default: None.
+            return_latents (bool): Whether to return style latents.
+                Default: False.
+        """
+        # style codes -> latents with Style MLP layer
+        if not input_is_latent:
+            styles = [self.style_mlp(s) for s in styles]
+        # noises
+        if noise is None:
+            if randomize_noise:
+                noise = [None] * self.num_layers  # for each style conv layer
+            else:  # use the stored noise
+                noise = [getattr(self.noises, f'noise{i}') for i in range(self.num_layers)]
+        # style truncation
+        if truncation < 1:
+            style_truncation = []
+            for style in styles:
+                style_truncation.append(truncation_latent + truncation * (style - truncation_latent))
+            styles = style_truncation
+        # get style latent with injection
+        if len(styles) == 1:
+            inject_index = self.num_latent
+
+            if styles[0].ndim < 3:
+                # repeat latent code for all the layers
+                latent = styles[0].unsqueeze(1).repeat(1, inject_index, 1)
+            else:  # used for encoder with different latent code for each layer
+                latent = styles[0]
+        elif len(styles) == 2:  # mixing noises
+            if inject_index is None:
+                inject_index = random.randint(1, self.num_latent - 1)
+            latent1 = styles[0].unsqueeze(1).repeat(1, inject_index, 1)
+            latent2 = styles[1].unsqueeze(1).repeat(1, self.num_latent - inject_index, 1)
+            latent = torch.cat([latent1, latent2], 1)
+
+        # main generation
+        out = self.constant_input(latent.shape[0])
+        out = self.style_conv1(out, latent[:, 0], noise=noise[0])
+        skip = self.to_rgb1(out, latent[:, 1])
+
+        i = 1
+        for conv1, conv2, noise1, noise2, to_rgb in zip(self.style_convs[::2], self.style_convs[1::2], noise[1::2],
+                                                        noise[2::2], self.to_rgbs):
+            out = conv1(out, latent[:, i], noise=noise1)
+            out = conv2(out, latent[:, i + 1], noise=noise2)
+            skip = to_rgb(out, latent[:, i + 2], skip)
+            i += 2
+
+        image = skip
+
+        if return_latents:
+            return image, latent
+        else:
+            return image, None
+
+
+class ScaledLeakyReLU(nn.Module):
+    """Scaled LeakyReLU.
+
+    Args:
+        negative_slope (float): Negative slope. Default: 0.2.
+    """
+
+    def __init__(self, negative_slope=0.2):
+        super(ScaledLeakyReLU, self).__init__()
+        self.negative_slope = negative_slope
+
+    def forward(self, x):
+        out = F.leaky_relu(x, negative_slope=self.negative_slope)
+        return out * math.sqrt(2)
+
+
+class EqualConv2d(nn.Module):
+    """Equalized Linear as StyleGAN2.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+        kernel_size (int): Size of the convolving kernel.
+        stride (int): Stride of the convolution. Default: 1
+        padding (int): Zero-padding added to both sides of the input.
+            Default: 0.
+        bias (bool): If ``True``, adds a learnable bias to the output.
+            Default: ``True``.
+        bias_init_val (float): Bias initialized value. Default: 0.
+    """
+
+    def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0, bias=True, bias_init_val=0):
+        super(EqualConv2d, self).__init__()
+        self.in_channels = in_channels
+        self.out_channels = out_channels
+        self.kernel_size = kernel_size
+        self.stride = stride
+        self.padding = padding
+        self.scale = 1 / math.sqrt(in_channels * kernel_size**2)
+
+        self.weight = nn.Parameter(torch.randn(out_channels, in_channels, kernel_size, kernel_size))
+        if bias:
+            self.bias = nn.Parameter(torch.zeros(out_channels).fill_(bias_init_val))
+        else:
+            self.register_parameter('bias', None)
+
+    def forward(self, x):
+        out = F.conv2d(
+            x,
+            self.weight * self.scale,
+            bias=self.bias,
+            stride=self.stride,
+            padding=self.padding,
+        )
+
+        return out
+
+    def __repr__(self):
+        return (f'{self.__class__.__name__}(in_channels={self.in_channels}, '
+                f'out_channels={self.out_channels}, '
+                f'kernel_size={self.kernel_size},'
+                f' stride={self.stride}, padding={self.padding}, '
+                f'bias={self.bias is not None})')
+
+
+class ConvLayer(nn.Sequential):
+    """Conv Layer used in StyleGAN2 Discriminator.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+        kernel_size (int): Kernel size.
+        downsample (bool): Whether downsample by a factor of 2.
+            Default: False.
+        bias (bool): Whether with bias. Default: True.
+        activate (bool): Whether use activateion. Default: True.
+    """
+
+    def __init__(self,
+                 in_channels,
+                 out_channels,
+                 kernel_size,
+                 downsample=False,
+                 bias=True,
+                 activate=True,
+                 interpolation_mode='bilinear'):
+        layers = []
+        self.interpolation_mode = interpolation_mode
+        # downsample
+        if downsample:
+            if self.interpolation_mode == 'nearest':
+                self.align_corners = None
+            else:
+                self.align_corners = False
+
+            layers.append(
+                torch.nn.Upsample(scale_factor=0.5, mode=interpolation_mode, align_corners=self.align_corners))
+        stride = 1
+        self.padding = kernel_size // 2
+        # conv
+        layers.append(
+            EqualConv2d(
+                in_channels, out_channels, kernel_size, stride=stride, padding=self.padding, bias=bias
+                and not activate))
+        # activation
+        if activate:
+            if bias:
+                layers.append(FusedLeakyReLU(out_channels))
+            else:
+                layers.append(ScaledLeakyReLU(0.2))
+
+        super(ConvLayer, self).__init__(*layers)
+
+
+class ResBlock(nn.Module):
+    """Residual block used in StyleGAN2 Discriminator.
+
+    Args:
+        in_channels (int): Channel number of the input.
+        out_channels (int): Channel number of the output.
+    """
+
+    def __init__(self, in_channels, out_channels, interpolation_mode='bilinear'):
+        super(ResBlock, self).__init__()
+
+        self.conv1 = ConvLayer(in_channels, in_channels, 3, bias=True, activate=True)
+        self.conv2 = ConvLayer(
+            in_channels,
+            out_channels,
+            3,
+            downsample=True,
+            interpolation_mode=interpolation_mode,
+            bias=True,
+            activate=True)
+        self.skip = ConvLayer(
+            in_channels,
+            out_channels,
+            1,
+            downsample=True,
+            interpolation_mode=interpolation_mode,
+            bias=False,
+            activate=False)
+
+    def forward(self, x):
+        out = self.conv1(x)
+        out = self.conv2(out)
+        skip = self.skip(x)
+        out = (out + skip) / math.sqrt(2)
+        return out

basicsr/archs/swinir_arch.py

@@ -0,0 +1,956 @@
+# Modified from https://github.com/JingyunLiang/SwinIR
+# SwinIR: Image Restoration Using Swin Transformer, https://arxiv.org/abs/2108.10257
+# Originally Written by Ze Liu, Modified by Jingyun Liang.
+
+import math
+import torch
+import torch.nn as nn
+import torch.utils.checkpoint as checkpoint
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .arch_util import to_2tuple, trunc_normal_
+
+
+def drop_path(x, drop_prob: float = 0., training: bool = False):
+    """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
+
+    From: https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/drop.py
+    """
+    if drop_prob == 0. or not training:
+        return x
+    keep_prob = 1 - drop_prob
+    shape = (x.shape[0], ) + (1, ) * (x.ndim - 1)  # work with diff dim tensors, not just 2D ConvNets
+    random_tensor = keep_prob + torch.rand(shape, dtype=x.dtype, device=x.device)
+    random_tensor.floor_()  # binarize
+    output = x.div(keep_prob) * random_tensor
+    return output
+
+
+class DropPath(nn.Module):
+    """Drop paths (Stochastic Depth) per sample  (when applied in main path of residual blocks).
+
+    From: https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/drop.py
+    """
+
+    def __init__(self, drop_prob=None):
+        super(DropPath, self).__init__()
+        self.drop_prob = drop_prob
+
+    def forward(self, x):
+        return drop_path(x, self.drop_prob, self.training)
+
+
+class Mlp(nn.Module):
+
+    def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.):
+        super().__init__()
+        out_features = out_features or in_features
+        hidden_features = hidden_features or in_features
+        self.fc1 = nn.Linear(in_features, hidden_features)
+        self.act = act_layer()
+        self.fc2 = nn.Linear(hidden_features, out_features)
+        self.drop = nn.Dropout(drop)
+
+    def forward(self, x):
+        x = self.fc1(x)
+        x = self.act(x)
+        x = self.drop(x)
+        x = self.fc2(x)
+        x = self.drop(x)
+        return x
+
+
+def window_partition(x, window_size):
+    """
+    Args:
+        x: (b, h, w, c)
+        window_size (int): window size
+
+    Returns:
+        windows: (num_windows*b, window_size, window_size, c)
+    """
+    b, h, w, c = x.shape
+    x = x.view(b, h // window_size, window_size, w // window_size, window_size, c)
+    windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, c)
+    return windows
+
+
+def window_reverse(windows, window_size, h, w):
+    """
+    Args:
+        windows: (num_windows*b, window_size, window_size, c)
+        window_size (int): Window size
+        h (int): Height of image
+        w (int): Width of image
+
+    Returns:
+        x: (b, h, w, c)
+    """
+    b = int(windows.shape[0] / (h * w / window_size / window_size))
+    x = windows.view(b, h // window_size, w // window_size, window_size, window_size, -1)
+    x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(b, h, w, -1)
+    return x
+
+
+class WindowAttention(nn.Module):
+    r""" Window based multi-head self attention (W-MSA) module with relative position bias.
+    It supports both of shifted and non-shifted window.
+
+    Args:
+        dim (int): Number of input channels.
+        window_size (tuple[int]): The height and width of the window.
+        num_heads (int): Number of attention heads.
+        qkv_bias (bool, optional):  If True, add a learnable bias to query, key, value. Default: True
+        qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set
+        attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0
+        proj_drop (float, optional): Dropout ratio of output. Default: 0.0
+    """
+
+    def __init__(self, dim, window_size, num_heads, qkv_bias=True, qk_scale=None, attn_drop=0., proj_drop=0.):
+
+        super().__init__()
+        self.dim = dim
+        self.window_size = window_size  # Wh, Ww
+        self.num_heads = num_heads
+        head_dim = dim // num_heads
+        self.scale = qk_scale or head_dim**-0.5
+
+        # define a parameter table of relative position bias
+        self.relative_position_bias_table = nn.Parameter(
+            torch.zeros((2 * window_size[0] - 1) * (2 * window_size[1] - 1), num_heads))  # 2*Wh-1 * 2*Ww-1, nH
+
+        # get pair-wise relative position index for each token inside the window
+        coords_h = torch.arange(self.window_size[0])
+        coords_w = torch.arange(self.window_size[1])
+        coords = torch.stack(torch.meshgrid([coords_h, coords_w]))  # 2, Wh, Ww
+        coords_flatten = torch.flatten(coords, 1)  # 2, Wh*Ww
+        relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :]  # 2, Wh*Ww, Wh*Ww
+        relative_coords = relative_coords.permute(1, 2, 0).contiguous()  # Wh*Ww, Wh*Ww, 2
+        relative_coords[:, :, 0] += self.window_size[0] - 1  # shift to start from 0
+        relative_coords[:, :, 1] += self.window_size[1] - 1
+        relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
+        relative_position_index = relative_coords.sum(-1)  # Wh*Ww, Wh*Ww
+        self.register_buffer('relative_position_index', relative_position_index)
+
+        self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
+        self.attn_drop = nn.Dropout(attn_drop)
+        self.proj = nn.Linear(dim, dim)
+
+        self.proj_drop = nn.Dropout(proj_drop)
+
+        trunc_normal_(self.relative_position_bias_table, std=.02)
+        self.softmax = nn.Softmax(dim=-1)
+
+    def forward(self, x, mask=None):
+        """
+        Args:
+            x: input features with shape of (num_windows*b, n, c)
+            mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None
+        """
+        b_, n, c = x.shape
+        qkv = self.qkv(x).reshape(b_, n, 3, self.num_heads, c // self.num_heads).permute(2, 0, 3, 1, 4)
+        q, k, v = qkv[0], qkv[1], qkv[2]  # make torchscript happy (cannot use tensor as tuple)
+
+        q = q * self.scale
+        attn = (q @ k.transpose(-2, -1))
+
+        relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
+            self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1)  # Wh*Ww,Wh*Ww,nH
+        relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous()  # nH, Wh*Ww, Wh*Ww
+        attn = attn + relative_position_bias.unsqueeze(0)
+
+        if mask is not None:
+            nw = mask.shape[0]
+            attn = attn.view(b_ // nw, nw, self.num_heads, n, n) + mask.unsqueeze(1).unsqueeze(0)
+            attn = attn.view(-1, self.num_heads, n, n)
+            attn = self.softmax(attn)
+        else:
+            attn = self.softmax(attn)
+
+        attn = self.attn_drop(attn)
+
+        x = (attn @ v).transpose(1, 2).reshape(b_, n, c)
+        x = self.proj(x)
+        x = self.proj_drop(x)
+        return x
+
+    def extra_repr(self) -> str:
+        return f'dim={self.dim}, window_size={self.window_size}, num_heads={self.num_heads}'
+
+    def flops(self, n):
+        # calculate flops for 1 window with token length of n
+        flops = 0
+        # qkv = self.qkv(x)
+        flops += n * self.dim * 3 * self.dim
+        # attn = (q @ k.transpose(-2, -1))
+        flops += self.num_heads * n * (self.dim // self.num_heads) * n
+        #  x = (attn @ v)
+        flops += self.num_heads * n * n * (self.dim // self.num_heads)
+        # x = self.proj(x)
+        flops += n * self.dim * self.dim
+        return flops
+
+
+class SwinTransformerBlock(nn.Module):
+    r""" Swin Transformer Block.
+
+    Args:
+        dim (int): Number of input channels.
+        input_resolution (tuple[int]): Input resolution.
+        num_heads (int): Number of attention heads.
+        window_size (int): Window size.
+        shift_size (int): Shift size for SW-MSA.
+        mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
+        qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True
+        qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set.
+        drop (float, optional): Dropout rate. Default: 0.0
+        attn_drop (float, optional): Attention dropout rate. Default: 0.0
+        drop_path (float, optional): Stochastic depth rate. Default: 0.0
+        act_layer (nn.Module, optional): Activation layer. Default: nn.GELU
+        norm_layer (nn.Module, optional): Normalization layer.  Default: nn.LayerNorm
+    """
+
+    def __init__(self,
+                 dim,
+                 input_resolution,
+                 num_heads,
+                 window_size=7,
+                 shift_size=0,
+                 mlp_ratio=4.,
+                 qkv_bias=True,
+                 qk_scale=None,
+                 drop=0.,
+                 attn_drop=0.,
+                 drop_path=0.,
+                 act_layer=nn.GELU,
+                 norm_layer=nn.LayerNorm):
+        super().__init__()
+        self.dim = dim
+        self.input_resolution = input_resolution
+        self.num_heads = num_heads
+        self.window_size = window_size
+        self.shift_size = shift_size
+        self.mlp_ratio = mlp_ratio
+        if min(self.input_resolution) <= self.window_size:
+            # if window size is larger than input resolution, we don't partition windows
+            self.shift_size = 0
+            self.window_size = min(self.input_resolution)
+        assert 0 <= self.shift_size < self.window_size, 'shift_size must in 0-window_size'
+
+        self.norm1 = norm_layer(dim)
+        self.attn = WindowAttention(
+            dim,
+            window_size=to_2tuple(self.window_size),
+            num_heads=num_heads,
+            qkv_bias=qkv_bias,
+            qk_scale=qk_scale,
+            attn_drop=attn_drop,
+            proj_drop=drop)
+
+        self.drop_path = DropPath(drop_path) if drop_path > 0. else nn.Identity()
+        self.norm2 = norm_layer(dim)
+        mlp_hidden_dim = int(dim * mlp_ratio)
+        self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)
+
+        if self.shift_size > 0:
+            attn_mask = self.calculate_mask(self.input_resolution)
+        else:
+            attn_mask = None
+
+        self.register_buffer('attn_mask', attn_mask)
+
+    def calculate_mask(self, x_size):
+        # calculate attention mask for SW-MSA
+        h, w = x_size
+        img_mask = torch.zeros((1, h, w, 1))  # 1 h w 1
+        h_slices = (slice(0, -self.window_size), slice(-self.window_size,
+                                                       -self.shift_size), slice(-self.shift_size, None))
+        w_slices = (slice(0, -self.window_size), slice(-self.window_size,
+                                                       -self.shift_size), slice(-self.shift_size, None))
+        cnt = 0
+        for h in h_slices:
+            for w in w_slices:
+                img_mask[:, h, w, :] = cnt
+                cnt += 1
+
+        mask_windows = window_partition(img_mask, self.window_size)  # nw, window_size, window_size, 1
+        mask_windows = mask_windows.view(-1, self.window_size * self.window_size)
+        attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)
+        attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
+
+        return attn_mask
+
+    def forward(self, x, x_size):
+        h, w = x_size
+        b, _, c = x.shape
+        # assert seq_len == h * w, "input feature has wrong size"
+
+        shortcut = x
+        x = self.norm1(x)
+        x = x.view(b, h, w, c)
+
+        # cyclic shift
+        if self.shift_size > 0:
+            shifted_x = torch.roll(x, shifts=(-self.shift_size, -self.shift_size), dims=(1, 2))
+        else:
+            shifted_x = x
+
+        # partition windows
+        x_windows = window_partition(shifted_x, self.window_size)  # nw*b, window_size, window_size, c
+        x_windows = x_windows.view(-1, self.window_size * self.window_size, c)  # nw*b, window_size*window_size, c
+
+        # W-MSA/SW-MSA (to be compatible for testing on images whose shapes are the multiple of window size
+        if self.input_resolution == x_size:
+            attn_windows = self.attn(x_windows, mask=self.attn_mask)  # nw*b, window_size*window_size, c
+        else:
+            attn_windows = self.attn(x_windows, mask=self.calculate_mask(x_size).to(x.device))
+
+        # merge windows
+        attn_windows = attn_windows.view(-1, self.window_size, self.window_size, c)
+        shifted_x = window_reverse(attn_windows, self.window_size, h, w)  # b h' w' c
+
+        # reverse cyclic shift
+        if self.shift_size > 0:
+            x = torch.roll(shifted_x, shifts=(self.shift_size, self.shift_size), dims=(1, 2))
+        else:
+            x = shifted_x
+        x = x.view(b, h * w, c)
+
+        # FFN
+        x = shortcut + self.drop_path(x)
+        x = x + self.drop_path(self.mlp(self.norm2(x)))
+
+        return x
+
+    def extra_repr(self) -> str:
+        return (f'dim={self.dim}, input_resolution={self.input_resolution}, num_heads={self.num_heads}, '
+                f'window_size={self.window_size}, shift_size={self.shift_size}, mlp_ratio={self.mlp_ratio}')
+
+    def flops(self):
+        flops = 0
+        h, w = self.input_resolution
+        # norm1
+        flops += self.dim * h * w
+        # W-MSA/SW-MSA
+        nw = h * w / self.window_size / self.window_size
+        flops += nw * self.attn.flops(self.window_size * self.window_size)
+        # mlp
+        flops += 2 * h * w * self.dim * self.dim * self.mlp_ratio
+        # norm2
+        flops += self.dim * h * w
+        return flops
+
+
+class PatchMerging(nn.Module):
+    r""" Patch Merging Layer.
+
+    Args:
+        input_resolution (tuple[int]): Resolution of input feature.
+        dim (int): Number of input channels.
+        norm_layer (nn.Module, optional): Normalization layer.  Default: nn.LayerNorm
+    """
+
+    def __init__(self, input_resolution, dim, norm_layer=nn.LayerNorm):
+        super().__init__()
+        self.input_resolution = input_resolution
+        self.dim = dim
+        self.reduction = nn.Linear(4 * dim, 2 * dim, bias=False)
+        self.norm = norm_layer(4 * dim)
+
+    def forward(self, x):
+        """
+        x: b, h*w, c
+        """
+        h, w = self.input_resolution
+        b, seq_len, c = x.shape
+        assert seq_len == h * w, 'input feature has wrong size'
+        assert h % 2 == 0 and w % 2 == 0, f'x size ({h}*{w}) are not even.'
+
+        x = x.view(b, h, w, c)
+
+        x0 = x[:, 0::2, 0::2, :]  # b h/2 w/2 c
+        x1 = x[:, 1::2, 0::2, :]  # b h/2 w/2 c
+        x2 = x[:, 0::2, 1::2, :]  # b h/2 w/2 c
+        x3 = x[:, 1::2, 1::2, :]  # b h/2 w/2 c
+        x = torch.cat([x0, x1, x2, x3], -1)  # b h/2 w/2 4*c
+        x = x.view(b, -1, 4 * c)  # b h/2*w/2 4*c
+
+        x = self.norm(x)
+        x = self.reduction(x)
+
+        return x
+
+    def extra_repr(self) -> str:
+        return f'input_resolution={self.input_resolution}, dim={self.dim}'
+
+    def flops(self):
+        h, w = self.input_resolution
+        flops = h * w * self.dim
+        flops += (h // 2) * (w // 2) * 4 * self.dim * 2 * self.dim
+        return flops
+
+
+class BasicLayer(nn.Module):
+    """ A basic Swin Transformer layer for one stage.
+
+    Args:
+        dim (int): Number of input channels.
+        input_resolution (tuple[int]): Input resolution.
+        depth (int): Number of blocks.
+        num_heads (int): Number of attention heads.
+        window_size (int): Local window size.
+        mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
+        qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True
+        qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set.
+        drop (float, optional): Dropout rate. Default: 0.0
+        attn_drop (float, optional): Attention dropout rate. Default: 0.0
+        drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0
+        norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm
+        downsample (nn.Module | None, optional): Downsample layer at the end of the layer. Default: None
+        use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False.
+    """
+
+    def __init__(self,
+                 dim,
+                 input_resolution,
+                 depth,
+                 num_heads,
+                 window_size,
+                 mlp_ratio=4.,
+                 qkv_bias=True,
+                 qk_scale=None,
+                 drop=0.,
+                 attn_drop=0.,
+                 drop_path=0.,
+                 norm_layer=nn.LayerNorm,
+                 downsample=None,
+                 use_checkpoint=False):
+
+        super().__init__()
+        self.dim = dim
+        self.input_resolution = input_resolution
+        self.depth = depth
+        self.use_checkpoint = use_checkpoint
+
+        # build blocks
+        self.blocks = nn.ModuleList([
+            SwinTransformerBlock(
+                dim=dim,
+                input_resolution=input_resolution,
+                num_heads=num_heads,
+                window_size=window_size,
+                shift_size=0 if (i % 2 == 0) else window_size // 2,
+                mlp_ratio=mlp_ratio,
+                qkv_bias=qkv_bias,
+                qk_scale=qk_scale,
+                drop=drop,
+                attn_drop=attn_drop,
+                drop_path=drop_path[i] if isinstance(drop_path, list) else drop_path,
+                norm_layer=norm_layer) for i in range(depth)
+        ])
+
+        # patch merging layer
+        if downsample is not None:
+            self.downsample = downsample(input_resolution, dim=dim, norm_layer=norm_layer)
+        else:
+            self.downsample = None
+
+    def forward(self, x, x_size):
+        for blk in self.blocks:
+            if self.use_checkpoint:
+                x = checkpoint.checkpoint(blk, x)
+            else:
+                x = blk(x, x_size)
+        if self.downsample is not None:
+            x = self.downsample(x)
+        return x
+
+    def extra_repr(self) -> str:
+        return f'dim={self.dim}, input_resolution={self.input_resolution}, depth={self.depth}'
+
+    def flops(self):
+        flops = 0
+        for blk in self.blocks:
+            flops += blk.flops()
+        if self.downsample is not None:
+            flops += self.downsample.flops()
+        return flops
+
+
+class RSTB(nn.Module):
+    """Residual Swin Transformer Block (RSTB).
+
+    Args:
+        dim (int): Number of input channels.
+        input_resolution (tuple[int]): Input resolution.
+        depth (int): Number of blocks.
+        num_heads (int): Number of attention heads.
+        window_size (int): Local window size.
+        mlp_ratio (float): Ratio of mlp hidden dim to embedding dim.
+        qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True
+        qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set.
+        drop (float, optional): Dropout rate. Default: 0.0
+        attn_drop (float, optional): Attention dropout rate. Default: 0.0
+        drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0
+        norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm
+        downsample (nn.Module | None, optional): Downsample layer at the end of the layer. Default: None
+        use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False.
+        img_size: Input image size.
+        patch_size: Patch size.
+        resi_connection: The convolutional block before residual connection.
+    """
+
+    def __init__(self,
+                 dim,
+                 input_resolution,
+                 depth,
+                 num_heads,
+                 window_size,
+                 mlp_ratio=4.,
+                 qkv_bias=True,
+                 qk_scale=None,
+                 drop=0.,
+                 attn_drop=0.,
+                 drop_path=0.,
+                 norm_layer=nn.LayerNorm,
+                 downsample=None,
+                 use_checkpoint=False,
+                 img_size=224,
+                 patch_size=4,
+                 resi_connection='1conv'):
+        super(RSTB, self).__init__()
+
+        self.dim = dim
+        self.input_resolution = input_resolution
+
+        self.residual_group = BasicLayer(
+            dim=dim,
+            input_resolution=input_resolution,
+            depth=depth,
+            num_heads=num_heads,
+            window_size=window_size,
+            mlp_ratio=mlp_ratio,
+            qkv_bias=qkv_bias,
+            qk_scale=qk_scale,
+            drop=drop,
+            attn_drop=attn_drop,
+            drop_path=drop_path,
+            norm_layer=norm_layer,
+            downsample=downsample,
+            use_checkpoint=use_checkpoint)
+
+        if resi_connection == '1conv':
+            self.conv = nn.Conv2d(dim, dim, 3, 1, 1)
+        elif resi_connection == '3conv':
+            # to save parameters and memory
+            self.conv = nn.Sequential(
+                nn.Conv2d(dim, dim // 4, 3, 1, 1), nn.LeakyReLU(negative_slope=0.2, inplace=True),
+                nn.Conv2d(dim // 4, dim // 4, 1, 1, 0), nn.LeakyReLU(negative_slope=0.2, inplace=True),
+                nn.Conv2d(dim // 4, dim, 3, 1, 1))
+
+        self.patch_embed = PatchEmbed(
+            img_size=img_size, patch_size=patch_size, in_chans=0, embed_dim=dim, norm_layer=None)
+
+        self.patch_unembed = PatchUnEmbed(
+            img_size=img_size, patch_size=patch_size, in_chans=0, embed_dim=dim, norm_layer=None)
+
+    def forward(self, x, x_size):
+        return self.patch_embed(self.conv(self.patch_unembed(self.residual_group(x, x_size), x_size))) + x
+
+    def flops(self):
+        flops = 0
+        flops += self.residual_group.flops()
+        h, w = self.input_resolution
+        flops += h * w * self.dim * self.dim * 9
+        flops += self.patch_embed.flops()
+        flops += self.patch_unembed.flops()
+
+        return flops
+
+
+class PatchEmbed(nn.Module):
+    r""" Image to Patch Embedding
+
+    Args:
+        img_size (int): Image size.  Default: 224.
+        patch_size (int): Patch token size. Default: 4.
+        in_chans (int): Number of input image channels. Default: 3.
+        embed_dim (int): Number of linear projection output channels. Default: 96.
+        norm_layer (nn.Module, optional): Normalization layer. Default: None
+    """
+
+    def __init__(self, img_size=224, patch_size=4, in_chans=3, embed_dim=96, norm_layer=None):
+        super().__init__()
+        img_size = to_2tuple(img_size)
+        patch_size = to_2tuple(patch_size)
+        patches_resolution = [img_size[0] // patch_size[0], img_size[1] // patch_size[1]]
+        self.img_size = img_size
+        self.patch_size = patch_size
+        self.patches_resolution = patches_resolution
+        self.num_patches = patches_resolution[0] * patches_resolution[1]
+
+        self.in_chans = in_chans
+        self.embed_dim = embed_dim
+
+        if norm_layer is not None:
+            self.norm = norm_layer(embed_dim)
+        else:
+            self.norm = None
+
+    def forward(self, x):
+        x = x.flatten(2).transpose(1, 2)  # b Ph*Pw c
+        if self.norm is not None:
+            x = self.norm(x)
+        return x
+
+    def flops(self):
+        flops = 0
+        h, w = self.img_size
+        if self.norm is not None:
+            flops += h * w * self.embed_dim
+        return flops
+
+
+class PatchUnEmbed(nn.Module):
+    r""" Image to Patch Unembedding
+
+    Args:
+        img_size (int): Image size.  Default: 224.
+        patch_size (int): Patch token size. Default: 4.
+        in_chans (int): Number of input image channels. Default: 3.
+        embed_dim (int): Number of linear projection output channels. Default: 96.
+        norm_layer (nn.Module, optional): Normalization layer. Default: None
+    """
+
+    def __init__(self, img_size=224, patch_size=4, in_chans=3, embed_dim=96, norm_layer=None):
+        super().__init__()
+        img_size = to_2tuple(img_size)
+        patch_size = to_2tuple(patch_size)
+        patches_resolution = [img_size[0] // patch_size[0], img_size[1] // patch_size[1]]
+        self.img_size = img_size
+        self.patch_size = patch_size
+        self.patches_resolution = patches_resolution
+        self.num_patches = patches_resolution[0] * patches_resolution[1]
+
+        self.in_chans = in_chans
+        self.embed_dim = embed_dim
+
+    def forward(self, x, x_size):
+        x = x.transpose(1, 2).view(x.shape[0], self.embed_dim, x_size[0], x_size[1])  # b Ph*Pw c
+        return x
+
+    def flops(self):
+        flops = 0
+        return flops
+
+
+class Upsample(nn.Sequential):
+    """Upsample module.
+
+    Args:
+        scale (int): Scale factor. Supported scales: 2^n and 3.
+        num_feat (int): Channel number of intermediate features.
+    """
+
+    def __init__(self, scale, num_feat):
+        m = []
+        if (scale & (scale - 1)) == 0:  # scale = 2^n
+            for _ in range(int(math.log(scale, 2))):
+                m.append(nn.Conv2d(num_feat, 4 * num_feat, 3, 1, 1))
+                m.append(nn.PixelShuffle(2))
+        elif scale == 3:
+            m.append(nn.Conv2d(num_feat, 9 * num_feat, 3, 1, 1))
+            m.append(nn.PixelShuffle(3))
+        else:
+            raise ValueError(f'scale {scale} is not supported. Supported scales: 2^n and 3.')
+        super(Upsample, self).__init__(*m)
+
+
+class UpsampleOneStep(nn.Sequential):
+    """UpsampleOneStep module (the difference with Upsample is that it always only has 1conv + 1pixelshuffle)
+       Used in lightweight SR to save parameters.
+
+    Args:
+        scale (int): Scale factor. Supported scales: 2^n and 3.
+        num_feat (int): Channel number of intermediate features.
+
+    """
+
+    def __init__(self, scale, num_feat, num_out_ch, input_resolution=None):
+        self.num_feat = num_feat
+        self.input_resolution = input_resolution
+        m = []
+        m.append(nn.Conv2d(num_feat, (scale**2) * num_out_ch, 3, 1, 1))
+        m.append(nn.PixelShuffle(scale))
+        super(UpsampleOneStep, self).__init__(*m)
+
+    def flops(self):
+        h, w = self.input_resolution
+        flops = h * w * self.num_feat * 3 * 9
+        return flops
+
+
+@ARCH_REGISTRY.register()
+class SwinIR(nn.Module):
+    r""" SwinIR
+        A PyTorch impl of : `SwinIR: Image Restoration Using Swin Transformer`, based on Swin Transformer.
+
+    Args:
+        img_size (int | tuple(int)): Input image size. Default 64
+        patch_size (int | tuple(int)): Patch size. Default: 1
+        in_chans (int): Number of input image channels. Default: 3
+        embed_dim (int): Patch embedding dimension. Default: 96
+        depths (tuple(int)): Depth of each Swin Transformer layer.
+        num_heads (tuple(int)): Number of attention heads in different layers.
+        window_size (int): Window size. Default: 7
+        mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4
+        qkv_bias (bool): If True, add a learnable bias to query, key, value. Default: True
+        qk_scale (float): Override default qk scale of head_dim ** -0.5 if set. Default: None
+        drop_rate (float): Dropout rate. Default: 0
+        attn_drop_rate (float): Attention dropout rate. Default: 0
+        drop_path_rate (float): Stochastic depth rate. Default: 0.1
+        norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm.
+        ape (bool): If True, add absolute position embedding to the patch embedding. Default: False
+        patch_norm (bool): If True, add normalization after patch embedding. Default: True
+        use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False
+        upscale: Upscale factor. 2/3/4/8 for image SR, 1 for denoising and compress artifact reduction
+        img_range: Image range. 1. or 255.
+        upsampler: The reconstruction reconstruction module. 'pixelshuffle'/'pixelshuffledirect'/'nearest+conv'/None
+        resi_connection: The convolutional block before residual connection. '1conv'/'3conv'
+    """
+
+    def __init__(self,
+                 img_size=64,
+                 patch_size=1,
+                 in_chans=3,
+                 embed_dim=96,
+                 depths=(6, 6, 6, 6),
+                 num_heads=(6, 6, 6, 6),
+                 window_size=7,
+                 mlp_ratio=4.,
+                 qkv_bias=True,
+                 qk_scale=None,
+                 drop_rate=0.,
+                 attn_drop_rate=0.,
+                 drop_path_rate=0.1,
+                 norm_layer=nn.LayerNorm,
+                 ape=False,
+                 patch_norm=True,
+                 use_checkpoint=False,
+                 upscale=2,
+                 img_range=1.,
+                 upsampler='',
+                 resi_connection='1conv',
+                 **kwargs):
+        super(SwinIR, self).__init__()
+        num_in_ch = in_chans
+        num_out_ch = in_chans
+        num_feat = 64
+        self.img_range = img_range
+        if in_chans == 3:
+            rgb_mean = (0.4488, 0.4371, 0.4040)
+            self.mean = torch.Tensor(rgb_mean).view(1, 3, 1, 1)
+        else:
+            self.mean = torch.zeros(1, 1, 1, 1)
+        self.upscale = upscale
+        self.upsampler = upsampler
+
+        # ------------------------- 1, shallow feature extraction ------------------------- #
+        self.conv_first = nn.Conv2d(num_in_ch, embed_dim, 3, 1, 1)
+
+        # ------------------------- 2, deep feature extraction ------------------------- #
+        self.num_layers = len(depths)
+        self.embed_dim = embed_dim
+        self.ape = ape
+        self.patch_norm = patch_norm
+        self.num_features = embed_dim
+        self.mlp_ratio = mlp_ratio
+
+        # split image into non-overlapping patches
+        self.patch_embed = PatchEmbed(
+            img_size=img_size,
+            patch_size=patch_size,
+            in_chans=embed_dim,
+            embed_dim=embed_dim,
+            norm_layer=norm_layer if self.patch_norm else None)
+        num_patches = self.patch_embed.num_patches
+        patches_resolution = self.patch_embed.patches_resolution
+        self.patches_resolution = patches_resolution
+
+        # merge non-overlapping patches into image
+        self.patch_unembed = PatchUnEmbed(
+            img_size=img_size,
+            patch_size=patch_size,
+            in_chans=embed_dim,
+            embed_dim=embed_dim,
+            norm_layer=norm_layer if self.patch_norm else None)
+
+        # absolute position embedding
+        if self.ape:
+            self.absolute_pos_embed = nn.Parameter(torch.zeros(1, num_patches, embed_dim))
+            trunc_normal_(self.absolute_pos_embed, std=.02)
+
+        self.pos_drop = nn.Dropout(p=drop_rate)
+
+        # stochastic depth
+        dpr = [x.item() for x in torch.linspace(0, drop_path_rate, sum(depths))]  # stochastic depth decay rule
+
+        # build Residual Swin Transformer blocks (RSTB)
+        self.layers = nn.ModuleList()
+        for i_layer in range(self.num_layers):
+            layer = RSTB(
+                dim=embed_dim,
+                input_resolution=(patches_resolution[0], patches_resolution[1]),
+                depth=depths[i_layer],
+                num_heads=num_heads[i_layer],
+                window_size=window_size,
+                mlp_ratio=self.mlp_ratio,
+                qkv_bias=qkv_bias,
+                qk_scale=qk_scale,
+                drop=drop_rate,
+                attn_drop=attn_drop_rate,
+                drop_path=dpr[sum(depths[:i_layer]):sum(depths[:i_layer + 1])],  # no impact on SR results
+                norm_layer=norm_layer,
+                downsample=None,
+                use_checkpoint=use_checkpoint,
+                img_size=img_size,
+                patch_size=patch_size,
+                resi_connection=resi_connection)
+            self.layers.append(layer)
+        self.norm = norm_layer(self.num_features)
+
+        # build the last conv layer in deep feature extraction
+        if resi_connection == '1conv':
+            self.conv_after_body = nn.Conv2d(embed_dim, embed_dim, 3, 1, 1)
+        elif resi_connection == '3conv':
+            # to save parameters and memory
+            self.conv_after_body = nn.Sequential(
+                nn.Conv2d(embed_dim, embed_dim // 4, 3, 1, 1), nn.LeakyReLU(negative_slope=0.2, inplace=True),
+                nn.Conv2d(embed_dim // 4, embed_dim // 4, 1, 1, 0), nn.LeakyReLU(negative_slope=0.2, inplace=True),
+                nn.Conv2d(embed_dim // 4, embed_dim, 3, 1, 1))
+
+        # ------------------------- 3, high quality image reconstruction ------------------------- #
+        if self.upsampler == 'pixelshuffle':
+            # for classical SR
+            self.conv_before_upsample = nn.Sequential(
+                nn.Conv2d(embed_dim, num_feat, 3, 1, 1), nn.LeakyReLU(inplace=True))
+            self.upsample = Upsample(upscale, num_feat)
+            self.conv_last = nn.Conv2d(num_feat, num_out_ch, 3, 1, 1)
+        elif self.upsampler == 'pixelshuffledirect':
+            # for lightweight SR (to save parameters)
+            self.upsample = UpsampleOneStep(upscale, embed_dim, num_out_ch,
+                                            (patches_resolution[0], patches_resolution[1]))
+        elif self.upsampler == 'nearest+conv':
+            # for real-world SR (less artifacts)
+            assert self.upscale == 4, 'only support x4 now.'
+            self.conv_before_upsample = nn.Sequential(
+                nn.Conv2d(embed_dim, num_feat, 3, 1, 1), nn.LeakyReLU(inplace=True))
+            self.conv_up1 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+            self.conv_up2 = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+            self.conv_hr = nn.Conv2d(num_feat, num_feat, 3, 1, 1)
+            self.conv_last = nn.Conv2d(num_feat, num_out_ch, 3, 1, 1)
+            self.lrelu = nn.LeakyReLU(negative_slope=0.2, inplace=True)
+        else:
+            # for image denoising and JPEG compression artifact reduction
+            self.conv_last = nn.Conv2d(embed_dim, num_out_ch, 3, 1, 1)
+
+        self.apply(self._init_weights)
+
+    def _init_weights(self, m):
+        if isinstance(m, nn.Linear):
+            trunc_normal_(m.weight, std=.02)
+            if isinstance(m, nn.Linear) and m.bias is not None:
+                nn.init.constant_(m.bias, 0)
+        elif isinstance(m, nn.LayerNorm):
+            nn.init.constant_(m.bias, 0)
+            nn.init.constant_(m.weight, 1.0)
+
+    @torch.jit.ignore
+    def no_weight_decay(self):
+        return {'absolute_pos_embed'}
+
+    @torch.jit.ignore
+    def no_weight_decay_keywords(self):
+        return {'relative_position_bias_table'}
+
+    def forward_features(self, x):
+        x_size = (x.shape[2], x.shape[3])
+        x = self.patch_embed(x)
+        if self.ape:
+            x = x + self.absolute_pos_embed
+        x = self.pos_drop(x)
+
+        for layer in self.layers:
+            x = layer(x, x_size)
+
+        x = self.norm(x)  # b seq_len c
+        x = self.patch_unembed(x, x_size)
+
+        return x
+
+    def forward(self, x):
+        self.mean = self.mean.type_as(x)
+        x = (x - self.mean) * self.img_range
+
+        if self.upsampler == 'pixelshuffle':
+            # for classical SR
+            x = self.conv_first(x)
+            x = self.conv_after_body(self.forward_features(x)) + x
+            x = self.conv_before_upsample(x)
+            x = self.conv_last(self.upsample(x))
+        elif self.upsampler == 'pixelshuffledirect':
+            # for lightweight SR
+            x = self.conv_first(x)
+            x = self.conv_after_body(self.forward_features(x)) + x
+            x = self.upsample(x)
+        elif self.upsampler == 'nearest+conv':
+            # for real-world SR
+            x = self.conv_first(x)
+            x = self.conv_after_body(self.forward_features(x)) + x
+            x = self.conv_before_upsample(x)
+            x = self.lrelu(self.conv_up1(torch.nn.functional.interpolate(x, scale_factor=2, mode='nearest')))
+            x = self.lrelu(self.conv_up2(torch.nn.functional.interpolate(x, scale_factor=2, mode='nearest')))
+            x = self.conv_last(self.lrelu(self.conv_hr(x)))
+        else:
+            # for image denoising and JPEG compression artifact reduction
+            x_first = self.conv_first(x)
+            res = self.conv_after_body(self.forward_features(x_first)) + x_first
+            x = x + self.conv_last(res)
+
+        x = x / self.img_range + self.mean
+
+        return x
+
+    def flops(self):
+        flops = 0
+        h, w = self.patches_resolution
+        flops += h * w * 3 * self.embed_dim * 9
+        flops += self.patch_embed.flops()
+        for layer in self.layers:
+            flops += layer.flops()
+        flops += h * w * 3 * self.embed_dim * self.embed_dim
+        flops += self.upsample.flops()
+        return flops
+
+
+if __name__ == '__main__':
+    upscale = 4
+    window_size = 8
+    height = (1024 // upscale // window_size + 1) * window_size
+    width = (720 // upscale // window_size + 1) * window_size
+    model = SwinIR(
+        upscale=2,
+        img_size=(height, width),
+        window_size=window_size,
+        img_range=1.,
+        depths=[6, 6, 6, 6],
+        embed_dim=60,
+        num_heads=[6, 6, 6, 6],
+        mlp_ratio=2,
+        upsampler='pixelshuffledirect')
+    print(model)
+    print(height, width, model.flops() / 1e9)
+
+    x = torch.randn((1, 3, height, width))
+    x = model(x)
+    print(x.shape)

basicsr/archs/tof_arch.py

@@ -0,0 +1,172 @@
+import torch
+from torch import nn as nn
+from torch.nn import functional as F
+
+from basicsr.utils.registry import ARCH_REGISTRY
+from .arch_util import flow_warp
+
+
+class BasicModule(nn.Module):
+    """Basic module of SPyNet.
+
+    Note that unlike the architecture in spynet_arch.py, the basic module
+    here contains batch normalization.
+    """
+
+    def __init__(self):
+        super(BasicModule, self).__init__()
+        self.basic_module = nn.Sequential(
+            nn.Conv2d(in_channels=8, out_channels=32, kernel_size=7, stride=1, padding=3, bias=False),
+            nn.BatchNorm2d(32), nn.ReLU(inplace=True),
+            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=7, stride=1, padding=3, bias=False),
+            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
+            nn.Conv2d(in_channels=64, out_channels=32, kernel_size=7, stride=1, padding=3, bias=False),
+            nn.BatchNorm2d(32), nn.ReLU(inplace=True),
+            nn.Conv2d(in_channels=32, out_channels=16, kernel_size=7, stride=1, padding=3, bias=False),
+            nn.BatchNorm2d(16), nn.ReLU(inplace=True),
+            nn.Conv2d(in_channels=16, out_channels=2, kernel_size=7, stride=1, padding=3))
+
+    def forward(self, tensor_input):
+        """
+        Args:
+            tensor_input (Tensor): Input tensor with shape (b, 8, h, w).
+                8 channels contain:
+                [reference image (3), neighbor image (3), initial flow (2)].
+
+        Returns:
+            Tensor: Estimated flow with shape (b, 2, h, w)
+        """
+        return self.basic_module(tensor_input)
+
+
+class SPyNetTOF(nn.Module):
+    """SPyNet architecture for TOF.
+
+    Note that this implementation is specifically for TOFlow. Please use :file:`spynet_arch.py` for general use.
+    They differ in the following aspects:
+
+    1. The basic modules here contain BatchNorm.
+    2. Normalization and denormalization are not done here, as they are done in TOFlow.
+
+    ``Paper: Optical Flow Estimation using a Spatial Pyramid Network``
+
+    Reference: https://github.com/Coldog2333/pytoflow
+
+    Args:
+        load_path (str): Path for pretrained SPyNet. Default: None.
+    """
+
+    def __init__(self, load_path=None):
+        super(SPyNetTOF, self).__init__()
+
+        self.basic_module = nn.ModuleList([BasicModule() for _ in range(4)])
+        if load_path:
+            self.load_state_dict(torch.load(load_path, map_location=lambda storage, loc: storage)['params'])
+
+    def forward(self, ref, supp):
+        """
+        Args:
+            ref (Tensor): Reference image with shape of (b, 3, h, w).
+            supp: The supporting image to be warped: (b, 3, h, w).
+
+        Returns:
+            Tensor: Estimated optical flow: (b, 2, h, w).
+        """
+        num_batches, _, h, w = ref.size()
+        ref = [ref]
+        supp = [supp]
+
+        # generate downsampled frames
+        for _ in range(3):
+            ref.insert(0, F.avg_pool2d(input=ref[0], kernel_size=2, stride=2, count_include_pad=False))
+            supp.insert(0, F.avg_pool2d(input=supp[0], kernel_size=2, stride=2, count_include_pad=False))
+
+        # flow computation
+        flow = ref[0].new_zeros(num_batches, 2, h // 16, w // 16)
+        for i in range(4):
+            flow_up = F.interpolate(input=flow, scale_factor=2, mode='bilinear', align_corners=True) * 2.0
+            flow = flow_up + self.basic_module[i](
+                torch.cat([ref[i], flow_warp(supp[i], flow_up.permute(0, 2, 3, 1)), flow_up], 1))
+        return flow
+
+
+@ARCH_REGISTRY.register()
+class TOFlow(nn.Module):
+    """PyTorch implementation of TOFlow.
+
+    In TOFlow, the LR frames are pre-upsampled and have the same size with the GT frames.
+
+    ``Paper: Video Enhancement with Task-Oriented Flow``
+
+    Reference: https://github.com/anchen1011/toflow
+
+    Reference: https://github.com/Coldog2333/pytoflow
+
+    Args:
+        adapt_official_weights (bool): Whether to adapt the weights translated
+            from the official implementation. Set to false if you want to
+            train from scratch. Default: False
+    """
+
+    def __init__(self, adapt_official_weights=False):
+        super(TOFlow, self).__init__()
+        self.adapt_official_weights = adapt_official_weights
+        self.ref_idx = 0 if adapt_official_weights else 3
+
+        self.register_buffer('mean', torch.Tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
+        self.register_buffer('std', torch.Tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
+
+        # flow estimation module
+        self.spynet = SPyNetTOF()
+
+        # reconstruction module
+        self.conv_1 = nn.Conv2d(3 * 7, 64, 9, 1, 4)
+        self.conv_2 = nn.Conv2d(64, 64, 9, 1, 4)
+        self.conv_3 = nn.Conv2d(64, 64, 1)
+        self.conv_4 = nn.Conv2d(64, 3, 1)
+
+        # activation function
+        self.relu = nn.ReLU(inplace=True)
+
+    def normalize(self, img):
+        return (img - self.mean) / self.std
+
+    def denormalize(self, img):
+        return img * self.std + self.mean
+
+    def forward(self, lrs):
+        """
+        Args:
+            lrs: Input lr frames: (b, 7, 3, h, w).
+
+        Returns:
+            Tensor: SR frame: (b, 3, h, w).
+        """
+        # In the official implementation, the 0-th frame is the reference frame
+        if self.adapt_official_weights:
+            lrs = lrs[:, [3, 0, 1, 2, 4, 5, 6], :, :, :]
+
+        num_batches, num_lrs, _, h, w = lrs.size()
+
+        lrs = self.normalize(lrs.view(-1, 3, h, w))
+        lrs = lrs.view(num_batches, num_lrs, 3, h, w)
+
+        lr_ref = lrs[:, self.ref_idx, :, :, :]
+        lr_aligned = []
+        for i in range(7):  # 7 frames
+            if i == self.ref_idx:
+                lr_aligned.append(lr_ref)
+            else:
+                lr_supp = lrs[:, i, :, :, :]
+                flow = self.spynet(lr_ref, lr_supp)
+                lr_aligned.append(flow_warp(lr_supp, flow.permute(0, 2, 3, 1)))
+
+        # reconstruction
+        hr = torch.stack(lr_aligned, dim=1)
+        hr = hr.view(num_batches, -1, h, w)
+        hr = self.relu(self.conv_1(hr))
+        hr = self.relu(self.conv_2(hr))
+        hr = self.relu(self.conv_3(hr))
+        hr = self.conv_4(hr) + lr_ref
+
+        return self.denormalize(hr)

basicsr/archs/vgg_arch.py

@@ -0,0 +1,161 @@
+import os
+import torch
+from collections import OrderedDict
+from torch import nn as nn
+from torchvision.models import vgg as vgg
+
+from basicsr.utils.registry import ARCH_REGISTRY
+
+VGG_PRETRAIN_PATH = 'experiments/pretrained_models/vgg19-dcbb9e9d.pth'
+NAMES = {
+    'vgg11': [
+        'conv1_1', 'relu1_1', 'pool1', 'conv2_1', 'relu2_1', 'pool2', 'conv3_1', 'relu3_1', 'conv3_2', 'relu3_2',
+        'pool3', 'conv4_1', 'relu4_1', 'conv4_2', 'relu4_2', 'pool4', 'conv5_1', 'relu5_1', 'conv5_2', 'relu5_2',
+        'pool5'
+    ],
+    'vgg13': [
+        'conv1_1', 'relu1_1', 'conv1_2', 'relu1_2', 'pool1', 'conv2_1', 'relu2_1', 'conv2_2', 'relu2_2', 'pool2',
+        'conv3_1', 'relu3_1', 'conv3_2', 'relu3_2', 'pool3', 'conv4_1', 'relu4_1', 'conv4_2', 'relu4_2', 'pool4',
+        'conv5_1', 'relu5_1', 'conv5_2', 'relu5_2', 'pool5'
+    ],
+    'vgg16': [
+        'conv1_1', 'relu1_1', 'conv1_2', 'relu1_2', 'pool1', 'conv2_1', 'relu2_1', 'conv2_2', 'relu2_2', 'pool2',
+        'conv3_1', 'relu3_1', 'conv3_2', 'relu3_2', 'conv3_3', 'relu3_3', 'pool3', 'conv4_1', 'relu4_1', 'conv4_2',
+        'relu4_2', 'conv4_3', 'relu4_3', 'pool4', 'conv5_1', 'relu5_1', 'conv5_2', 'relu5_2', 'conv5_3', 'relu5_3',
+        'pool5'
+    ],
+    'vgg19': [
+        'conv1_1', 'relu1_1', 'conv1_2', 'relu1_2', 'pool1', 'conv2_1', 'relu2_1', 'conv2_2', 'relu2_2', 'pool2',
+        'conv3_1', 'relu3_1', 'conv3_2', 'relu3_2', 'conv3_3', 'relu3_3', 'conv3_4', 'relu3_4', 'pool3', 'conv4_1',
+        'relu4_1', 'conv4_2', 'relu4_2', 'conv4_3', 'relu4_3', 'conv4_4', 'relu4_4', 'pool4', 'conv5_1', 'relu5_1',
+        'conv5_2', 'relu5_2', 'conv5_3', 'relu5_3', 'conv5_4', 'relu5_4', 'pool5'
+    ]
+}
+
+
+def insert_bn(names):
+    """Insert bn layer after each conv.
+
+    Args:
+        names (list): The list of layer names.
+
+    Returns:
+        list: The list of layer names with bn layers.
+    """
+    names_bn = []
+    for name in names:
+        names_bn.append(name)
+        if 'conv' in name:
+            position = name.replace('conv', '')
+            names_bn.append('bn' + position)
+    return names_bn
+
+
+@ARCH_REGISTRY.register()
+class VGGFeatureExtractor(nn.Module):
+    """VGG network for feature extraction.
+
+    In this implementation, we allow users to choose whether use normalization
+    in the input feature and the type of vgg network. Note that the pretrained
+    path must fit the vgg type.
+
+    Args:
+        layer_name_list (list[str]): Forward function returns the corresponding
+            features according to the layer_name_list.
+            Example: {'relu1_1', 'relu2_1', 'relu3_1'}.
+        vgg_type (str): Set the type of vgg network. Default: 'vgg19'.
+        use_input_norm (bool): If True, normalize the input image. Importantly,
+            the input feature must in the range [0, 1]. Default: True.
+        range_norm (bool): If True, norm images with range [-1, 1] to [0, 1].
+            Default: False.
+        requires_grad (bool): If true, the parameters of VGG network will be
+            optimized. Default: False.
+        remove_pooling (bool): If true, the max pooling operations in VGG net
+            will be removed. Default: False.
+        pooling_stride (int): The stride of max pooling operation. Default: 2.
+    """
+
+    def __init__(self,
+                 layer_name_list,
+                 vgg_type='vgg19',
+                 use_input_norm=True,
+                 range_norm=False,
+                 requires_grad=False,
+                 remove_pooling=False,
+                 pooling_stride=2):
+        super(VGGFeatureExtractor, self).__init__()
+
+        self.layer_name_list = layer_name_list
+        self.use_input_norm = use_input_norm
+        self.range_norm = range_norm
+
+        self.names = NAMES[vgg_type.replace('_bn', '')]
+        if 'bn' in vgg_type:
+            self.names = insert_bn(self.names)
+
+        # only borrow layers that will be used to avoid unused params
+        max_idx = 0
+        for v in layer_name_list:
+            idx = self.names.index(v)
+            if idx > max_idx:
+                max_idx = idx
+
+        if os.path.exists(VGG_PRETRAIN_PATH):
+            vgg_net = getattr(vgg, vgg_type)(pretrained=False)
+            state_dict = torch.load(VGG_PRETRAIN_PATH, map_location=lambda storage, loc: storage)
+            vgg_net.load_state_dict(state_dict)
+        else:
+            vgg_net = getattr(vgg, vgg_type)(pretrained=True)
+
+        features = vgg_net.features[:max_idx + 1]
+
+        modified_net = OrderedDict()
+        for k, v in zip(self.names, features):
+            if 'pool' in k:
+                # if remove_pooling is true, pooling operation will be removed
+                if remove_pooling:
+                    continue
+                else:
+                    # in some cases, we may want to change the default stride
+                    modified_net[k] = nn.MaxPool2d(kernel_size=2, stride=pooling_stride)
+            else:
+                modified_net[k] = v
+
+        self.vgg_net = nn.Sequential(modified_net)
+
+        if not requires_grad:
+            self.vgg_net.eval()
+            for param in self.parameters():
+                param.requires_grad = False
+        else:
+            self.vgg_net.train()
+            for param in self.parameters():
+                param.requires_grad = True
+
+        if self.use_input_norm:
+            # the mean is for image with range [0, 1]
+            self.register_buffer('mean', torch.Tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
+            # the std is for image with range [0, 1]
+            self.register_buffer('std', torch.Tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
+
+    def forward(self, x):
+        """Forward function.
+
+        Args:
+            x (Tensor): Input tensor with shape (n, c, h, w).
+
+        Returns:
+            Tensor: Forward results.
+        """
+        if self.range_norm:
+            x = (x + 1) / 2
+        if self.use_input_norm:
+            x = (x - self.mean) / self.std
+
+        output = {}
+        for key, layer in self.vgg_net._modules.items():
+            x = layer(x)
+            if key in self.layer_name_list:
+                output[key] = x.clone()
+
+        return output

basicsr/data/__init__.py

@@ -0,0 +1,101 @@
+import importlib
+import numpy as np
+import random
+import torch
+import torch.utils.data
+from copy import deepcopy
+from functools import partial
+from os import path as osp
+
+from basicsr.data.prefetch_dataloader import PrefetchDataLoader
+from basicsr.utils import get_root_logger, scandir
+from basicsr.utils.dist_util import get_dist_info
+from basicsr.utils.registry import DATASET_REGISTRY
+
+__all__ = ['build_dataset', 'build_dataloader']
+
+# automatically scan and import dataset modules for registry
+# scan all the files under the data folder with '_dataset' in file names
+data_folder = osp.dirname(osp.abspath(__file__))
+dataset_filenames = [osp.splitext(osp.basename(v))[0] for v in scandir(data_folder) if v.endswith('_dataset.py')]
+# import all the dataset modules
+_dataset_modules = [importlib.import_module(f'basicsr.data.{file_name}') for file_name in dataset_filenames]
+
+
+def build_dataset(dataset_opt):
+    """Build dataset from options.
+
+    Args:
+        dataset_opt (dict): Configuration for dataset. It must contain:
+            name (str): Dataset name.
+            type (str): Dataset type.
+    """
+    dataset_opt = deepcopy(dataset_opt)
+    dataset = DATASET_REGISTRY.get(dataset_opt['type'])(dataset_opt)
+    logger = get_root_logger()
+    logger.info(f'Dataset [{dataset.__class__.__name__}] - {dataset_opt["name"]} is built.')
+    return dataset
+
+
+def build_dataloader(dataset, dataset_opt, num_gpu=1, dist=False, sampler=None, seed=None):
+    """Build dataloader.
+
+    Args:
+        dataset (torch.utils.data.Dataset): Dataset.
+        dataset_opt (dict): Dataset options. It contains the following keys:
+            phase (str): 'train' or 'val'.
+            num_worker_per_gpu (int): Number of workers for each GPU.
+            batch_size_per_gpu (int): Training batch size for each GPU.
+        num_gpu (int): Number of GPUs. Used only in the train phase.
+            Default: 1.
+        dist (bool): Whether in distributed training. Used only in the train
+            phase. Default: False.
+        sampler (torch.utils.data.sampler): Data sampler. Default: None.
+        seed (int | None): Seed. Default: None
+    """
+    phase = dataset_opt['phase']
+    rank, _ = get_dist_info()
+    if phase == 'train':
+        if dist:  # distributed training
+            batch_size = dataset_opt['batch_size_per_gpu']
+            num_workers = dataset_opt['num_worker_per_gpu']
+        else:  # non-distributed training
+            multiplier = 1 if num_gpu == 0 else num_gpu
+            batch_size = dataset_opt['batch_size_per_gpu'] * multiplier
+            num_workers = dataset_opt['num_worker_per_gpu'] * multiplier
+        dataloader_args = dict(
+            dataset=dataset,
+            batch_size=batch_size,
+            shuffle=False,
+            num_workers=num_workers,
+            sampler=sampler,
+            drop_last=True)
+        if sampler is None:
+            dataloader_args['shuffle'] = True
+        dataloader_args['worker_init_fn'] = partial(
+            worker_init_fn, num_workers=num_workers, rank=rank, seed=seed) if seed is not None else None
+    elif phase in ['val', 'test']:  # validation
+        dataloader_args = dict(dataset=dataset, batch_size=1, shuffle=False, num_workers=0)
+    else:
+        raise ValueError(f"Wrong dataset phase: {phase}. Supported ones are 'train', 'val' and 'test'.")
+
+    dataloader_args['pin_memory'] = dataset_opt.get('pin_memory', False)
+    dataloader_args['persistent_workers'] = dataset_opt.get('persistent_workers', False)
+
+    prefetch_mode = dataset_opt.get('prefetch_mode')
+    if prefetch_mode == 'cpu':  # CPUPrefetcher
+        num_prefetch_queue = dataset_opt.get('num_prefetch_queue', 1)
+        logger = get_root_logger()
+        logger.info(f'Use {prefetch_mode} prefetch dataloader: num_prefetch_queue = {num_prefetch_queue}')
+        return PrefetchDataLoader(num_prefetch_queue=num_prefetch_queue, **dataloader_args)
+    else:
+        # prefetch_mode=None: Normal dataloader
+        # prefetch_mode='cuda': dataloader for CUDAPrefetcher
+        return torch.utils.data.DataLoader(**dataloader_args)
+
+
+def worker_init_fn(worker_id, num_workers, rank, seed):
+    # Set the worker seed to num_workers * rank + worker_id + seed
+    worker_seed = num_workers * rank + worker_id + seed
+    np.random.seed(worker_seed)
+    random.seed(worker_seed)

basicsr/data/data_sampler.py

@@ -0,0 +1,48 @@
+import math
+import torch
+from torch.utils.data.sampler import Sampler
+
+
+class EnlargedSampler(Sampler):
+    """Sampler that restricts data loading to a subset of the dataset.
+
+    Modified from torch.utils.data.distributed.DistributedSampler
+    Support enlarging the dataset for iteration-based training, for saving
+    time when restart the dataloader after each epoch
+
+    Args:
+        dataset (torch.utils.data.Dataset): Dataset used for sampling.
+        num_replicas (int | None): Number of processes participating in
+            the training. It is usually the world_size.
+        rank (int | None): Rank of the current process within num_replicas.
+        ratio (int): Enlarging ratio. Default: 1.
+    """
+
+    def __init__(self, dataset, num_replicas, rank, ratio=1):
+        self.dataset = dataset
+        self.num_replicas = num_replicas
+        self.rank = rank
+        self.epoch = 0
+        self.num_samples = math.ceil(len(self.dataset) * ratio / self.num_replicas)
+        self.total_size = self.num_samples * self.num_replicas
+
+    def __iter__(self):
+        # deterministically shuffle based on epoch
+        g = torch.Generator()
+        g.manual_seed(self.epoch)
+        indices = torch.randperm(self.total_size, generator=g).tolist()
+
+        dataset_size = len(self.dataset)
+        indices = [v % dataset_size for v in indices]
+
+        # subsample
+        indices = indices[self.rank:self.total_size:self.num_replicas]
+        assert len(indices) == self.num_samples
+
+        return iter(indices)
+
+    def __len__(self):
+        return self.num_samples
+
+    def set_epoch(self, epoch):
+        self.epoch = epoch

basicsr/data/data_util.py

@@ -0,0 +1,315 @@
+import cv2
+import numpy as np
+import torch
+from os import path as osp
+from torch.nn import functional as F
+
+from basicsr.data.transforms import mod_crop
+from basicsr.utils import img2tensor, scandir
+
+
+def read_img_seq(path, require_mod_crop=False, scale=1, return_imgname=False):
+    """Read a sequence of images from a given folder path.
+
+    Args:
+        path (list[str] | str): List of image paths or image folder path.
+        require_mod_crop (bool): Require mod crop for each image.
+            Default: False.
+        scale (int): Scale factor for mod_crop. Default: 1.
+        return_imgname(bool): Whether return image names. Default False.
+
+    Returns:
+        Tensor: size (t, c, h, w), RGB, [0, 1].
+        list[str]: Returned image name list.
+    """
+    if isinstance(path, list):
+        img_paths = path
+    else:
+        img_paths = sorted(list(scandir(path, full_path=True)))
+    imgs = [cv2.imread(v).astype(np.float32) / 255. for v in img_paths]
+
+    if require_mod_crop:
+        imgs = [mod_crop(img, scale) for img in imgs]
+    imgs = img2tensor(imgs, bgr2rgb=True, float32=True)
+    imgs = torch.stack(imgs, dim=0)
+
+    if return_imgname:
+        imgnames = [osp.splitext(osp.basename(path))[0] for path in img_paths]
+        return imgs, imgnames
+    else:
+        return imgs
+
+
+def generate_frame_indices(crt_idx, max_frame_num, num_frames, padding='reflection'):
+    """Generate an index list for reading `num_frames` frames from a sequence
+    of images.
+
+    Args:
+        crt_idx (int): Current center index.
+        max_frame_num (int): Max number of the sequence of images (from 1).
+        num_frames (int): Reading num_frames frames.
+        padding (str): Padding mode, one of
+            'replicate' | 'reflection' | 'reflection_circle' | 'circle'
+            Examples: current_idx = 0, num_frames = 5
+            The generated frame indices under different padding mode:
+            replicate: [0, 0, 0, 1, 2]
+            reflection: [2, 1, 0, 1, 2]
+            reflection_circle: [4, 3, 0, 1, 2]
+            circle: [3, 4, 0, 1, 2]
+
+    Returns:
+        list[int]: A list of indices.
+    """
+    assert num_frames % 2 == 1, 'num_frames should be an odd number.'
+    assert padding in ('replicate', 'reflection', 'reflection_circle', 'circle'), f'Wrong padding mode: {padding}.'
+
+    max_frame_num = max_frame_num - 1  # start from 0
+    num_pad = num_frames // 2
+
+    indices = []
+    for i in range(crt_idx - num_pad, crt_idx + num_pad + 1):
+        if i < 0:
+            if padding == 'replicate':
+                pad_idx = 0
+            elif padding == 'reflection':
+                pad_idx = -i
+            elif padding == 'reflection_circle':
+                pad_idx = crt_idx + num_pad - i
+            else:
+                pad_idx = num_frames + i
+        elif i > max_frame_num:
+            if padding == 'replicate':
+                pad_idx = max_frame_num
+            elif padding == 'reflection':
+                pad_idx = max_frame_num * 2 - i
+            elif padding == 'reflection_circle':
+                pad_idx = (crt_idx - num_pad) - (i - max_frame_num)
+            else:
+                pad_idx = i - num_frames
+        else:
+            pad_idx = i
+        indices.append(pad_idx)
+    return indices
+
+
+def paired_paths_from_lmdb(folders, keys):
+    """Generate paired paths from lmdb files.
+
+    Contents of lmdb. Taking the `lq.lmdb` for example, the file structure is:
+
+    ::
+
+        lq.lmdb
+        ├── data.mdb
+        ├── lock.mdb
+        ├── meta_info.txt
+
+    The data.mdb and lock.mdb are standard lmdb files and you can refer to
+    https://lmdb.readthedocs.io/en/release/ for more details.
+
+    The meta_info.txt is a specified txt file to record the meta information
+    of our datasets. It will be automatically created when preparing
+    datasets by our provided dataset tools.
+    Each line in the txt file records
+    1)image name (with extension),
+    2)image shape,
+    3)compression level, separated by a white space.
+    Example: `baboon.png (120,125,3) 1`
+
+    We use the image name without extension as the lmdb key.
+    Note that we use the same key for the corresponding lq and gt images.
+
+    Args:
+        folders (list[str]): A list of folder path. The order of list should
+            be [input_folder, gt_folder].
+        keys (list[str]): A list of keys identifying folders. The order should
+            be in consistent with folders, e.g., ['lq', 'gt'].
+            Note that this key is different from lmdb keys.
+
+    Returns:
+        list[str]: Returned path list.
+    """
+    assert len(folders) == 2, ('The len of folders should be 2 with [input_folder, gt_folder]. '
+                               f'But got {len(folders)}')
+    assert len(keys) == 2, f'The len of keys should be 2 with [input_key, gt_key]. But got {len(keys)}'
+    input_folder, gt_folder = folders
+    input_key, gt_key = keys
+
+    if not (input_folder.endswith('.lmdb') and gt_folder.endswith('.lmdb')):
+        raise ValueError(f'{input_key} folder and {gt_key} folder should both in lmdb '
+                         f'formats. But received {input_key}: {input_folder}; '
+                         f'{gt_key}: {gt_folder}')
+    # ensure that the two meta_info files are the same
+    with open(osp.join(input_folder, 'meta_info.txt')) as fin:
+        input_lmdb_keys = [line.split('.')[0] for line in fin]
+    with open(osp.join(gt_folder, 'meta_info.txt')) as fin:
+        gt_lmdb_keys = [line.split('.')[0] for line in fin]
+    if set(input_lmdb_keys) != set(gt_lmdb_keys):
+        raise ValueError(f'Keys in {input_key}_folder and {gt_key}_folder are different.')
+    else:
+        paths = []
+        for lmdb_key in sorted(input_lmdb_keys):
+            paths.append(dict([(f'{input_key}_path', lmdb_key), (f'{gt_key}_path', lmdb_key)]))
+        return paths
+
+
+def paired_paths_from_meta_info_file(folders, keys, meta_info_file, filename_tmpl):
+    """Generate paired paths from an meta information file.
+
+    Each line in the meta information file contains the image names and
+    image shape (usually for gt), separated by a white space.
+
+    Example of an meta information file:
+    ```
+    0001_s001.png (480,480,3)
+    0001_s002.png (480,480,3)
+    ```
+
+    Args:
+        folders (list[str]): A list of folder path. The order of list should
+            be [input_folder, gt_folder].
+        keys (list[str]): A list of keys identifying folders. The order should
+            be in consistent with folders, e.g., ['lq', 'gt'].
+        meta_info_file (str): Path to the meta information file.
+        filename_tmpl (str): Template for each filename. Note that the
+            template excludes the file extension. Usually the filename_tmpl is
+            for files in the input folder.
+
+    Returns:
+        list[str]: Returned path list.
+    """
+    assert len(folders) == 2, ('The len of folders should be 2 with [input_folder, gt_folder]. '
+                               f'But got {len(folders)}')
+    assert len(keys) == 2, f'The len of keys should be 2 with [input_key, gt_key]. But got {len(keys)}'
+    input_folder, gt_folder = folders
+    input_key, gt_key = keys
+
+    with open(meta_info_file, 'r') as fin:
+        gt_names = [line.strip().split(' ')[0] for line in fin]
+
+    paths = []
+    for gt_name in gt_names:
+        basename, ext = osp.splitext(osp.basename(gt_name))
+        input_name = f'{filename_tmpl.format(basename)}{ext}'
+        input_path = osp.join(input_folder, input_name)
+        gt_path = osp.join(gt_folder, gt_name)
+        paths.append(dict([(f'{input_key}_path', input_path), (f'{gt_key}_path', gt_path)]))
+    return paths
+
+
+def paired_paths_from_folder(folders, keys, filename_tmpl):
+    """Generate paired paths from folders.
+
+    Args:
+        folders (list[str]): A list of folder path. The order of list should
+            be [input_folder, gt_folder].
+        keys (list[str]): A list of keys identifying folders. The order should
+            be in consistent with folders, e.g., ['lq', 'gt'].
+        filename_tmpl (str): Template for each filename. Note that the
+            template excludes the file extension. Usually the filename_tmpl is
+            for files in the input folder.
+
+    Returns:
+        list[str]: Returned path list.
+    """
+    assert len(folders) == 2, ('The len of folders should be 2 with [input_folder, gt_folder]. '
+                               f'But got {len(folders)}')
+    assert len(keys) == 2, f'The len of keys should be 2 with [input_key, gt_key]. But got {len(keys)}'
+    input_folder, gt_folder = folders
+    input_key, gt_key = keys
+
+    input_paths = list(scandir(input_folder))
+    gt_paths = list(scandir(gt_folder))
+    assert len(input_paths) == len(gt_paths), (f'{input_key} and {gt_key} datasets have different number of images: '
+                                               f'{len(input_paths)}, {len(gt_paths)}.')
+    paths = []
+    for gt_path in gt_paths:
+        basename, ext = osp.splitext(osp.basename(gt_path))
+        input_name = f'{filename_tmpl.format(basename)}{ext}'
+        input_path = osp.join(input_folder, input_name)
+        assert input_name in input_paths, f'{input_name} is not in {input_key}_paths.'
+        gt_path = osp.join(gt_folder, gt_path)
+        paths.append(dict([(f'{input_key}_path', input_path), (f'{gt_key}_path', gt_path)]))
+    return paths
+
+
+def paths_from_folder(folder):
+    """Generate paths from folder.
+
+    Args:
+        folder (str): Folder path.
+
+    Returns:
+        list[str]: Returned path list.
+    """
+
+    paths = list(scandir(folder))
+    paths = [osp.join(folder, path) for path in paths]
+    return paths
+
+
+def paths_from_lmdb(folder):
+    """Generate paths from lmdb.
+
+    Args:
+        folder (str): Folder path.
+
+    Returns:
+        list[str]: Returned path list.
+    """
+    if not folder.endswith('.lmdb'):
+        raise ValueError(f'Folder {folder}folder should in lmdb format.')
+    with open(osp.join(folder, 'meta_info.txt')) as fin:
+        paths = [line.split('.')[0] for line in fin]
+    return paths
+
+
+def generate_gaussian_kernel(kernel_size=13, sigma=1.6):
+    """Generate Gaussian kernel used in `duf_downsample`.
+
+    Args:
+        kernel_size (int): Kernel size. Default: 13.
+        sigma (float): Sigma of the Gaussian kernel. Default: 1.6.
+
+    Returns:
+        np.array: The Gaussian kernel.
+    """
+    from scipy.ndimage import filters as filters
+    kernel = np.zeros((kernel_size, kernel_size))
+    # set element at the middle to one, a dirac delta
+    kernel[kernel_size // 2, kernel_size // 2] = 1
+    # gaussian-smooth the dirac, resulting in a gaussian filter
+    return filters.gaussian_filter(kernel, sigma)
+
+
+def duf_downsample(x, kernel_size=13, scale=4):
+    """Downsamping with Gaussian kernel used in the DUF official code.
+
+    Args:
+        x (Tensor): Frames to be downsampled, with shape (b, t, c, h, w).
+        kernel_size (int): Kernel size. Default: 13.
+        scale (int): Downsampling factor. Supported scale: (2, 3, 4).
+            Default: 4.
+
+    Returns:
+        Tensor: DUF downsampled frames.
+    """
+    assert scale in (2, 3, 4), f'Only support scale (2, 3, 4), but got {scale}.'
+
+    squeeze_flag = False
+    if x.ndim == 4:
+        squeeze_flag = True
+        x = x.unsqueeze(0)
+    b, t, c, h, w = x.size()
+    x = x.view(-1, 1, h, w)
+    pad_w, pad_h = kernel_size // 2 + scale * 2, kernel_size // 2 + scale * 2
+    x = F.pad(x, (pad_w, pad_w, pad_h, pad_h), 'reflect')
+
+    gaussian_filter = generate_gaussian_kernel(kernel_size, 0.4 * scale)
+    gaussian_filter = torch.from_numpy(gaussian_filter).type_as(x).unsqueeze(0).unsqueeze(0)
+    x = F.conv2d(x, gaussian_filter, stride=scale)
+    x = x[:, :, 2:-2, 2:-2]
+    x = x.view(b, t, c, x.size(2), x.size(3))
+    if squeeze_flag:
+        x = x.squeeze(0)
+    return x

basicsr/data/degradations.py

@@ -0,0 +1,764 @@
+import cv2
+import math
+import numpy as np
+import random
+import torch
+from scipy import special
+from scipy.stats import multivariate_normal
+from torchvision.transforms.functional_tensor import rgb_to_grayscale
+
+# -------------------------------------------------------------------- #
+# --------------------------- blur kernels --------------------------- #
+# -------------------------------------------------------------------- #
+
+
+# --------------------------- util functions --------------------------- #
+def sigma_matrix2(sig_x, sig_y, theta):
+    """Calculate the rotated sigma matrix (two dimensional matrix).
+
+    Args:
+        sig_x (float):
+        sig_y (float):
+        theta (float): Radian measurement.
+
+    Returns:
+        ndarray: Rotated sigma matrix.
+    """
+    d_matrix = np.array([[sig_x**2, 0], [0, sig_y**2]])
+    u_matrix = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])
+    return np.dot(u_matrix, np.dot(d_matrix, u_matrix.T))
+
+
+def mesh_grid(kernel_size):
+    """Generate the mesh grid, centering at zero.
+
+    Args:
+        kernel_size (int):
+
+    Returns:
+        xy (ndarray): with the shape (kernel_size, kernel_size, 2)
+        xx (ndarray): with the shape (kernel_size, kernel_size)
+        yy (ndarray): with the shape (kernel_size, kernel_size)
+    """
+    ax = np.arange(-kernel_size // 2 + 1., kernel_size // 2 + 1.)
+    xx, yy = np.meshgrid(ax, ax)
+    xy = np.hstack((xx.reshape((kernel_size * kernel_size, 1)), yy.reshape(kernel_size * kernel_size,
+                                                                           1))).reshape(kernel_size, kernel_size, 2)
+    return xy, xx, yy
+
+
+def pdf2(sigma_matrix, grid):
+    """Calculate PDF of the bivariate Gaussian distribution.
+
+    Args:
+        sigma_matrix (ndarray): with the shape (2, 2)
+        grid (ndarray): generated by :func:`mesh_grid`,
+            with the shape (K, K, 2), K is the kernel size.
+
+    Returns:
+        kernel (ndarrray): un-normalized kernel.
+    """
+    inverse_sigma = np.linalg.inv(sigma_matrix)
+    kernel = np.exp(-0.5 * np.sum(np.dot(grid, inverse_sigma) * grid, 2))
+    return kernel
+
+
+def cdf2(d_matrix, grid):
+    """Calculate the CDF of the standard bivariate Gaussian distribution.
+        Used in skewed Gaussian distribution.
+
+    Args:
+        d_matrix (ndarrasy): skew matrix.
+        grid (ndarray): generated by :func:`mesh_grid`,
+            with the shape (K, K, 2), K is the kernel size.
+
+    Returns:
+        cdf (ndarray): skewed cdf.
+    """
+    rv = multivariate_normal([0, 0], [[1, 0], [0, 1]])
+    grid = np.dot(grid, d_matrix)
+    cdf = rv.cdf(grid)
+    return cdf
+
+
+def bivariate_Gaussian(kernel_size, sig_x, sig_y, theta, grid=None, isotropic=True):
+    """Generate a bivariate isotropic or anisotropic Gaussian kernel.
+
+    In the isotropic mode, only `sig_x` is used. `sig_y` and `theta` is ignored.
+
+    Args:
+        kernel_size (int):
+        sig_x (float):
+        sig_y (float):
+        theta (float): Radian measurement.
+        grid (ndarray, optional): generated by :func:`mesh_grid`,
+            with the shape (K, K, 2), K is the kernel size. Default: None
+        isotropic (bool):
+
+    Returns:
+        kernel (ndarray): normalized kernel.
+    """
+    if grid is None:
+        grid, _, _ = mesh_grid(kernel_size)
+    if isotropic:
+        sigma_matrix = np.array([[sig_x**2, 0], [0, sig_x**2]])
+    else:
+        sigma_matrix = sigma_matrix2(sig_x, sig_y, theta)
+    kernel = pdf2(sigma_matrix, grid)
+    kernel = kernel / np.sum(kernel)
+    return kernel
+
+
+def bivariate_generalized_Gaussian(kernel_size, sig_x, sig_y, theta, beta, grid=None, isotropic=True):
+    """Generate a bivariate generalized Gaussian kernel.
+
+    ``Paper: Parameter Estimation For Multivariate Generalized Gaussian Distributions``
+
+    In the isotropic mode, only `sig_x` is used. `sig_y` and `theta` is ignored.
+
+    Args:
+        kernel_size (int):
+        sig_x (float):
+        sig_y (float):
+        theta (float): Radian measurement.
+        beta (float): shape parameter, beta = 1 is the normal distribution.
+        grid (ndarray, optional): generated by :func:`mesh_grid`,
+            with the shape (K, K, 2), K is the kernel size. Default: None
+
+    Returns:
+        kernel (ndarray): normalized kernel.
+    """
+    if grid is None:
+        grid, _, _ = mesh_grid(kernel_size)
+    if isotropic:
+        sigma_matrix = np.array([[sig_x**2, 0], [0, sig_x**2]])
+    else:
+        sigma_matrix = sigma_matrix2(sig_x, sig_y, theta)
+    inverse_sigma = np.linalg.inv(sigma_matrix)
+    kernel = np.exp(-0.5 * np.power(np.sum(np.dot(grid, inverse_sigma) * grid, 2), beta))
+    kernel = kernel / np.sum(kernel)
+    return kernel
+
+
+def bivariate_plateau(kernel_size, sig_x, sig_y, theta, beta, grid=None, isotropic=True):
+    """Generate a plateau-like anisotropic kernel.
+
+    1 / (1+x^(beta))
+
+    Reference: https://stats.stackexchange.com/questions/203629/is-there-a-plateau-shaped-distribution
+
+    In the isotropic mode, only `sig_x` is used. `sig_y` and `theta` is ignored.
+
+    Args:
+        kernel_size (int):
+        sig_x (float):
+        sig_y (float):
+        theta (float): Radian measurement.
+        beta (float): shape parameter, beta = 1 is the normal distribution.
+        grid (ndarray, optional): generated by :func:`mesh_grid`,
+            with the shape (K, K, 2), K is the kernel size. Default: None
+
+    Returns:
+        kernel (ndarray): normalized kernel.
+    """
+    if grid is None:
+        grid, _, _ = mesh_grid(kernel_size)
+    if isotropic:
+        sigma_matrix = np.array([[sig_x**2, 0], [0, sig_x**2]])
+    else:
+        sigma_matrix = sigma_matrix2(sig_x, sig_y, theta)
+    inverse_sigma = np.linalg.inv(sigma_matrix)
+    kernel = np.reciprocal(np.power(np.sum(np.dot(grid, inverse_sigma) * grid, 2), beta) + 1)
+    kernel = kernel / np.sum(kernel)
+    return kernel
+
+
+def random_bivariate_Gaussian(kernel_size,
+                              sigma_x_range,
+                              sigma_y_range,
+                              rotation_range,
+                              noise_range=None,
+                              isotropic=True):
+    """Randomly generate bivariate isotropic or anisotropic Gaussian kernels.
+
+    In the isotropic mode, only `sigma_x_range` is used. `sigma_y_range` and `rotation_range` is ignored.
+
+    Args:
+        kernel_size (int):
+        sigma_x_range (tuple): [0.6, 5]
+        sigma_y_range (tuple): [0.6, 5]
+        rotation range (tuple): [-math.pi, math.pi]
+        noise_range(tuple, optional): multiplicative kernel noise,
+            [0.75, 1.25]. Default: None
+
+    Returns:
+        kernel (ndarray):
+    """
+    assert kernel_size % 2 == 1, 'Kernel size must be an odd number.'
+    assert sigma_x_range[0] < sigma_x_range[1], 'Wrong sigma_x_range.'
+    sigma_x = np.random.uniform(sigma_x_range[0], sigma_x_range[1])
+    if isotropic is False:
+        assert sigma_y_range[0] < sigma_y_range[1], 'Wrong sigma_y_range.'
+        assert rotation_range[0] < rotation_range[1], 'Wrong rotation_range.'
+        sigma_y = np.random.uniform(sigma_y_range[0], sigma_y_range[1])
+        rotation = np.random.uniform(rotation_range[0], rotation_range[1])
+    else:
+        sigma_y = sigma_x
+        rotation = 0
+
+    kernel = bivariate_Gaussian(kernel_size, sigma_x, sigma_y, rotation, isotropic=isotropic)
+
+    # add multiplicative noise
+    if noise_range is not None:
+        assert noise_range[0] < noise_range[1], 'Wrong noise range.'
+        noise = np.random.uniform(noise_range[0], noise_range[1], size=kernel.shape)
+        kernel = kernel * noise
+    kernel = kernel / np.sum(kernel)
+    return kernel
+
+
+def random_bivariate_generalized_Gaussian(kernel_size,
+                                          sigma_x_range,
+                                          sigma_y_range,
+                                          rotation_range,
+                                          beta_range,
+                                          noise_range=None,
+                                          isotropic=True):
+    """Randomly generate bivariate generalized Gaussian kernels.
+
+    In the isotropic mode, only `sigma_x_range` is used. `sigma_y_range` and `rotation_range` is ignored.
+
+    Args:
+        kernel_size (int):
+        sigma_x_range (tuple): [0.6, 5]
+        sigma_y_range (tuple): [0.6, 5]
+        rotation range (tuple): [-math.pi, math.pi]
+        beta_range (tuple): [0.5, 8]
+        noise_range(tuple, optional): multiplicative kernel noise,
+            [0.75, 1.25]. Default: None
+
+    Returns:
+        kernel (ndarray):
+    """
+    assert kernel_size % 2 == 1, 'Kernel size must be an odd number.'
+    assert sigma_x_range[0] < sigma_x_range[1], 'Wrong sigma_x_range.'
+    sigma_x = np.random.uniform(sigma_x_range[0], sigma_x_range[1])
+    if isotropic is False:
+        assert sigma_y_range[0] < sigma_y_range[1], 'Wrong sigma_y_range.'
+        assert rotation_range[0] < rotation_range[1], 'Wrong rotation_range.'
+        sigma_y = np.random.uniform(sigma_y_range[0], sigma_y_range[1])
+        rotation = np.random.uniform(rotation_range[0], rotation_range[1])
+    else:
+        sigma_y = sigma_x
+        rotation = 0
+
+    # assume beta_range[0] < 1 < beta_range[1]
+    if np.random.uniform() < 0.5:
+        beta = np.random.uniform(beta_range[0], 1)
+    else:
+        beta = np.random.uniform(1, beta_range[1])
+
+    kernel = bivariate_generalized_Gaussian(kernel_size, sigma_x, sigma_y, rotation, beta, isotropic=isotropic)
+
+    # add multiplicative noise
+    if noise_range is not None:
+        assert noise_range[0] < noise_range[1], 'Wrong noise range.'
+        noise = np.random.uniform(noise_range[0], noise_range[1], size=kernel.shape)
+        kernel = kernel * noise
+    kernel = kernel / np.sum(kernel)
+    return kernel
+
+
+def random_bivariate_plateau(kernel_size,
+                             sigma_x_range,
+                             sigma_y_range,
+                             rotation_range,
+                             beta_range,
+                             noise_range=None,
+                             isotropic=True):
+    """Randomly generate bivariate plateau kernels.
+
+    In the isotropic mode, only `sigma_x_range` is used. `sigma_y_range` and `rotation_range` is ignored.
+
+    Args:
+        kernel_size (int):
+        sigma_x_range (tuple): [0.6, 5]
+        sigma_y_range (tuple): [0.6, 5]
+        rotation range (tuple): [-math.pi/2, math.pi/2]
+        beta_range (tuple): [1, 4]
+        noise_range(tuple, optional): multiplicative kernel noise,
+            [0.75, 1.25]. Default: None
+
+    Returns:
+        kernel (ndarray):
+    """
+    assert kernel_size % 2 == 1, 'Kernel size must be an odd number.'
+    assert sigma_x_range[0] < sigma_x_range[1], 'Wrong sigma_x_range.'
+    sigma_x = np.random.uniform(sigma_x_range[0], sigma_x_range[1])
+    if isotropic is False:
+        assert sigma_y_range[0] < sigma_y_range[1], 'Wrong sigma_y_range.'
+        assert rotation_range[0] < rotation_range[1], 'Wrong rotation_range.'
+        sigma_y = np.random.uniform(sigma_y_range[0], sigma_y_range[1])
+        rotation = np.random.uniform(rotation_range[0], rotation_range[1])
+    else:
+        sigma_y = sigma_x
+        rotation = 0
+
+    # TODO: this may be not proper
+    if np.random.uniform() < 0.5:
+        beta = np.random.uniform(beta_range[0], 1)
+    else:
+        beta = np.random.uniform(1, beta_range[1])
+
+    kernel = bivariate_plateau(kernel_size, sigma_x, sigma_y, rotation, beta, isotropic=isotropic)
+    # add multiplicative noise
+    if noise_range is not None:
+        assert noise_range[0] < noise_range[1], 'Wrong noise range.'
+        noise = np.random.uniform(noise_range[0], noise_range[1], size=kernel.shape)
+        kernel = kernel * noise
+    kernel = kernel / np.sum(kernel)
+
+    return kernel
+
+
+def random_mixed_kernels(kernel_list,
+                         kernel_prob,
+                         kernel_size=21,
+                         sigma_x_range=(0.6, 5),
+                         sigma_y_range=(0.6, 5),
+                         rotation_range=(-math.pi, math.pi),
+                         betag_range=(0.5, 8),
+                         betap_range=(0.5, 8),
+                         noise_range=None):
+    """Randomly generate mixed kernels.
+
+    Args:
+        kernel_list (tuple): a list name of kernel types,
+            support ['iso', 'aniso', 'skew', 'generalized', 'plateau_iso',
+            'plateau_aniso']
+        kernel_prob (tuple): corresponding kernel probability for each
+            kernel type
+        kernel_size (int):
+        sigma_x_range (tuple): [0.6, 5]
+        sigma_y_range (tuple): [0.6, 5]
+        rotation range (tuple): [-math.pi, math.pi]
+        beta_range (tuple): [0.5, 8]
+        noise_range(tuple, optional): multiplicative kernel noise,
+            [0.75, 1.25]. Default: None
+
+    Returns:
+        kernel (ndarray):
+    """
+    kernel_type = random.choices(kernel_list, kernel_prob)[0]
+    if kernel_type == 'iso':
+        kernel = random_bivariate_Gaussian(
+            kernel_size, sigma_x_range, sigma_y_range, rotation_range, noise_range=noise_range, isotropic=True)
+    elif kernel_type == 'aniso':
+        kernel = random_bivariate_Gaussian(
+            kernel_size, sigma_x_range, sigma_y_range, rotation_range, noise_range=noise_range, isotropic=False)
+    elif kernel_type == 'generalized_iso':
+        kernel = random_bivariate_generalized_Gaussian(
+            kernel_size,
+            sigma_x_range,
+            sigma_y_range,
+            rotation_range,
+            betag_range,
+            noise_range=noise_range,
+            isotropic=True)
+    elif kernel_type == 'generalized_aniso':
+        kernel = random_bivariate_generalized_Gaussian(
+            kernel_size,
+            sigma_x_range,
+            sigma_y_range,
+            rotation_range,
+            betag_range,
+            noise_range=noise_range,
+            isotropic=False)
+    elif kernel_type == 'plateau_iso':
+        kernel = random_bivariate_plateau(
+            kernel_size, sigma_x_range, sigma_y_range, rotation_range, betap_range, noise_range=None, isotropic=True)
+    elif kernel_type == 'plateau_aniso':
+        kernel = random_bivariate_plateau(
+            kernel_size, sigma_x_range, sigma_y_range, rotation_range, betap_range, noise_range=None, isotropic=False)
+    return kernel
+
+
+np.seterr(divide='ignore', invalid='ignore')
+
+
+def circular_lowpass_kernel(cutoff, kernel_size, pad_to=0):
+    """2D sinc filter
+
+    Reference: https://dsp.stackexchange.com/questions/58301/2-d-circularly-symmetric-low-pass-filter
+
+    Args:
+        cutoff (float): cutoff frequency in radians (pi is max)
+        kernel_size (int): horizontal and vertical size, must be odd.
+        pad_to (int): pad kernel size to desired size, must be odd or zero.
+    """
+    assert kernel_size % 2 == 1, 'Kernel size must be an odd number.'
+    kernel = np.fromfunction(
+        lambda x, y: cutoff * special.j1(cutoff * np.sqrt(
+            (x - (kernel_size - 1) / 2)**2 + (y - (kernel_size - 1) / 2)**2)) / (2 * np.pi * np.sqrt(
+                (x - (kernel_size - 1) / 2)**2 + (y - (kernel_size - 1) / 2)**2)), [kernel_size, kernel_size])
+    kernel[(kernel_size - 1) // 2, (kernel_size - 1) // 2] = cutoff**2 / (4 * np.pi)
+    kernel = kernel / np.sum(kernel)
+    if pad_to > kernel_size:
+        pad_size = (pad_to - kernel_size) // 2
+        kernel = np.pad(kernel, ((pad_size, pad_size), (pad_size, pad_size)))
+    return kernel
+
+
+# ------------------------------------------------------------- #
+# --------------------------- noise --------------------------- #
+# ------------------------------------------------------------- #
+
+# ----------------------- Gaussian Noise ----------------------- #
+
+
+def generate_gaussian_noise(img, sigma=10, gray_noise=False):
+    """Generate Gaussian noise.
+
+    Args:
+        img (Numpy array): Input image, shape (h, w, c), range [0, 1], float32.
+        sigma (float): Noise scale (measured in range 255). Default: 10.
+
+    Returns:
+        (Numpy array): Returned noisy image, shape (h, w, c), range[0, 1],
+            float32.
+    """
+    if gray_noise:
+        noise = np.float32(np.random.randn(*(img.shape[0:2]))) * sigma / 255.
+        noise = np.expand_dims(noise, axis=2).repeat(3, axis=2)
+    else:
+        noise = np.float32(np.random.randn(*(img.shape))) * sigma / 255.
+    return noise
+
+
+def add_gaussian_noise(img, sigma=10, clip=True, rounds=False, gray_noise=False):
+    """Add Gaussian noise.
+
+    Args:
+        img (Numpy array): Input image, shape (h, w, c), range [0, 1], float32.
+        sigma (float): Noise scale (measured in range 255). Default: 10.
+
+    Returns:
+        (Numpy array): Returned noisy image, shape (h, w, c), range[0, 1],
+            float32.
+    """
+    noise = generate_gaussian_noise(img, sigma, gray_noise)
+    out = img + noise
+    if clip and rounds:
+        out = np.clip((out * 255.0).round(), 0, 255) / 255.
+    elif clip:
+        out = np.clip(out, 0, 1)
+    elif rounds:
+        out = (out * 255.0).round() / 255.
+    return out
+
+
+def generate_gaussian_noise_pt(img, sigma=10, gray_noise=0):
+    """Add Gaussian noise (PyTorch version).
+
+    Args:
+        img (Tensor): Shape (b, c, h, w), range[0, 1], float32.
+        scale (float | Tensor): Noise scale. Default: 1.0.
+
+    Returns:
+        (Tensor): Returned noisy image, shape (b, c, h, w), range[0, 1],
+            float32.
+    """
+    b, _, h, w = img.size()
+    if not isinstance(sigma, (float, int)):
+        sigma = sigma.view(img.size(0), 1, 1, 1)
+    if isinstance(gray_noise, (float, int)):
+        cal_gray_noise = gray_noise > 0
+    else:
+        gray_noise = gray_noise.view(b, 1, 1, 1)
+        cal_gray_noise = torch.sum(gray_noise) > 0
+
+    if cal_gray_noise:
+        noise_gray = torch.randn(*img.size()[2:4], dtype=img.dtype, device=img.device) * sigma / 255.
+        noise_gray = noise_gray.view(b, 1, h, w)
+
+    # always calculate color noise
+    noise = torch.randn(*img.size(), dtype=img.dtype, device=img.device) * sigma / 255.
+
+    if cal_gray_noise:
+        noise = noise * (1 - gray_noise) + noise_gray * gray_noise
+    return noise
+
+
+def add_gaussian_noise_pt(img, sigma=10, gray_noise=0, clip=True, rounds=False):
+    """Add Gaussian noise (PyTorch version).
+
+    Args:
+        img (Tensor): Shape (b, c, h, w), range[0, 1], float32.
+        scale (float | Tensor): Noise scale. Default: 1.0.
+
+    Returns:
+        (Tensor): Returned noisy image, shape (b, c, h, w), range[0, 1],
+            float32.
+    """
+    noise = generate_gaussian_noise_pt(img, sigma, gray_noise)
+    out = img + noise
+    if clip and rounds:
+        out = torch.clamp((out * 255.0).round(), 0, 255) / 255.
+    elif clip:
+        out = torch.clamp(out, 0, 1)
+    elif rounds:
+        out = (out * 255.0).round() / 255.
+    return out
+
+
+# ----------------------- Random Gaussian Noise ----------------------- #
+def random_generate_gaussian_noise(img, sigma_range=(0, 10), gray_prob=0):
+    sigma = np.random.uniform(sigma_range[0], sigma_range[1])
+    if np.random.uniform() < gray_prob:
+        gray_noise = True
+    else:
+        gray_noise = False
+    return generate_gaussian_noise(img, sigma, gray_noise)
+
+
+def random_add_gaussian_noise(img, sigma_range=(0, 1.0), gray_prob=0, clip=True, rounds=False):
+    noise = random_generate_gaussian_noise(img, sigma_range, gray_prob)
+    out = img + noise
+    if clip and rounds:
+        out = np.clip((out * 255.0).round(), 0, 255) / 255.
+    elif clip:
+        out = np.clip(out, 0, 1)
+    elif rounds:
+        out = (out * 255.0).round() / 255.
+    return out
+
+
+def random_generate_gaussian_noise_pt(img, sigma_range=(0, 10), gray_prob=0):
+    sigma = torch.rand(
+        img.size(0), dtype=img.dtype, device=img.device) * (sigma_range[1] - sigma_range[0]) + sigma_range[0]
+    gray_noise = torch.rand(img.size(0), dtype=img.dtype, device=img.device)
+    gray_noise = (gray_noise < gray_prob).float()
+    return generate_gaussian_noise_pt(img, sigma, gray_noise)
+
+
+def random_add_gaussian_noise_pt(img, sigma_range=(0, 1.0), gray_prob=0, clip=True, rounds=False):
+    noise = random_generate_gaussian_noise_pt(img, sigma_range, gray_prob)
+    out = img + noise
+    if clip and rounds:
+        out = torch.clamp((out * 255.0).round(), 0, 255) / 255.
+    elif clip:
+        out = torch.clamp(out, 0, 1)
+    elif rounds:
+        out = (out * 255.0).round() / 255.
+    return out
+
+
+# ----------------------- Poisson (Shot) Noise ----------------------- #
+
+
+def generate_poisson_noise(img, scale=1.0, gray_noise=False):
+    """Generate poisson noise.
+
+    Reference: https://github.com/scikit-image/scikit-image/blob/main/skimage/util/noise.py#L37-L219
+
+    Args:
+        img (Numpy array): Input image, shape (h, w, c), range [0, 1], float32.
+        scale (float): Noise scale. Default: 1.0.
+        gray_noise (bool): Whether generate gray noise. Default: False.
+
+    Returns:
+        (Numpy array): Returned noisy image, shape (h, w, c), range[0, 1],
+            float32.
+    """
+    if gray_noise:
+        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
+    # round and clip image for counting vals correctly
+    img = np.clip((img * 255.0).round(), 0, 255) / 255.
+    vals = len(np.unique(img))
+    vals = 2**np.ceil(np.log2(vals))
+    out = np.float32(np.random.poisson(img * vals) / float(vals))
+    noise = out - img
+    if gray_noise:
+        noise = np.repeat(noise[:, :, np.newaxis], 3, axis=2)
+    return noise * scale
+
+
+def add_poisson_noise(img, scale=1.0, clip=True, rounds=False, gray_noise=False):
+    """Add poisson noise.
+
+    Args:
+        img (Numpy array): Input image, shape (h, w, c), range [0, 1], float32.
+        scale (float): Noise scale. Default: 1.0.
+        gray_noise (bool): Whether generate gray noise. Default: False.
+
+    Returns:
+        (Numpy array): Returned noisy image, shape (h, w, c), range[0, 1],
+            float32.
+    """
+    noise = generate_poisson_noise(img, scale, gray_noise)
+    out = img + noise
+    if clip and rounds:
+        out = np.clip((out * 255.0).round(), 0, 255) / 255.
+    elif clip:
+        out = np.clip(out, 0, 1)
+    elif rounds:
+        out = (out * 255.0).round() / 255.
+    return out
+
+
+def generate_poisson_noise_pt(img, scale=1.0, gray_noise=0):
+    """Generate a batch of poisson noise (PyTorch version)
+
+    Args:
+        img (Tensor): Input image, shape (b, c, h, w), range [0, 1], float32.
+        scale (float | Tensor): Noise scale. Number or Tensor with shape (b).
+            Default: 1.0.
+        gray_noise (float | Tensor): 0-1 number or Tensor with shape (b).
+            0 for False, 1 for True. Default: 0.
+
+    Returns:
+        (Tensor): Returned noisy image, shape (b, c, h, w), range[0, 1],
+            float32.
+    """
+    b, _, h, w = img.size()
+    if isinstance(gray_noise, (float, int)):
+        cal_gray_noise = gray_noise > 0
+    else:
+        gray_noise = gray_noise.view(b, 1, 1, 1)
+        cal_gray_noise = torch.sum(gray_noise) > 0
+    if cal_gray_noise:
+        img_gray = rgb_to_grayscale(img, num_output_channels=1)
+        # round and clip image for counting vals correctly
+        img_gray = torch.clamp((img_gray * 255.0).round(), 0, 255) / 255.
+        # use for-loop to get the unique values for each sample
+        vals_list = [len(torch.unique(img_gray[i, :, :, :])) for i in range(b)]
+        vals_list = [2**np.ceil(np.log2(vals)) for vals in vals_list]
+        vals = img_gray.new_tensor(vals_list).view(b, 1, 1, 1)
+        out = torch.poisson(img_gray * vals) / vals
+        noise_gray = out - img_gray
+        noise_gray = noise_gray.expand(b, 3, h, w)
+
+    # always calculate color noise
+    # round and clip image for counting vals correctly
+    img = torch.clamp((img * 255.0).round(), 0, 255) / 255.
+    # use for-loop to get the unique values for each sample
+    vals_list = [len(torch.unique(img[i, :, :, :])) for i in range(b)]
+    vals_list = [2**np.ceil(np.log2(vals)) for vals in vals_list]
+    vals = img.new_tensor(vals_list).view(b, 1, 1, 1)
+    out = torch.poisson(img * vals) / vals
+    noise = out - img
+    if cal_gray_noise:
+        noise = noise * (1 - gray_noise) + noise_gray * gray_noise
+    if not isinstance(scale, (float, int)):
+        scale = scale.view(b, 1, 1, 1)
+    return noise * scale
+
+
+def add_poisson_noise_pt(img, scale=1.0, clip=True, rounds=False, gray_noise=0):
+    """Add poisson noise to a batch of images (PyTorch version).
+
+    Args:
+        img (Tensor): Input image, shape (b, c, h, w), range [0, 1], float32.
+        scale (float | Tensor): Noise scale. Number or Tensor with shape (b).
+            Default: 1.0.
+        gray_noise (float | Tensor): 0-1 number or Tensor with shape (b).
+            0 for False, 1 for True. Default: 0.
+
+    Returns:
+        (Tensor): Returned noisy image, shape (b, c, h, w), range[0, 1],
+            float32.
+    """
+    noise = generate_poisson_noise_pt(img, scale, gray_noise)
+    out = img + noise
+    if clip and rounds:
+        out = torch.clamp((out * 255.0).round(), 0, 255) / 255.
+    elif clip:
+        out = torch.clamp(out, 0, 1)
+    elif rounds:
+        out = (out * 255.0).round() / 255.
+    return out
+
+
+# ----------------------- Random Poisson (Shot) Noise ----------------------- #
+
+
+def random_generate_poisson_noise(img, scale_range=(0, 1.0), gray_prob=0):
+    scale = np.random.uniform(scale_range[0], scale_range[1])
+    if np.random.uniform() < gray_prob:
+        gray_noise = True
+    else:
+        gray_noise = False
+    return generate_poisson_noise(img, scale, gray_noise)
+
+
+def random_add_poisson_noise(img, scale_range=(0, 1.0), gray_prob=0, clip=True, rounds=False):
+    noise = random_generate_poisson_noise(img, scale_range, gray_prob)
+    out = img + noise
+    if clip and rounds:
+        out = np.clip((out * 255.0).round(), 0, 255) / 255.
+    elif clip:
+        out = np.clip(out, 0, 1)
+    elif rounds:
+        out = (out * 255.0).round() / 255.
+    return out
+
+
+def random_generate_poisson_noise_pt(img, scale_range=(0, 1.0), gray_prob=0):
+    scale = torch.rand(
+        img.size(0), dtype=img.dtype, device=img.device) * (scale_range[1] - scale_range[0]) + scale_range[0]
+    gray_noise = torch.rand(img.size(0), dtype=img.dtype, device=img.device)
+    gray_noise = (gray_noise < gray_prob).float()
+    return generate_poisson_noise_pt(img, scale, gray_noise)
+
+
+def random_add_poisson_noise_pt(img, scale_range=(0, 1.0), gray_prob=0, clip=True, rounds=False):
+    noise = random_generate_poisson_noise_pt(img, scale_range, gray_prob)
+    out = img + noise
+    if clip and rounds:
+        out = torch.clamp((out * 255.0).round(), 0, 255) / 255.
+    elif clip:
+        out = torch.clamp(out, 0, 1)
+    elif rounds:
+        out = (out * 255.0).round() / 255.
+    return out
+
+
+# ------------------------------------------------------------------------ #
+# --------------------------- JPEG compression --------------------------- #
+# ------------------------------------------------------------------------ #
+
+
+def add_jpg_compression(img, quality=90):
+    """Add JPG compression artifacts.
+
+    Args:
+        img (Numpy array): Input image, shape (h, w, c), range [0, 1], float32.
+        quality (float): JPG compression quality. 0 for lowest quality, 100 for
+            best quality. Default: 90.
+
+    Returns:
+        (Numpy array): Returned image after JPG, shape (h, w, c), range[0, 1],
+            float32.
+    """
+    img = np.clip(img, 0, 1)
+    encode_param = [int(cv2.IMWRITE_JPEG_QUALITY), quality]
+    _, encimg = cv2.imencode('.jpg', img * 255., encode_param)
+    img = np.float32(cv2.imdecode(encimg, 1)) / 255.
+    return img
+
+
+def random_add_jpg_compression(img, quality_range=(90, 100)):
+    """Randomly add JPG compression artifacts.
+
+    Args:
+        img (Numpy array): Input image, shape (h, w, c), range [0, 1], float32.
+        quality_range (tuple[float] | list[float]): JPG compression quality
+            range. 0 for lowest quality, 100 for best quality.
+            Default: (90, 100).
+
+    Returns:
+        (Numpy array): Returned image after JPG, shape (h, w, c), range[0, 1],
+            float32.
+    """
+    quality = np.random.uniform(quality_range[0], quality_range[1])
+    return add_jpg_compression(img, quality)

basicsr/data/ffhq_dataset.py

@@ -0,0 +1,80 @@
+import random
+import time
+from os import path as osp
+from torch.utils import data as data
+from torchvision.transforms.functional import normalize
+
+from basicsr.data.transforms import augment
+from basicsr.utils import FileClient, get_root_logger, imfrombytes, img2tensor
+from basicsr.utils.registry import DATASET_REGISTRY
+
+
+@DATASET_REGISTRY.register()
+class FFHQDataset(data.Dataset):
+    """FFHQ dataset for StyleGAN.
+
+    Args:
+        opt (dict): Config for train datasets. It contains the following keys:
+            dataroot_gt (str): Data root path for gt.
+            io_backend (dict): IO backend type and other kwarg.
+            mean (list | tuple): Image mean.
+            std (list | tuple): Image std.
+            use_hflip (bool): Whether to horizontally flip.
+
+    """
+
+    def __init__(self, opt):
+        super(FFHQDataset, self).__init__()
+        self.opt = opt
+        # file client (io backend)
+        self.file_client = None
+        self.io_backend_opt = opt['io_backend']
+
+        self.gt_folder = opt['dataroot_gt']
+        self.mean = opt['mean']
+        self.std = opt['std']
+
+        if self.io_backend_opt['type'] == 'lmdb':
+            self.io_backend_opt['db_paths'] = self.gt_folder
+            if not self.gt_folder.endswith('.lmdb'):
+                raise ValueError("'dataroot_gt' should end with '.lmdb', but received {self.gt_folder}")
+            with open(osp.join(self.gt_folder, 'meta_info.txt')) as fin:
+                self.paths = [line.split('.')[0] for line in fin]
+        else:
+            # FFHQ has 70000 images in total
+            self.paths = [osp.join(self.gt_folder, f'{v:08d}.png') for v in range(70000)]
+
+    def __getitem__(self, index):
+        if self.file_client is None:
+            self.file_client = FileClient(self.io_backend_opt.pop('type'), **self.io_backend_opt)
+
+        # load gt image
+        gt_path = self.paths[index]
+        # avoid errors caused by high latency in reading files
+        retry = 3
+        while retry > 0:
+            try:
+                img_bytes = self.file_client.get(gt_path)
+            except Exception as e:
+                logger = get_root_logger()
+                logger.warning(f'File client error: {e}, remaining retry times: {retry - 1}')
+                # change another file to read
+                index = random.randint(0, self.__len__())
+                gt_path = self.paths[index]
+                time.sleep(1)  # sleep 1s for occasional server congestion
+            else:
+                break
+            finally:
+                retry -= 1
+        img_gt = imfrombytes(img_bytes, float32=True)
+
+        # random horizontal flip
+        img_gt = augment(img_gt, hflip=self.opt['use_hflip'], rotation=False)
+        # BGR to RGB, HWC to CHW, numpy to tensor
+        img_gt = img2tensor(img_gt, bgr2rgb=True, float32=True)
+        # normalize
+        normalize(img_gt, self.mean, self.std, inplace=True)
+        return {'gt': img_gt, 'gt_path': gt_path}
+
+    def __len__(self):
+        return len(self.paths)

basicsr/data/meta_info/meta_info_REDS4_test_GT.txt

@@ -0,0 +1,4 @@
+000 100 (720,1280,3)
+011 100 (720,1280,3)
+015 100 (720,1280,3)
+020 100 (720,1280,3)

basicsr/data/meta_info/meta_info_REDS_GT.txt

@@ -0,0 +1,270 @@
+000 100 (720,1280,3)
+001 100 (720,1280,3)
+002 100 (720,1280,3)
+003 100 (720,1280,3)
+004 100 (720,1280,3)
+005 100 (720,1280,3)
+006 100 (720,1280,3)
+007 100 (720,1280,3)
+008 100 (720,1280,3)
+009 100 (720,1280,3)
+010 100 (720,1280,3)
+011 100 (720,1280,3)
+012 100 (720,1280,3)
+013 100 (720,1280,3)
+014 100 (720,1280,3)
+015 100 (720,1280,3)
+016 100 (720,1280,3)
+017 100 (720,1280,3)
+018 100 (720,1280,3)
+019 100 (720,1280,3)
+020 100 (720,1280,3)
+021 100 (720,1280,3)
+022 100 (720,1280,3)
+023 100 (720,1280,3)
+024 100 (720,1280,3)
+025 100 (720,1280,3)
+026 100 (720,1280,3)
+027 100 (720,1280,3)
+028 100 (720,1280,3)
+029 100 (720,1280,3)
+030 100 (720,1280,3)
+031 100 (720,1280,3)
+032 100 (720,1280,3)
+033 100 (720,1280,3)
+034 100 (720,1280,3)
+035 100 (720,1280,3)
+036 100 (720,1280,3)
+037 100 (720,1280,3)
+038 100 (720,1280,3)
+039 100 (720,1280,3)
+040 100 (720,1280,3)
+041 100 (720,1280,3)
+042 100 (720,1280,3)
+043 100 (720,1280,3)
+044 100 (720,1280,3)
+045 100 (720,1280,3)
+046 100 (720,1280,3)
+047 100 (720,1280,3)
+048 100 (720,1280,3)
+049 100 (720,1280,3)
+050 100 (720,1280,3)
+051 100 (720,1280,3)
+052 100 (720,1280,3)
+053 100 (720,1280,3)
+054 100 (720,1280,3)
+055 100 (720,1280,3)
+056 100 (720,1280,3)
+057 100 (720,1280,3)
+058 100 (720,1280,3)
+059 100 (720,1280,3)
+060 100 (720,1280,3)
+061 100 (720,1280,3)
+062 100 (720,1280,3)
+063 100 (720,1280,3)
+064 100 (720,1280,3)
+065 100 (720,1280,3)
+066 100 (720,1280,3)
+067 100 (720,1280,3)
+068 100 (720,1280,3)
+069 100 (720,1280,3)
+070 100 (720,1280,3)
+071 100 (720,1280,3)
+072 100 (720,1280,3)
+073 100 (720,1280,3)
+074 100 (720,1280,3)
+075 100 (720,1280,3)
+076 100 (720,1280,3)
+077 100 (720,1280,3)
+078 100 (720,1280,3)
+079 100 (720,1280,3)
+080 100 (720,1280,3)
+081 100 (720,1280,3)
+082 100 (720,1280,3)
+083 100 (720,1280,3)
+084 100 (720,1280,3)
+085 100 (720,1280,3)
+086 100 (720,1280,3)
+087 100 (720,1280,3)
+088 100 (720,1280,3)
+089 100 (720,1280,3)
+090 100 (720,1280,3)
+091 100 (720,1280,3)
+092 100 (720,1280,3)
+093 100 (720,1280,3)
+094 100 (720,1280,3)
+095 100 (720,1280,3)
+096 100 (720,1280,3)
+097 100 (720,1280,3)
+098 100 (720,1280,3)
+099 100 (720,1280,3)
+100 100 (720,1280,3)
+101 100 (720,1280,3)
+102 100 (720,1280,3)
+103 100 (720,1280,3)
+104 100 (720,1280,3)
+105 100 (720,1280,3)
+106 100 (720,1280,3)
+107 100 (720,1280,3)
+108 100 (720,1280,3)
+109 100 (720,1280,3)
+110 100 (720,1280,3)
+111 100 (720,1280,3)
+112 100 (720,1280,3)
+113 100 (720,1280,3)
+114 100 (720,1280,3)
+115 100 (720,1280,3)
+116 100 (720,1280,3)
+117 100 (720,1280,3)
+118 100 (720,1280,3)
+119 100 (720,1280,3)
+120 100 (720,1280,3)
+121 100 (720,1280,3)
+122 100 (720,1280,3)
+123 100 (720,1280,3)
+124 100 (720,1280,3)
+125 100 (720,1280,3)
+126 100 (720,1280,3)
+127 100 (720,1280,3)
+128 100 (720,1280,3)
+129 100 (720,1280,3)
+130 100 (720,1280,3)
+131 100 (720,1280,3)
+132 100 (720,1280,3)
+133 100 (720,1280,3)
+134 100 (720,1280,3)
+135 100 (720,1280,3)
+136 100 (720,1280,3)
+137 100 (720,1280,3)
+138 100 (720,1280,3)
+139 100 (720,1280,3)
+140 100 (720,1280,3)
+141 100 (720,1280,3)
+142 100 (720,1280,3)
+143 100 (720,1280,3)
+144 100 (720,1280,3)
+145 100 (720,1280,3)
+146 100 (720,1280,3)
+147 100 (720,1280,3)
+148 100 (720,1280,3)
+149 100 (720,1280,3)
+150 100 (720,1280,3)
+151 100 (720,1280,3)
+152 100 (720,1280,3)
+153 100 (720,1280,3)
+154 100 (720,1280,3)
+155 100 (720,1280,3)
+156 100 (720,1280,3)
+157 100 (720,1280,3)
+158 100 (720,1280,3)
+159 100 (720,1280,3)
+160 100 (720,1280,3)
+161 100 (720,1280,3)
+162 100 (720,1280,3)
+163 100 (720,1280,3)
+164 100 (720,1280,3)
+165 100 (720,1280,3)
+166 100 (720,1280,3)
+167 100 (720,1280,3)
+168 100 (720,1280,3)
+169 100 (720,1280,3)
+170 100 (720,1280,3)
+171 100 (720,1280,3)
+172 100 (720,1280,3)
+173 100 (720,1280,3)
+174 100 (720,1280,3)
+175 100 (720,1280,3)
+176 100 (720,1280,3)
+177 100 (720,1280,3)
+178 100 (720,1280,3)
+179 100 (720,1280,3)
+180 100 (720,1280,3)
+181 100 (720,1280,3)
+182 100 (720,1280,3)
+183 100 (720,1280,3)
+184 100 (720,1280,3)
+185 100 (720,1280,3)
+186 100 (720,1280,3)
+187 100 (720,1280,3)
+188 100 (720,1280,3)
+189 100 (720,1280,3)
+190 100 (720,1280,3)
+191 100 (720,1280,3)
+192 100 (720,1280,3)
+193 100 (720,1280,3)
+194 100 (720,1280,3)
+195 100 (720,1280,3)
+196 100 (720,1280,3)
+197 100 (720,1280,3)
+198 100 (720,1280,3)
+199 100 (720,1280,3)
+200 100 (720,1280,3)
+201 100 (720,1280,3)
+202 100 (720,1280,3)
+203 100 (720,1280,3)
+204 100 (720,1280,3)
+205 100 (720,1280,3)
+206 100 (720,1280,3)
+207 100 (720,1280,3)
+208 100 (720,1280,3)
+209 100 (720,1280,3)
+210 100 (720,1280,3)
+211 100 (720,1280,3)
+212 100 (720,1280,3)
+213 100 (720,1280,3)
+214 100 (720,1280,3)
+215 100 (720,1280,3)
+216 100 (720,1280,3)
+217 100 (720,1280,3)
+218 100 (720,1280,3)
+219 100 (720,1280,3)
+220 100 (720,1280,3)
+221 100 (720,1280,3)
+222 100 (720,1280,3)
+223 100 (720,1280,3)
+224 100 (720,1280,3)
+225 100 (720,1280,3)
+226 100 (720,1280,3)
+227 100 (720,1280,3)
+228 100 (720,1280,3)
+229 100 (720,1280,3)
+230 100 (720,1280,3)
+231 100 (720,1280,3)
+232 100 (720,1280,3)
+233 100 (720,1280,3)
+234 100 (720,1280,3)
+235 100 (720,1280,3)
+236 100 (720,1280,3)
+237 100 (720,1280,3)
+238 100 (720,1280,3)
+239 100 (720,1280,3)
+240 100 (720,1280,3)
+241 100 (720,1280,3)
+242 100 (720,1280,3)
+243 100 (720,1280,3)
+244 100 (720,1280,3)
+245 100 (720,1280,3)
+246 100 (720,1280,3)
+247 100 (720,1280,3)
+248 100 (720,1280,3)
+249 100 (720,1280,3)
+250 100 (720,1280,3)
+251 100 (720,1280,3)
+252 100 (720,1280,3)
+253 100 (720,1280,3)
+254 100 (720,1280,3)
+255 100 (720,1280,3)
+256 100 (720,1280,3)
+257 100 (720,1280,3)
+258 100 (720,1280,3)
+259 100 (720,1280,3)
+260 100 (720,1280,3)
+261 100 (720,1280,3)
+262 100 (720,1280,3)
+263 100 (720,1280,3)
+264 100 (720,1280,3)
+265 100 (720,1280,3)
+266 100 (720,1280,3)
+267 100 (720,1280,3)
+268 100 (720,1280,3)
+269 100 (720,1280,3)

basicsr/data/meta_info/meta_info_REDSofficial4_test_GT.txt

@@ -0,0 +1,4 @@
+240 100 (720,1280,3)
+241 100 (720,1280,3)
+246 100 (720,1280,3)
+257 100 (720,1280,3)

basicsr/data/meta_info/meta_info_REDSval_official_test_GT.txt

@@ -0,0 +1,30 @@
+240 100 (720,1280,3)
+241 100 (720,1280,3)
+242 100 (720,1280,3)
+243 100 (720,1280,3)
+244 100 (720,1280,3)
+245 100 (720,1280,3)
+246 100 (720,1280,3)
+247 100 (720,1280,3)
+248 100 (720,1280,3)
+249 100 (720,1280,3)
+250 100 (720,1280,3)
+251 100 (720,1280,3)
+252 100 (720,1280,3)
+253 100 (720,1280,3)
+254 100 (720,1280,3)
+255 100 (720,1280,3)
+256 100 (720,1280,3)
+257 100 (720,1280,3)
+258 100 (720,1280,3)
+259 100 (720,1280,3)
+260 100 (720,1280,3)
+261 100 (720,1280,3)
+262 100 (720,1280,3)
+263 100 (720,1280,3)
+264 100 (720,1280,3)
+265 100 (720,1280,3)
+266 100 (720,1280,3)
+267 100 (720,1280,3)
+268 100 (720,1280,3)
+269 100 (720,1280,3)

basicsr/data/meta_info/meta_info_Vimeo90K_test_fast_GT.txt

@@ -0,0 +1,1225 @@
+00001/0625 7 (256,448,3)
+00001/0632 7 (256,448,3)
+00001/0807 7 (256,448,3)
+00001/0832 7 (256,448,3)
+00001/0834 7 (256,448,3)
+00001/0836 7 (256,448,3)
+00002/0004 7 (256,448,3)
+00002/0112 7 (256,448,3)
+00002/0116 7 (256,448,3)
+00002/0123 7 (256,448,3)
+00002/0455 7 (256,448,3)
+00002/0602 7 (256,448,3)
+00002/0976 7 (256,448,3)
+00002/0980 7 (256,448,3)
+00002/0983 7 (256,448,3)
+00002/1000 7 (256,448,3)
+00003/0022 7 (256,448,3)
+00003/0031 7 (256,448,3)
+00003/0035 7 (256,448,3)
+00003/0041 7 (256,448,3)
+00003/0073 7 (256,448,3)
+00003/0107 7 (256,448,3)
+00003/0111 7 (256,448,3)
+00003/0114 7 (256,448,3)
+00003/0117 7 (256,448,3)
+00003/0121 7 (256,448,3)
+00003/0499 7 (256,448,3)
+00003/0501 7 (256,448,3)
+00003/0507 7 (256,448,3)
+00003/0510 7 (256,448,3)
+00003/0517 7 (256,448,3)
+00003/0522 7 (256,448,3)
+00003/0531 7 (256,448,3)
+00003/0533 7 (256,448,3)
+00003/0534 7 (256,448,3)
+00003/0682 7 (256,448,3)
+00003/0687 7 (256,448,3)
+00003/0715 7 (256,448,3)
+00003/0742 7 (256,448,3)
+00003/0751 7 (256,448,3)
+00003/0984 7 (256,448,3)
+00004/0042 7 (256,448,3)
+00004/0165 7 (256,448,3)
+00004/0321 7 (256,448,3)
+00004/0569 7 (256,448,3)
+00004/0572 7 (256,448,3)
+00004/0619 7 (256,448,3)
+00004/0776 7 (256,448,3)
+00004/0780 7 (256,448,3)
+00004/0825 7 (256,448,3)
+00004/0832 7 (256,448,3)
+00004/0853 7 (256,448,3)
+00004/0876 7 (256,448,3)
+00004/0888 7 (256,448,3)
+00005/0015 7 (256,448,3)
+00005/0021 7 (256,448,3)
+00005/0022 7 (256,448,3)
+00005/0024 7 (256,448,3)
+00005/0026 7 (256,448,3)
+00005/0394 7 (256,448,3)
+00005/0403 7 (256,448,3)
+00005/0531 7 (256,448,3)
+00005/0546 7 (256,448,3)
+00005/0554 7 (256,448,3)
+00005/0694 7 (256,448,3)
+00005/0700 7 (256,448,3)
+00005/0740 7 (256,448,3)
+00005/0826 7 (256,448,3)
+00005/0832 7 (256,448,3)
+00005/0834 7 (256,448,3)
+00005/0943 7 (256,448,3)
+00006/0184 7 (256,448,3)
+00006/0205 7 (256,448,3)
+00006/0206 7 (256,448,3)
+00006/0211 7 (256,448,3)
+00006/0271 7 (256,448,3)
+00006/0273 7 (256,448,3)
+00006/0277 7 (256,448,3)
+00006/0283 7 (256,448,3)
+00006/0287 7 (256,448,3)
+00006/0298 7 (256,448,3)
+00006/0310 7 (256,448,3)
+00006/0356 7 (256,448,3)
+00006/0357 7 (256,448,3)
+00006/0544 7 (256,448,3)
+00006/0565 7 (256,448,3)
+00006/0569 7 (256,448,3)
+00006/0573 7 (256,448,3)
+00006/0592 7 (256,448,3)
+00006/0613 7 (256,448,3)
+00006/0633 7 (256,448,3)
+00006/0637 7 (256,448,3)
+00006/0646 7 (256,448,3)
+00006/0649 7 (256,448,3)
+00006/0655 7 (256,448,3)
+00006/0658 7 (256,448,3)
+00006/0662 7 (256,448,3)
+00006/0666 7 (256,448,3)
+00006/0673 7 (256,448,3)
+00007/0248 7 (256,448,3)
+00007/0253 7 (256,448,3)
+00007/0430 7 (256,448,3)
+00007/0434 7 (256,448,3)
+00007/0436 7 (256,448,3)
+00007/0452 7 (256,448,3)
+00007/0464 7 (256,448,3)
+00007/0470 7 (256,448,3)
+00007/0472 7 (256,448,3)
+00007/0483 7 (256,448,3)
+00007/0484 7 (256,448,3)
+00007/0493 7 (256,448,3)
+00007/0508 7 (256,448,3)
+00007/0514 7 (256,448,3)
+00007/0697 7 (256,448,3)
+00007/0698 7 (256,448,3)
+00007/0744 7 (256,448,3)
+00007/0775 7 (256,448,3)
+00007/0786 7 (256,448,3)
+00007/0790 7 (256,448,3)
+00007/0800 7 (256,448,3)
+00007/0833 7 (256,448,3)
+00007/0867 7 (256,448,3)
+00007/0879 7 (256,448,3)
+00007/0899 7 (256,448,3)
+00008/0251 7 (256,448,3)
+00008/0322 7 (256,448,3)
+00008/0971 7 (256,448,3)
+00008/0976 7 (256,448,3)
+00009/0016 7 (256,448,3)
+00009/0036 7 (256,448,3)
+00009/0037 7 (256,448,3)
+00009/0609 7 (256,448,3)
+00009/0812 7 (256,448,3)
+00009/0821 7 (256,448,3)
+00009/0947 7 (256,448,3)
+00009/0952 7 (256,448,3)
+00009/0955 7 (256,448,3)
+00009/0970 7 (256,448,3)
+00010/0072 7 (256,448,3)
+00010/0074 7 (256,448,3)
+00010/0079 7 (256,448,3)
+00010/0085 7 (256,448,3)
+00010/0139 7 (256,448,3)
+00010/0140 7 (256,448,3)
+00010/0183 7 (256,448,3)
+00010/0200 7 (256,448,3)
+00010/0223 7 (256,448,3)
+00010/0305 7 (256,448,3)
+00010/0323 7 (256,448,3)
+00010/0338 7 (256,448,3)
+00010/0342 7 (256,448,3)
+00010/0350 7 (256,448,3)
+00010/0356 7 (256,448,3)
+00010/0362 7 (256,448,3)
+00010/0366 7 (256,448,3)
+00010/0375 7 (256,448,3)
+00010/0404 7 (256,448,3)
+00010/0407 7 (256,448,3)
+00010/0414 7 (256,448,3)
+00010/0418 7 (256,448,3)
+00010/0429 7 (256,448,3)
+00010/0557 7 (256,448,3)
+00010/0564 7 (256,448,3)
+00010/0733 7 (256,448,3)
+00010/0935 7 (256,448,3)
+00010/0939 7 (256,448,3)
+00010/0943 7 (256,448,3)
+00011/0242 7 (256,448,3)
+00011/0259 7 (256,448,3)
+00011/0263 7 (256,448,3)
+00011/0266 7 (256,448,3)
+00011/0278 7 (256,448,3)
+00011/0890 7 (256,448,3)
+00011/0894 7 (256,448,3)
+00011/0903 7 (256,448,3)
+00011/0906 7 (256,448,3)
+00011/0913 7 (256,448,3)
+00012/0011 7 (256,448,3)
+00012/0014 7 (256,448,3)
+00012/0126 7 (256,448,3)
+00012/0127 7 (256,448,3)
+00012/0526 7 (256,448,3)
+00012/0551 7 (256,448,3)
+00012/0896 7 (256,448,3)
+00012/0910 7 (256,448,3)
+00012/0915 7 (256,448,3)
+00013/0167 7 (256,448,3)
+00013/0794 7 (256,448,3)
+00013/0807 7 (256,448,3)
+00013/0846 7 (256,448,3)
+00013/0882 7 (256,448,3)
+00013/0889 7 (256,448,3)
+00013/0910 7 (256,448,3)
+00013/0913 7 (256,448,3)
+00013/0924 7 (256,448,3)
+00013/0931 7 (256,448,3)
+00013/0944 7 (256,448,3)
+00013/0955 7 (256,448,3)
+00013/0962 7 (256,448,3)
+00013/0969 7 (256,448,3)
+00014/0012 7 (256,448,3)
+00014/0025 7 (256,448,3)
+00014/0473 7 (256,448,3)
+00014/0499 7 (256,448,3)
+00014/0524 7 (256,448,3)
+00014/0739 7 (256,448,3)
+00014/0753 7 (256,448,3)
+00014/0771 7 (256,448,3)
+00014/0832 7 (256,448,3)
+00014/0836 7 (256,448,3)
+00014/0838 7 (256,448,3)
+00014/0839 7 (256,448,3)
+00014/0843 7 (256,448,3)
+00014/0846 7 (256,448,3)
+00014/0849 7 (256,448,3)
+00014/0859 7 (256,448,3)
+00014/0880 7 (256,448,3)
+00014/0906 7 (256,448,3)
+00015/0030 7 (256,448,3)
+00015/0067 7 (256,448,3)
+00015/0084 7 (256,448,3)
+00015/0190 7 (256,448,3)
+00015/0575 7 (256,448,3)
+00015/0784 7 (256,448,3)
+00015/0855 7 (256,448,3)
+00015/0904 7 (256,448,3)
+00015/0914 7 (256,448,3)
+00015/0936 7 (256,448,3)
+00015/0939 7 (256,448,3)
+00015/0943 7 (256,448,3)
+00015/0957 7 (256,448,3)
+00016/0131 7 (256,448,3)
+00016/0173 7 (256,448,3)
+00016/0320 7 (256,448,3)
+00016/0328 7 (256,448,3)
+00016/0334 7 (256,448,3)
+00016/0338 7 (256,448,3)
+00016/0339 7 (256,448,3)
+00016/0345 7 (256,448,3)
+00016/0365 7 (256,448,3)
+00016/0584 7 (256,448,3)
+00016/0634 7 (256,448,3)
+00017/0342 7 (256,448,3)
+00017/0346 7 (256,448,3)
+00017/0350 7 (256,448,3)
+00017/0766 7 (256,448,3)
+00017/0786 7 (256,448,3)
+00017/0911 7 (256,448,3)
+00017/0914 7 (256,448,3)
+00018/0217 7 (256,448,3)
+00018/0258 7 (256,448,3)
+00018/0307 7 (256,448,3)
+00018/0480 7 (256,448,3)
+00018/0491 7 (256,448,3)
+00018/0994 7 (256,448,3)
+00018/0995 7 (256,448,3)
+00018/0997 7 (256,448,3)
+00018/1000 7 (256,448,3)
+00019/0007 7 (256,448,3)
+00019/0016 7 (256,448,3)
+00019/0026 7 (256,448,3)
+00019/0030 7 (256,448,3)
+00019/0086 7 (256,448,3)
+00019/0089 7 (256,448,3)
+00019/0111 7 (256,448,3)
+00019/0285 7 (256,448,3)
+00019/0415 7 (256,448,3)
+00019/0434 7 (256,448,3)
+00019/0437 7 (256,448,3)
+00019/0568 7 (256,448,3)
+00019/0570 7 (256,448,3)
+00019/0591 7 (256,448,3)
+00019/0596 7 (256,448,3)
+00019/0603 7 (256,448,3)
+00019/0607 7 (256,448,3)
+00019/0637 7 (256,448,3)
+00019/0644 7 (256,448,3)
+00019/0647 7 (256,448,3)
+00019/0787 7 (256,448,3)
+00019/0993 7 (256,448,3)
+00019/0998 7 (256,448,3)
+00021/0232 7 (256,448,3)
+00021/0255 7 (256,448,3)
+00021/0646 7 (256,448,3)
+00021/0653 7 (256,448,3)
+00021/0657 7 (256,448,3)
+00021/0668 7 (256,448,3)
+00021/0672 7 (256,448,3)
+00021/0725 7 (256,448,3)
+00021/0750 7 (256,448,3)
+00021/0764 7 (256,448,3)
+00021/0821 7 (256,448,3)
+00022/0192 7 (256,448,3)
+00022/0391 7 (256,448,3)
+00022/0514 7 (256,448,3)
+00022/0567 7 (256,448,3)
+00022/0674 7 (256,448,3)
+00022/0686 7 (256,448,3)
+00022/0700 7 (256,448,3)
+00023/0020 7 (256,448,3)
+00023/0024 7 (256,448,3)
+00023/0025 7 (256,448,3)
+00023/0042 7 (256,448,3)
+00023/0050 7 (256,448,3)
+00023/0094 7 (256,448,3)
+00023/0107 7 (256,448,3)
+00023/0635 7 (256,448,3)
+00023/0698 7 (256,448,3)
+00023/0774 7 (256,448,3)
+00023/0795 7 (256,448,3)
+00023/0821 7 (256,448,3)
+00023/0839 7 (256,448,3)
+00023/0846 7 (256,448,3)
+00023/0869 7 (256,448,3)
+00023/0879 7 (256,448,3)
+00023/0887 7 (256,448,3)
+00023/0899 7 (256,448,3)
+00023/0910 7 (256,448,3)
+00023/0920 7 (256,448,3)
+00023/0929 7 (256,448,3)
+00023/0941 7 (256,448,3)
+00023/0942 7 (256,448,3)
+00023/0952 7 (256,448,3)
+00024/0066 7 (256,448,3)
+00024/0072 7 (256,448,3)
+00024/0080 7 (256,448,3)
+00024/0093 7 (256,448,3)
+00024/0107 7 (256,448,3)
+00024/0262 7 (256,448,3)
+00024/0283 7 (256,448,3)
+00024/0294 7 (256,448,3)
+00024/0296 7 (256,448,3)
+00024/0304 7 (256,448,3)
+00024/0315 7 (256,448,3)
+00024/0322 7 (256,448,3)
+00024/0648 7 (256,448,3)
+00024/0738 7 (256,448,3)
+00024/0743 7 (256,448,3)
+00025/0542 7 (256,448,3)
+00025/0769 7 (256,448,3)
+00025/0984 7 (256,448,3)
+00025/0985 7 (256,448,3)
+00025/0989 7 (256,448,3)
+00025/0991 7 (256,448,3)
+00026/0009 7 (256,448,3)
+00026/0013 7 (256,448,3)
+00026/0020 7 (256,448,3)
+00026/0021 7 (256,448,3)
+00026/0025 7 (256,448,3)
+00026/0135 7 (256,448,3)
+00026/0200 7 (256,448,3)
+00026/0297 7 (256,448,3)
+00026/0306 7 (256,448,3)
+00026/0444 7 (256,448,3)
+00026/0450 7 (256,448,3)
+00026/0453 7 (256,448,3)
+00026/0464 7 (256,448,3)
+00026/0486 7 (256,448,3)
+00026/0773 7 (256,448,3)
+00026/0785 7 (256,448,3)
+00026/0836 7 (256,448,3)
+00026/0838 7 (256,448,3)
+00026/0848 7 (256,448,3)
+00026/0885 7 (256,448,3)
+00026/0893 7 (256,448,3)
+00026/0939 7 (256,448,3)
+00026/0942 7 (256,448,3)
+00027/0092 7 (256,448,3)
+00027/0112 7 (256,448,3)
+00027/0115 7 (256,448,3)
+00027/0143 7 (256,448,3)
+00027/0175 7 (256,448,3)
+00027/0179 7 (256,448,3)
+00027/0183 7 (256,448,3)
+00027/0197 7 (256,448,3)
+00027/0199 7 (256,448,3)
+00027/0300 7 (256,448,3)
+00028/0015 7 (256,448,3)
+00028/0032 7 (256,448,3)
+00028/0048 7 (256,448,3)
+00028/0068 7 (256,448,3)
+00028/0219 7 (256,448,3)
+00028/0606 7 (256,448,3)
+00028/0626 7 (256,448,3)
+00028/0748 7 (256,448,3)
+00028/0764 7 (256,448,3)
+00028/0772 7 (256,448,3)
+00028/0780 7 (256,448,3)
+00028/0926 7 (256,448,3)
+00028/0947 7 (256,448,3)
+00028/0962 7 (256,448,3)
+00029/0085 7 (256,448,3)
+00029/0281 7 (256,448,3)
+00029/0284 7 (256,448,3)
+00029/0288 7 (256,448,3)
+00029/0294 7 (256,448,3)
+00029/0364 7 (256,448,3)
+00029/0369 7 (256,448,3)
+00029/0421 7 (256,448,3)
+00029/0425 7 (256,448,3)
+00029/0550 7 (256,448,3)
+00030/0014 7 (256,448,3)
+00030/0101 7 (256,448,3)
+00030/0143 7 (256,448,3)
+00030/0351 7 (256,448,3)
+00030/0356 7 (256,448,3)
+00030/0371 7 (256,448,3)
+00030/0484 7 (256,448,3)
+00030/0492 7 (256,448,3)
+00030/0503 7 (256,448,3)
+00030/0682 7 (256,448,3)
+00030/0696 7 (256,448,3)
+00030/0735 7 (256,448,3)
+00030/0737 7 (256,448,3)
+00030/0868 7 (256,448,3)
+00031/0161 7 (256,448,3)
+00031/0180 7 (256,448,3)
+00031/0194 7 (256,448,3)
+00031/0253 7 (256,448,3)
+00031/0293 7 (256,448,3)
+00031/0466 7 (256,448,3)
+00031/0477 7 (256,448,3)
+00031/0549 7 (256,448,3)
+00031/0600 7 (256,448,3)
+00031/0617 7 (256,448,3)
+00031/0649 7 (256,448,3)
+00032/0015 7 (256,448,3)
+00032/0020 7 (256,448,3)
+00032/0023 7 (256,448,3)
+00032/0048 7 (256,448,3)
+00032/0056 7 (256,448,3)
+00032/0872 7 (256,448,3)
+00033/0069 7 (256,448,3)
+00033/0073 7 (256,448,3)
+00033/0078 7 (256,448,3)
+00033/0079 7 (256,448,3)
+00033/0086 7 (256,448,3)
+00033/0088 7 (256,448,3)
+00033/0091 7 (256,448,3)
+00033/0096 7 (256,448,3)
+00033/0607 7 (256,448,3)
+00033/0613 7 (256,448,3)
+00033/0616 7 (256,448,3)
+00033/0619 7 (256,448,3)
+00033/0626 7 (256,448,3)
+00033/0628 7 (256,448,3)
+00033/0637 7 (256,448,3)
+00033/0686 7 (256,448,3)
+00033/0842 7 (256,448,3)
+00034/0261 7 (256,448,3)
+00034/0265 7 (256,448,3)
+00034/0269 7 (256,448,3)
+00034/0275 7 (256,448,3)
+00034/0286 7 (256,448,3)
+00034/0294 7 (256,448,3)
+00034/0431 7 (256,448,3)
+00034/0577 7 (256,448,3)
+00034/0685 7 (256,448,3)
+00034/0687 7 (256,448,3)
+00034/0703 7 (256,448,3)
+00034/0715 7 (256,448,3)
+00034/0935 7 (256,448,3)
+00034/0943 7 (256,448,3)
+00034/0963 7 (256,448,3)
+00034/0979 7 (256,448,3)
+00034/0990 7 (256,448,3)
+00035/0129 7 (256,448,3)
+00035/0153 7 (256,448,3)
+00035/0156 7 (256,448,3)
+00035/0474 7 (256,448,3)
+00035/0507 7 (256,448,3)
+00035/0532 7 (256,448,3)
+00035/0560 7 (256,448,3)
+00035/0572 7 (256,448,3)
+00035/0587 7 (256,448,3)
+00035/0588 7 (256,448,3)
+00035/0640 7 (256,448,3)
+00035/0654 7 (256,448,3)
+00035/0655 7 (256,448,3)
+00035/0737 7 (256,448,3)
+00035/0843 7 (256,448,3)
+00035/0932 7 (256,448,3)
+00035/0957 7 (256,448,3)
+00036/0029 7 (256,448,3)
+00036/0266 7 (256,448,3)
+00036/0276 7 (256,448,3)
+00036/0310 7 (256,448,3)
+00036/0314 7 (256,448,3)
+00036/0320 7 (256,448,3)
+00036/0333 7 (256,448,3)
+00036/0348 7 (256,448,3)
+00036/0357 7 (256,448,3)
+00036/0360 7 (256,448,3)
+00036/0368 7 (256,448,3)
+00036/0371 7 (256,448,3)
+00036/0378 7 (256,448,3)
+00036/0391 7 (256,448,3)
+00036/0440 7 (256,448,3)
+00036/0731 7 (256,448,3)
+00036/0733 7 (256,448,3)
+00036/0741 7 (256,448,3)
+00036/0743 7 (256,448,3)
+00036/0927 7 (256,448,3)
+00036/0931 7 (256,448,3)
+00036/0933 7 (256,448,3)
+00036/0938 7 (256,448,3)
+00036/0944 7 (256,448,3)
+00036/0946 7 (256,448,3)
+00036/0951 7 (256,448,3)
+00036/0953 7 (256,448,3)
+00036/0963 7 (256,448,3)
+00036/0964 7 (256,448,3)
+00036/0981 7 (256,448,3)
+00036/0991 7 (256,448,3)
+00037/0072 7 (256,448,3)
+00037/0079 7 (256,448,3)
+00037/0132 7 (256,448,3)
+00037/0135 7 (256,448,3)
+00037/0137 7 (256,448,3)
+00037/0141 7 (256,448,3)
+00037/0229 7 (256,448,3)
+00037/0234 7 (256,448,3)
+00037/0239 7 (256,448,3)
+00037/0242 7 (256,448,3)
+00037/0254 7 (256,448,3)
+00037/0269 7 (256,448,3)
+00037/0276 7 (256,448,3)
+00037/0279 7 (256,448,3)
+00037/0286 7 (256,448,3)
+00037/0345 7 (256,448,3)
+00037/0449 7 (256,448,3)
+00037/0450 7 (256,448,3)
+00037/0820 7 (256,448,3)
+00037/0824 7 (256,448,3)
+00037/0859 7 (256,448,3)
+00037/0899 7 (256,448,3)
+00037/0906 7 (256,448,3)
+00038/0535 7 (256,448,3)
+00038/0572 7 (256,448,3)
+00038/0675 7 (256,448,3)
+00038/0731 7 (256,448,3)
+00038/0732 7 (256,448,3)
+00038/0744 7 (256,448,3)
+00038/0755 7 (256,448,3)
+00039/0002 7 (256,448,3)
+00039/0013 7 (256,448,3)
+00039/0247 7 (256,448,3)
+00039/0489 7 (256,448,3)
+00039/0504 7 (256,448,3)
+00039/0558 7 (256,448,3)
+00039/0686 7 (256,448,3)
+00039/0727 7 (256,448,3)
+00039/0769 7 (256,448,3)
+00040/0081 7 (256,448,3)
+00040/0082 7 (256,448,3)
+00040/0402 7 (256,448,3)
+00040/0407 7 (256,448,3)
+00040/0408 7 (256,448,3)
+00040/0410 7 (256,448,3)
+00040/0411 7 (256,448,3)
+00040/0412 7 (256,448,3)
+00040/0413 7 (256,448,3)
+00040/0415 7 (256,448,3)
+00040/0421 7 (256,448,3)
+00040/0422 7 (256,448,3)
+00040/0426 7 (256,448,3)
+00040/0438 7 (256,448,3)
+00040/0439 7 (256,448,3)
+00040/0440 7 (256,448,3)
+00040/0443 7 (256,448,3)
+00040/0457 7 (256,448,3)
+00040/0459 7 (256,448,3)
+00040/0725 7 (256,448,3)
+00040/0727 7 (256,448,3)
+00040/0936 7 (256,448,3)
+00040/0959 7 (256,448,3)
+00040/0964 7 (256,448,3)
+00040/0968 7 (256,448,3)
+00040/0974 7 (256,448,3)
+00040/0978 7 (256,448,3)
+00040/0979 7 (256,448,3)
+00040/0989 7 (256,448,3)
+00040/0993 7 (256,448,3)
+00040/0994 7 (256,448,3)
+00040/0997 7 (256,448,3)
+00041/0001 7 (256,448,3)
+00041/0007 7 (256,448,3)
+00041/0019 7 (256,448,3)
+00041/0040 7 (256,448,3)
+00041/0350 7 (256,448,3)
+00041/0357 7 (256,448,3)
+00041/0393 7 (256,448,3)
+00041/0890 7 (256,448,3)
+00041/0909 7 (256,448,3)
+00041/0915 7 (256,448,3)
+00041/0933 7 (256,448,3)
+00042/0017 7 (256,448,3)
+00042/0332 7 (256,448,3)
+00042/0346 7 (256,448,3)
+00042/0350 7 (256,448,3)
+00042/0356 7 (256,448,3)
+00042/0382 7 (256,448,3)
+00042/0389 7 (256,448,3)
+00042/0539 7 (256,448,3)
+00042/0546 7 (256,448,3)
+00042/0550 7 (256,448,3)
+00042/0553 7 (256,448,3)
+00042/0555 7 (256,448,3)
+00042/0560 7 (256,448,3)
+00042/0570 7 (256,448,3)
+00043/0119 7 (256,448,3)
+00043/0122 7 (256,448,3)
+00043/0168 7 (256,448,3)
+00043/0274 7 (256,448,3)
+00043/0304 7 (256,448,3)
+00043/0731 7 (256,448,3)
+00043/0735 7 (256,448,3)
+00043/0739 7 (256,448,3)
+00043/0740 7 (256,448,3)
+00044/0212 7 (256,448,3)
+00044/0432 7 (256,448,3)
+00044/0934 7 (256,448,3)
+00044/0940 7 (256,448,3)
+00044/0987 7 (256,448,3)
+00045/0004 7 (256,448,3)
+00045/0009 7 (256,448,3)
+00045/0011 7 (256,448,3)
+00045/0019 7 (256,448,3)
+00045/0023 7 (256,448,3)
+00045/0289 7 (256,448,3)
+00045/0760 7 (256,448,3)
+00045/0779 7 (256,448,3)
+00045/0816 7 (256,448,3)
+00045/0820 7 (256,448,3)
+00046/0132 7 (256,448,3)
+00046/0350 7 (256,448,3)
+00046/0356 7 (256,448,3)
+00046/0357 7 (256,448,3)
+00046/0379 7 (256,448,3)
+00046/0410 7 (256,448,3)
+00046/0412 7 (256,448,3)
+00046/0481 7 (256,448,3)
+00046/0497 7 (256,448,3)
+00046/0510 7 (256,448,3)
+00046/0515 7 (256,448,3)
+00046/0529 7 (256,448,3)
+00046/0544 7 (256,448,3)
+00046/0545 7 (256,448,3)
+00046/0552 7 (256,448,3)
+00046/0559 7 (256,448,3)
+00046/0589 7 (256,448,3)
+00046/0642 7 (256,448,3)
+00046/0724 7 (256,448,3)
+00046/0758 7 (256,448,3)
+00046/0930 7 (256,448,3)
+00046/0953 7 (256,448,3)
+00047/0013 7 (256,448,3)
+00047/0014 7 (256,448,3)
+00047/0017 7 (256,448,3)
+00047/0076 7 (256,448,3)
+00047/0151 7 (256,448,3)
+00047/0797 7 (256,448,3)
+00048/0014 7 (256,448,3)
+00048/0021 7 (256,448,3)
+00048/0026 7 (256,448,3)
+00048/0030 7 (256,448,3)
+00048/0039 7 (256,448,3)
+00048/0045 7 (256,448,3)
+00048/0049 7 (256,448,3)
+00048/0145 7 (256,448,3)
+00048/0188 7 (256,448,3)
+00048/0302 7 (256,448,3)
+00048/0361 7 (256,448,3)
+00048/0664 7 (256,448,3)
+00048/0672 7 (256,448,3)
+00048/0681 7 (256,448,3)
+00048/0689 7 (256,448,3)
+00048/0690 7 (256,448,3)
+00048/0691 7 (256,448,3)
+00048/0711 7 (256,448,3)
+00049/0085 7 (256,448,3)
+00049/0810 7 (256,448,3)
+00049/0858 7 (256,448,3)
+00049/0865 7 (256,448,3)
+00049/0871 7 (256,448,3)
+00049/0903 7 (256,448,3)
+00049/0928 7 (256,448,3)
+00050/0092 7 (256,448,3)
+00050/0101 7 (256,448,3)
+00050/0108 7 (256,448,3)
+00050/0112 7 (256,448,3)
+00050/0120 7 (256,448,3)
+00050/0128 7 (256,448,3)
+00050/0383 7 (256,448,3)
+00050/0395 7 (256,448,3)
+00050/0405 7 (256,448,3)
+00050/0632 7 (256,448,3)
+00050/0648 7 (256,448,3)
+00050/0649 7 (256,448,3)
+00050/0659 7 (256,448,3)
+00050/0699 7 (256,448,3)
+00050/0708 7 (256,448,3)
+00050/0716 7 (256,448,3)
+00050/0758 7 (256,448,3)
+00050/0761 7 (256,448,3)
+00051/0572 7 (256,448,3)
+00052/0163 7 (256,448,3)
+00052/0242 7 (256,448,3)
+00052/0260 7 (256,448,3)
+00052/0322 7 (256,448,3)
+00052/0333 7 (256,448,3)
+00052/0806 7 (256,448,3)
+00052/0813 7 (256,448,3)
+00052/0821 7 (256,448,3)
+00052/0830 7 (256,448,3)
+00052/0914 7 (256,448,3)
+00052/0923 7 (256,448,3)
+00052/0959 7 (256,448,3)
+00053/0288 7 (256,448,3)
+00053/0290 7 (256,448,3)
+00053/0323 7 (256,448,3)
+00053/0337 7 (256,448,3)
+00053/0340 7 (256,448,3)
+00053/0437 7 (256,448,3)
+00053/0595 7 (256,448,3)
+00053/0739 7 (256,448,3)
+00053/0761 7 (256,448,3)
+00054/0014 7 (256,448,3)
+00054/0017 7 (256,448,3)
+00054/0178 7 (256,448,3)
+00054/0183 7 (256,448,3)
+00054/0196 7 (256,448,3)
+00054/0205 7 (256,448,3)
+00054/0214 7 (256,448,3)
+00054/0289 7 (256,448,3)
+00054/0453 7 (256,448,3)
+00054/0498 7 (256,448,3)
+00054/0502 7 (256,448,3)
+00054/0514 7 (256,448,3)
+00054/0773 7 (256,448,3)
+00055/0001 7 (256,448,3)
+00055/0115 7 (256,448,3)
+00055/0118 7 (256,448,3)
+00055/0171 7 (256,448,3)
+00055/0214 7 (256,448,3)
+00055/0354 7 (256,448,3)
+00055/0449 7 (256,448,3)
+00055/0473 7 (256,448,3)
+00055/0649 7 (256,448,3)
+00055/0800 7 (256,448,3)
+00055/0803 7 (256,448,3)
+00055/0990 7 (256,448,3)
+00056/0041 7 (256,448,3)
+00056/0120 7 (256,448,3)
+00056/0293 7 (256,448,3)
+00056/0357 7 (256,448,3)
+00056/0506 7 (256,448,3)
+00056/0561 7 (256,448,3)
+00056/0567 7 (256,448,3)
+00056/0575 7 (256,448,3)
+00057/0175 7 (256,448,3)
+00057/0495 7 (256,448,3)
+00057/0498 7 (256,448,3)
+00057/0506 7 (256,448,3)
+00057/0612 7 (256,448,3)
+00057/0620 7 (256,448,3)
+00057/0623 7 (256,448,3)
+00057/0635 7 (256,448,3)
+00057/0773 7 (256,448,3)
+00057/0778 7 (256,448,3)
+00057/0867 7 (256,448,3)
+00057/0976 7 (256,448,3)
+00057/0980 7 (256,448,3)
+00057/0985 7 (256,448,3)
+00057/0992 7 (256,448,3)
+00058/0009 7 (256,448,3)
+00058/0076 7 (256,448,3)
+00058/0078 7 (256,448,3)
+00058/0279 7 (256,448,3)
+00058/0283 7 (256,448,3)
+00058/0286 7 (256,448,3)
+00058/0350 7 (256,448,3)
+00058/0380 7 (256,448,3)
+00061/0132 7 (256,448,3)
+00061/0141 7 (256,448,3)
+00061/0156 7 (256,448,3)
+00061/0159 7 (256,448,3)
+00061/0168 7 (256,448,3)
+00061/0170 7 (256,448,3)
+00061/0186 7 (256,448,3)
+00061/0219 7 (256,448,3)
+00061/0227 7 (256,448,3)
+00061/0238 7 (256,448,3)
+00061/0256 7 (256,448,3)
+00061/0303 7 (256,448,3)
+00061/0312 7 (256,448,3)
+00061/0313 7 (256,448,3)
+00061/0325 7 (256,448,3)
+00061/0367 7 (256,448,3)
+00061/0369 7 (256,448,3)
+00061/0387 7 (256,448,3)
+00061/0396 7 (256,448,3)
+00061/0486 7 (256,448,3)
+00061/0895 7 (256,448,3)
+00061/0897 7 (256,448,3)
+00062/0846 7 (256,448,3)
+00063/0156 7 (256,448,3)
+00063/0184 7 (256,448,3)
+00063/0191 7 (256,448,3)
+00063/0334 7 (256,448,3)
+00063/0350 7 (256,448,3)
+00063/0499 7 (256,448,3)
+00063/0878 7 (256,448,3)
+00064/0004 7 (256,448,3)
+00064/0264 7 (256,448,3)
+00064/0735 7 (256,448,3)
+00064/0738 7 (256,448,3)
+00065/0105 7 (256,448,3)
+00065/0169 7 (256,448,3)
+00065/0305 7 (256,448,3)
+00065/0324 7 (256,448,3)
+00065/0353 7 (256,448,3)
+00065/0520 7 (256,448,3)
+00065/0533 7 (256,448,3)
+00065/0545 7 (256,448,3)
+00065/0551 7 (256,448,3)
+00065/0568 7 (256,448,3)
+00065/0603 7 (256,448,3)
+00065/0884 7 (256,448,3)
+00065/0988 7 (256,448,3)
+00066/0002 7 (256,448,3)
+00066/0011 7 (256,448,3)
+00066/0031 7 (256,448,3)
+00066/0037 7 (256,448,3)
+00066/0136 7 (256,448,3)
+00066/0137 7 (256,448,3)
+00066/0150 7 (256,448,3)
+00066/0166 7 (256,448,3)
+00066/0178 7 (256,448,3)
+00066/0357 7 (256,448,3)
+00066/0428 7 (256,448,3)
+00066/0483 7 (256,448,3)
+00066/0600 7 (256,448,3)
+00066/0863 7 (256,448,3)
+00066/0873 7 (256,448,3)
+00066/0875 7 (256,448,3)
+00066/0899 7 (256,448,3)
+00067/0020 7 (256,448,3)
+00067/0025 7 (256,448,3)
+00067/0132 7 (256,448,3)
+00067/0492 7 (256,448,3)
+00067/0726 7 (256,448,3)
+00067/0734 7 (256,448,3)
+00067/0744 7 (256,448,3)
+00067/0754 7 (256,448,3)
+00067/0779 7 (256,448,3)
+00068/0078 7 (256,448,3)
+00068/0083 7 (256,448,3)
+00068/0113 7 (256,448,3)
+00068/0117 7 (256,448,3)
+00068/0121 7 (256,448,3)
+00068/0206 7 (256,448,3)
+00068/0261 7 (256,448,3)
+00068/0321 7 (256,448,3)
+00068/0354 7 (256,448,3)
+00068/0380 7 (256,448,3)
+00068/0419 7 (256,448,3)
+00068/0547 7 (256,448,3)
+00068/0561 7 (256,448,3)
+00068/0565 7 (256,448,3)
+00068/0583 7 (256,448,3)
+00068/0599 7 (256,448,3)
+00068/0739 7 (256,448,3)
+00068/0743 7 (256,448,3)
+00068/0754 7 (256,448,3)
+00068/0812 7 (256,448,3)
+00069/0178 7 (256,448,3)
+00070/0025 7 (256,448,3)
+00070/0030 7 (256,448,3)
+00070/0036 7 (256,448,3)
+00070/0042 7 (256,448,3)
+00070/0078 7 (256,448,3)
+00070/0079 7 (256,448,3)
+00070/0362 7 (256,448,3)
+00071/0195 7 (256,448,3)
+00071/0210 7 (256,448,3)
+00071/0211 7 (256,448,3)
+00071/0221 7 (256,448,3)
+00071/0352 7 (256,448,3)
+00071/0354 7 (256,448,3)
+00071/0366 7 (256,448,3)
+00071/0454 7 (256,448,3)
+00071/0464 7 (256,448,3)
+00071/0487 7 (256,448,3)
+00071/0502 7 (256,448,3)
+00071/0561 7 (256,448,3)
+00071/0676 7 (256,448,3)
+00071/0808 7 (256,448,3)
+00071/0813 7 (256,448,3)
+00071/0836 7 (256,448,3)
+00072/0286 7 (256,448,3)
+00072/0290 7 (256,448,3)
+00072/0298 7 (256,448,3)
+00072/0302 7 (256,448,3)
+00072/0333 7 (256,448,3)
+00072/0590 7 (256,448,3)
+00072/0793 7 (256,448,3)
+00072/0803 7 (256,448,3)
+00072/0833 7 (256,448,3)
+00073/0049 7 (256,448,3)
+00073/0050 7 (256,448,3)
+00073/0388 7 (256,448,3)
+00073/0480 7 (256,448,3)
+00073/0485 7 (256,448,3)
+00073/0611 7 (256,448,3)
+00073/0616 7 (256,448,3)
+00073/0714 7 (256,448,3)
+00073/0724 7 (256,448,3)
+00073/0730 7 (256,448,3)
+00074/0034 7 (256,448,3)
+00074/0228 7 (256,448,3)
+00074/0239 7 (256,448,3)
+00074/0275 7 (256,448,3)
+00074/0527 7 (256,448,3)
+00074/0620 7 (256,448,3)
+00074/0764 7 (256,448,3)
+00074/0849 7 (256,448,3)
+00074/0893 7 (256,448,3)
+00075/0333 7 (256,448,3)
+00075/0339 7 (256,448,3)
+00075/0347 7 (256,448,3)
+00075/0399 7 (256,448,3)
+00075/0478 7 (256,448,3)
+00075/0494 7 (256,448,3)
+00075/0678 7 (256,448,3)
+00075/0688 7 (256,448,3)
+00075/0706 7 (256,448,3)
+00075/0709 7 (256,448,3)
+00075/0748 7 (256,448,3)
+00075/0769 7 (256,448,3)
+00075/0777 7 (256,448,3)
+00075/0781 7 (256,448,3)
+00076/0151 7 (256,448,3)
+00076/0159 7 (256,448,3)
+00076/0164 7 (256,448,3)
+00076/0265 7 (256,448,3)
+00076/0269 7 (256,448,3)
+00076/0433 7 (256,448,3)
+00076/0813 7 (256,448,3)
+00076/0817 7 (256,448,3)
+00076/0818 7 (256,448,3)
+00076/0827 7 (256,448,3)
+00076/0874 7 (256,448,3)
+00076/0880 7 (256,448,3)
+00076/0891 7 (256,448,3)
+00076/0894 7 (256,448,3)
+00076/0909 7 (256,448,3)
+00076/0913 7 (256,448,3)
+00076/0926 7 (256,448,3)
+00076/0962 7 (256,448,3)
+00076/0973 7 (256,448,3)
+00076/0986 7 (256,448,3)
+00077/0617 7 (256,448,3)
+00077/0623 7 (256,448,3)
+00077/0628 7 (256,448,3)
+00077/0629 7 (256,448,3)
+00077/0631 7 (256,448,3)
+00077/0639 7 (256,448,3)
+00077/0982 7 (256,448,3)
+00077/0984 7 (256,448,3)
+00077/0995 7 (256,448,3)
+00077/0998 7 (256,448,3)
+00078/0001 7 (256,448,3)
+00078/0015 7 (256,448,3)
+00078/0157 7 (256,448,3)
+00078/0161 7 (256,448,3)
+00078/0175 7 (256,448,3)
+00078/0178 7 (256,448,3)
+00078/0189 7 (256,448,3)
+00078/0192 7 (256,448,3)
+00078/0229 7 (256,448,3)
+00078/0237 7 (256,448,3)
+00078/0241 7 (256,448,3)
+00078/0249 7 (256,448,3)
+00078/0251 7 (256,448,3)
+00078/0254 7 (256,448,3)
+00078/0258 7 (256,448,3)
+00078/0311 7 (256,448,3)
+00078/0603 7 (256,448,3)
+00078/0607 7 (256,448,3)
+00078/0824 7 (256,448,3)
+00079/0045 7 (256,448,3)
+00079/0048 7 (256,448,3)
+00079/0054 7 (256,448,3)
+00080/0050 7 (256,448,3)
+00080/0488 7 (256,448,3)
+00080/0494 7 (256,448,3)
+00080/0496 7 (256,448,3)
+00080/0499 7 (256,448,3)
+00080/0502 7 (256,448,3)
+00080/0510 7 (256,448,3)
+00080/0534 7 (256,448,3)
+00080/0558 7 (256,448,3)
+00080/0571 7 (256,448,3)
+00080/0709 7 (256,448,3)
+00080/0882 7 (256,448,3)
+00081/0322 7 (256,448,3)
+00081/0428 7 (256,448,3)
+00081/0700 7 (256,448,3)
+00081/0706 7 (256,448,3)
+00081/0707 7 (256,448,3)
+00081/0937 7 (256,448,3)
+00082/0021 7 (256,448,3)
+00082/0424 7 (256,448,3)
+00082/0794 7 (256,448,3)
+00082/0807 7 (256,448,3)
+00082/0810 7 (256,448,3)
+00082/0824 7 (256,448,3)
+00083/0129 7 (256,448,3)
+00083/0131 7 (256,448,3)
+00083/0249 7 (256,448,3)
+00083/0250 7 (256,448,3)
+00083/0656 7 (256,448,3)
+00083/0812 7 (256,448,3)
+00083/0819 7 (256,448,3)
+00083/0824 7 (256,448,3)
+00083/0827 7 (256,448,3)
+00083/0841 7 (256,448,3)
+00083/0963 7 (256,448,3)
+00084/0047 7 (256,448,3)
+00084/0319 7 (256,448,3)
+00084/0334 7 (256,448,3)
+00084/0363 7 (256,448,3)
+00084/0493 7 (256,448,3)
+00084/0655 7 (256,448,3)
+00084/0752 7 (256,448,3)
+00084/0813 7 (256,448,3)
+00084/0886 7 (256,448,3)
+00084/0948 7 (256,448,3)
+00084/0976 7 (256,448,3)
+00085/0512 7 (256,448,3)
+00085/0641 7 (256,448,3)
+00085/0653 7 (256,448,3)
+00085/0655 7 (256,448,3)
+00085/0697 7 (256,448,3)
+00085/0698 7 (256,448,3)
+00085/0700 7 (256,448,3)
+00085/0703 7 (256,448,3)
+00085/0705 7 (256,448,3)
+00085/0709 7 (256,448,3)
+00085/0713 7 (256,448,3)
+00085/0739 7 (256,448,3)
+00085/0750 7 (256,448,3)
+00085/0763 7 (256,448,3)
+00085/0765 7 (256,448,3)
+00085/0769 7 (256,448,3)
+00085/0863 7 (256,448,3)
+00085/0868 7 (256,448,3)
+00085/0927 7 (256,448,3)
+00085/0936 7 (256,448,3)
+00085/0965 7 (256,448,3)
+00085/0969 7 (256,448,3)
+00085/0974 7 (256,448,3)
+00085/0981 7 (256,448,3)
+00085/0982 7 (256,448,3)
+00085/1000 7 (256,448,3)
+00086/0003 7 (256,448,3)
+00086/0009 7 (256,448,3)
+00086/0011 7 (256,448,3)
+00086/0028 7 (256,448,3)
+00086/0032 7 (256,448,3)
+00086/0034 7 (256,448,3)
+00086/0035 7 (256,448,3)
+00086/0042 7 (256,448,3)
+00086/0064 7 (256,448,3)
+00086/0066 7 (256,448,3)
+00086/0095 7 (256,448,3)
+00086/0099 7 (256,448,3)
+00086/0101 7 (256,448,3)
+00086/0104 7 (256,448,3)
+00086/0115 7 (256,448,3)
+00086/0116 7 (256,448,3)
+00086/0284 7 (256,448,3)
+00086/0291 7 (256,448,3)
+00086/0295 7 (256,448,3)
+00086/0302 7 (256,448,3)
+00086/0318 7 (256,448,3)
+00086/0666 7 (256,448,3)
+00086/0797 7 (256,448,3)
+00086/0851 7 (256,448,3)
+00086/0855 7 (256,448,3)
+00086/0874 7 (256,448,3)
+00086/0878 7 (256,448,3)
+00086/0881 7 (256,448,3)
+00086/0883 7 (256,448,3)
+00086/0896 7 (256,448,3)
+00086/0899 7 (256,448,3)
+00086/0903 7 (256,448,3)
+00086/0989 7 (256,448,3)
+00087/0008 7 (256,448,3)
+00087/0429 7 (256,448,3)
+00087/0511 7 (256,448,3)
+00088/0241 7 (256,448,3)
+00088/0319 7 (256,448,3)
+00088/0323 7 (256,448,3)
+00088/0411 7 (256,448,3)
+00088/0427 7 (256,448,3)
+00088/0452 7 (256,448,3)
+00088/0463 7 (256,448,3)
+00088/0476 7 (256,448,3)
+00088/0496 7 (256,448,3)
+00088/0559 7 (256,448,3)
+00089/0058 7 (256,448,3)
+00089/0061 7 (256,448,3)
+00089/0069 7 (256,448,3)
+00089/0077 7 (256,448,3)
+00089/0096 7 (256,448,3)
+00089/0099 7 (256,448,3)
+00089/0100 7 (256,448,3)
+00089/0211 7 (256,448,3)
+00089/0380 7 (256,448,3)
+00089/0381 7 (256,448,3)
+00089/0384 7 (256,448,3)
+00089/0390 7 (256,448,3)
+00089/0393 7 (256,448,3)
+00089/0394 7 (256,448,3)
+00089/0395 7 (256,448,3)
+00089/0406 7 (256,448,3)
+00089/0410 7 (256,448,3)
+00089/0412 7 (256,448,3)
+00089/0703 7 (256,448,3)
+00089/0729 7 (256,448,3)
+00089/0930 7 (256,448,3)
+00089/0952 7 (256,448,3)
+00090/0062 7 (256,448,3)
+00090/0101 7 (256,448,3)
+00090/0213 7 (256,448,3)
+00090/0216 7 (256,448,3)
+00090/0268 7 (256,448,3)
+00090/0406 7 (256,448,3)
+00090/0411 7 (256,448,3)
+00090/0442 7 (256,448,3)
+00090/0535 7 (256,448,3)
+00090/0542 7 (256,448,3)
+00090/0571 7 (256,448,3)
+00090/0934 7 (256,448,3)
+00090/0938 7 (256,448,3)
+00090/0947 7 (256,448,3)
+00091/0066 7 (256,448,3)
+00091/0448 7 (256,448,3)
+00091/0451 7 (256,448,3)
+00091/0454 7 (256,448,3)
+00091/0457 7 (256,448,3)
+00091/0467 7 (256,448,3)
+00091/0470 7 (256,448,3)
+00091/0477 7 (256,448,3)
+00091/0583 7 (256,448,3)
+00091/0981 7 (256,448,3)
+00091/0994 7 (256,448,3)
+00092/0112 7 (256,448,3)
+00092/0119 7 (256,448,3)
+00092/0129 7 (256,448,3)
+00092/0146 7 (256,448,3)
+00092/0149 7 (256,448,3)
+00092/0608 7 (256,448,3)
+00092/0643 7 (256,448,3)
+00092/0646 7 (256,448,3)
+00092/0766 7 (256,448,3)
+00092/0768 7 (256,448,3)
+00092/0779 7 (256,448,3)
+00093/0081 7 (256,448,3)
+00093/0085 7 (256,448,3)
+00093/0135 7 (256,448,3)
+00093/0241 7 (256,448,3)
+00093/0277 7 (256,448,3)
+00093/0283 7 (256,448,3)
+00093/0320 7 (256,448,3)
+00093/0598 7 (256,448,3)
+00094/0159 7 (256,448,3)
+00094/0253 7 (256,448,3)
+00094/0265 7 (256,448,3)
+00094/0267 7 (256,448,3)
+00094/0269 7 (256,448,3)
+00094/0281 7 (256,448,3)
+00094/0293 7 (256,448,3)
+00094/0404 7 (256,448,3)
+00094/0593 7 (256,448,3)
+00094/0612 7 (256,448,3)
+00094/0638 7 (256,448,3)
+00094/0656 7 (256,448,3)
+00094/0668 7 (256,448,3)
+00094/0786 7 (256,448,3)
+00094/0870 7 (256,448,3)
+00094/0897 7 (256,448,3)
+00094/0900 7 (256,448,3)
+00094/0944 7 (256,448,3)
+00094/0946 7 (256,448,3)
+00094/0952 7 (256,448,3)
+00094/0969 7 (256,448,3)
+00094/0973 7 (256,448,3)
+00094/0981 7 (256,448,3)
+00095/0088 7 (256,448,3)
+00095/0125 7 (256,448,3)
+00095/0130 7 (256,448,3)
+00095/0142 7 (256,448,3)
+00095/0151 7 (256,448,3)
+00095/0180 7 (256,448,3)
+00095/0192 7 (256,448,3)
+00095/0194 7 (256,448,3)
+00095/0195 7 (256,448,3)
+00095/0204 7 (256,448,3)
+00095/0245 7 (256,448,3)
+00095/0315 7 (256,448,3)
+00095/0321 7 (256,448,3)
+00095/0324 7 (256,448,3)
+00095/0327 7 (256,448,3)
+00095/0730 7 (256,448,3)
+00095/0731 7 (256,448,3)
+00095/0741 7 (256,448,3)
+00095/0948 7 (256,448,3)
+00096/0407 7 (256,448,3)
+00096/0420 7 (256,448,3)
+00096/0435 7 (256,448,3)
+00096/0682 7 (256,448,3)
+00096/0865 7 (256,448,3)

basicsr/data/meta_info/meta_info_Vimeo90K_test_slow_GT.txt

@@ -0,0 +1,1613 @@
+00001/0266 7 (256,448,3)
+00001/0268 7 (256,448,3)
+00001/0275 7 (256,448,3)
+00001/0278 7 (256,448,3)
+00001/0287 7 (256,448,3)
+00001/0291 7 (256,448,3)
+00001/0627 7 (256,448,3)
+00001/0636 7 (256,448,3)
+00001/0804 7 (256,448,3)
+00001/0837 7 (256,448,3)
+00001/0849 7 (256,448,3)
+00001/0851 7 (256,448,3)
+00001/0852 7 (256,448,3)
+00001/0986 7 (256,448,3)
+00001/0991 7 (256,448,3)
+00002/0007 7 (256,448,3)
+00002/0008 7 (256,448,3)
+00002/0016 7 (256,448,3)
+00002/0036 7 (256,448,3)
+00002/0091 7 (256,448,3)
+00002/0093 7 (256,448,3)
+00002/0209 7 (256,448,3)
+00002/0235 7 (256,448,3)
+00002/0236 7 (256,448,3)
+00002/0241 7 (256,448,3)
+00002/0466 7 (256,448,3)
+00002/0504 7 (256,448,3)
+00002/0960 7 (256,448,3)
+00002/0961 7 (256,448,3)
+00002/0964 7 (256,448,3)
+00003/0007 7 (256,448,3)
+00003/0069 7 (256,448,3)
+00003/0345 7 (256,448,3)
+00003/0347 7 (256,448,3)
+00003/0372 7 (256,448,3)
+00003/0525 7 (256,448,3)
+00003/0652 7 (256,448,3)
+00003/0667 7 (256,448,3)
+00003/0669 7 (256,448,3)
+00003/0706 7 (256,448,3)
+00003/0713 7 (256,448,3)
+00003/0721 7 (256,448,3)
+00003/0747 7 (256,448,3)
+00003/0829 7 (256,448,3)
+00003/0916 7 (256,448,3)
+00003/0918 7 (256,448,3)
+00003/0924 7 (256,448,3)
+00003/0926 7 (256,448,3)
+00003/0927 7 (256,448,3)
+00004/0288 7 (256,448,3)
+00004/0303 7 (256,448,3)
+00004/0307 7 (256,448,3)
+00004/0628 7 (256,448,3)
+00004/0713 7 (256,448,3)
+00004/0715 7 (256,448,3)
+00004/0719 7 (256,448,3)
+00004/0727 7 (256,448,3)
+00004/0821 7 (256,448,3)
+00005/0006 7 (256,448,3)
+00005/0007 7 (256,448,3)
+00005/0012 7 (256,448,3)
+00005/0013 7 (256,448,3)
+00005/0040 7 (256,448,3)
+00005/0055 7 (256,448,3)
+00005/0119 7 (256,448,3)
+00005/0130 7 (256,448,3)
+00005/0185 7 (256,448,3)
+00005/0198 7 (256,448,3)
+00005/0270 7 (256,448,3)
+00005/0541 7 (256,448,3)
+00005/0560 7 (256,448,3)
+00005/0660 7 (256,448,3)
+00005/0682 7 (256,448,3)
+00005/0683 7 (256,448,3)
+00005/0688 7 (256,448,3)
+00005/0706 7 (256,448,3)
+00005/0728 7 (256,448,3)
+00005/0732 7 (256,448,3)
+00005/0739 7 (256,448,3)
+00005/0804 7 (256,448,3)
+00005/0805 7 (256,448,3)
+00005/0827 7 (256,448,3)
+00005/0828 7 (256,448,3)
+00005/0857 7 (256,448,3)
+00005/0861 7 (256,448,3)
+00005/0862 7 (256,448,3)
+00005/0868 7 (256,448,3)
+00005/0872 7 (256,448,3)
+00005/0933 7 (256,448,3)
+00005/0958 7 (256,448,3)
+00005/0960 7 (256,448,3)
+00006/0087 7 (256,448,3)
+00006/0090 7 (256,448,3)
+00006/0351 7 (256,448,3)
+00006/0353 7 (256,448,3)
+00006/0558 7 (256,448,3)
+00006/0588 7 (256,448,3)
+00006/0619 7 (256,448,3)
+00006/0621 7 (256,448,3)
+00006/0748 7 (256,448,3)
+00006/0796 7 (256,448,3)
+00006/0805 7 (256,448,3)
+00006/0807 7 (256,448,3)
+00007/0236 7 (256,448,3)
+00007/0240 7 (256,448,3)
+00007/0243 7 (256,448,3)
+00007/0246 7 (256,448,3)
+00007/0247 7 (256,448,3)
+00007/0252 7 (256,448,3)
+00007/0322 7 (256,448,3)
+00007/0458 7 (256,448,3)
+00007/0492 7 (256,448,3)
+00007/0658 7 (256,448,3)
+00007/0717 7 (256,448,3)
+00007/0722 7 (256,448,3)
+00007/0725 7 (256,448,3)
+00007/0740 7 (256,448,3)
+00007/0748 7 (256,448,3)
+00007/0749 7 (256,448,3)
+00007/0759 7 (256,448,3)
+00007/0772 7 (256,448,3)
+00007/0783 7 (256,448,3)
+00007/0787 7 (256,448,3)
+00007/0883 7 (256,448,3)
+00008/0033 7 (256,448,3)
+00008/0035 7 (256,448,3)
+00008/0091 7 (256,448,3)
+00008/0154 7 (256,448,3)
+00008/0966 7 (256,448,3)
+00008/0987 7 (256,448,3)
+00009/0108 7 (256,448,3)
+00009/0607 7 (256,448,3)
+00009/0668 7 (256,448,3)
+00009/0683 7 (256,448,3)
+00009/0941 7 (256,448,3)
+00009/0949 7 (256,448,3)
+00009/0962 7 (256,448,3)
+00009/0972 7 (256,448,3)
+00009/0974 7 (256,448,3)
+00010/0014 7 (256,448,3)
+00010/0018 7 (256,448,3)
+00010/0043 7 (256,448,3)
+00010/0099 7 (256,448,3)
+00010/0252 7 (256,448,3)
+00010/0296 7 (256,448,3)
+00010/0413 7 (256,448,3)
+00010/0422 7 (256,448,3)
+00010/0516 7 (256,448,3)
+00010/0525 7 (256,448,3)
+00010/0556 7 (256,448,3)
+00010/0701 7 (256,448,3)
+00010/0740 7 (256,448,3)
+00010/0772 7 (256,448,3)
+00010/0831 7 (256,448,3)
+00010/0925 7 (256,448,3)
+00011/0013 7 (256,448,3)
+00011/0016 7 (256,448,3)
+00011/0017 7 (256,448,3)
+00011/0249 7 (256,448,3)
+00011/0826 7 (256,448,3)
+00011/0827 7 (256,448,3)
+00011/0831 7 (256,448,3)
+00011/0833 7 (256,448,3)
+00011/0835 7 (256,448,3)
+00011/0998 7 (256,448,3)
+00012/0023 7 (256,448,3)
+00012/0024 7 (256,448,3)
+00012/0027 7 (256,448,3)
+00012/0037 7 (256,448,3)
+00012/0444 7 (256,448,3)
+00012/0445 7 (256,448,3)
+00012/0451 7 (256,448,3)
+00012/0461 7 (256,448,3)
+00012/0521 7 (256,448,3)
+00012/0758 7 (256,448,3)
+00012/0760 7 (256,448,3)
+00012/0771 7 (256,448,3)
+00012/0903 7 (256,448,3)
+00012/0909 7 (256,448,3)
+00013/0581 7 (256,448,3)
+00013/0786 7 (256,448,3)
+00013/0789 7 (256,448,3)
+00013/0791 7 (256,448,3)
+00013/0798 7 (256,448,3)
+00013/0802 7 (256,448,3)
+00013/0820 7 (256,448,3)
+00013/0850 7 (256,448,3)
+00013/0854 7 (256,448,3)
+00013/0894 7 (256,448,3)
+00013/0919 7 (256,448,3)
+00013/0999 7 (256,448,3)
+00014/0001 7 (256,448,3)
+00014/0014 7 (256,448,3)
+00014/0018 7 (256,448,3)
+00014/0244 7 (256,448,3)
+00014/0475 7 (256,448,3)
+00014/0483 7 (256,448,3)
+00014/0680 7 (256,448,3)
+00014/0700 7 (256,448,3)
+00014/0701 7 (256,448,3)
+00014/0706 7 (256,448,3)
+00014/0712 7 (256,448,3)
+00014/0713 7 (256,448,3)
+00014/0717 7 (256,448,3)
+00014/0719 7 (256,448,3)
+00014/0728 7 (256,448,3)
+00014/0734 7 (256,448,3)
+00014/0736 7 (256,448,3)
+00014/0738 7 (256,448,3)
+00014/0742 7 (256,448,3)
+00014/0745 7 (256,448,3)
+00014/0746 7 (256,448,3)
+00014/0750 7 (256,448,3)
+00014/0769 7 (256,448,3)
+00014/0774 7 (256,448,3)
+00014/0781 7 (256,448,3)
+00014/0782 7 (256,448,3)
+00014/0852 7 (256,448,3)
+00014/0853 7 (256,448,3)
+00014/0855 7 (256,448,3)
+00014/0867 7 (256,448,3)
+00014/0876 7 (256,448,3)
+00014/0881 7 (256,448,3)
+00014/0890 7 (256,448,3)
+00014/0914 7 (256,448,3)
+00015/0033 7 (256,448,3)
+00015/0113 7 (256,448,3)
+00015/0125 7 (256,448,3)
+00015/0185 7 (256,448,3)
+00015/0194 7 (256,448,3)
+00015/0202 7 (256,448,3)
+00015/0312 7 (256,448,3)
+00015/0688 7 (256,448,3)
+00015/0698 7 (256,448,3)
+00015/0788 7 (256,448,3)
+00015/0854 7 (256,448,3)
+00015/0863 7 (256,448,3)
+00015/0864 7 (256,448,3)
+00015/0918 7 (256,448,3)
+00015/0931 7 (256,448,3)
+00016/0276 7 (256,448,3)
+00016/0301 7 (256,448,3)
+00016/0306 7 (256,448,3)
+00016/0324 7 (256,448,3)
+00016/0362 7 (256,448,3)
+00016/0364 7 (256,448,3)
+00016/0370 7 (256,448,3)
+00016/0378 7 (256,448,3)
+00016/0379 7 (256,448,3)
+00016/0402 7 (256,448,3)
+00016/0405 7 (256,448,3)
+00016/0418 7 (256,448,3)
+00016/0419 7 (256,448,3)
+00016/0435 7 (256,448,3)
+00016/0501 7 (256,448,3)
+00016/0561 7 (256,448,3)
+00016/0562 7 (256,448,3)
+00016/0569 7 (256,448,3)
+00016/0591 7 (256,448,3)
+00016/0599 7 (256,448,3)
+00016/0711 7 (256,448,3)
+00016/0713 7 (256,448,3)
+00016/0813 7 (256,448,3)
+00016/0953 7 (256,448,3)
+00016/0960 7 (256,448,3)
+00016/0961 7 (256,448,3)
+00017/0519 7 (256,448,3)
+00017/0523 7 (256,448,3)
+00017/0588 7 (256,448,3)
+00017/0608 7 (256,448,3)
+00017/0609 7 (256,448,3)
+00017/0719 7 (256,448,3)
+00017/0721 7 (256,448,3)
+00017/0727 7 (256,448,3)
+00017/0728 7 (256,448,3)
+00017/0769 7 (256,448,3)
+00017/0775 7 (256,448,3)
+00017/0787 7 (256,448,3)
+00017/0797 7 (256,448,3)
+00018/0043 7 (256,448,3)
+00018/0206 7 (256,448,3)
+00018/0209 7 (256,448,3)
+00018/0211 7 (256,448,3)
+00018/0216 7 (256,448,3)
+00018/0220 7 (256,448,3)
+00018/0221 7 (256,448,3)
+00018/0252 7 (256,448,3)
+00018/0260 7 (256,448,3)
+00018/0331 7 (256,448,3)
+00018/0333 7 (256,448,3)
+00018/0447 7 (256,448,3)
+00018/0523 7 (256,448,3)
+00019/0014 7 (256,448,3)
+00019/0015 7 (256,448,3)
+00019/0019 7 (256,448,3)
+00019/0049 7 (256,448,3)
+00019/0109 7 (256,448,3)
+00019/0114 7 (256,448,3)
+00019/0125 7 (256,448,3)
+00019/0137 7 (256,448,3)
+00019/0140 7 (256,448,3)
+00019/0148 7 (256,448,3)
+00019/0153 7 (256,448,3)
+00019/0155 7 (256,448,3)
+00019/0158 7 (256,448,3)
+00019/0159 7 (256,448,3)
+00019/0160 7 (256,448,3)
+00019/0162 7 (256,448,3)
+00019/0279 7 (256,448,3)
+00019/0282 7 (256,448,3)
+00019/0409 7 (256,448,3)
+00019/0427 7 (256,448,3)
+00019/0430 7 (256,448,3)
+00019/0545 7 (256,448,3)
+00019/0555 7 (256,448,3)
+00019/0558 7 (256,448,3)
+00019/0650 7 (256,448,3)
+00019/0681 7 (256,448,3)
+00019/0747 7 (256,448,3)
+00019/0748 7 (256,448,3)
+00019/0749 7 (256,448,3)
+00019/0752 7 (256,448,3)
+00019/0768 7 (256,448,3)
+00019/0772 7 (256,448,3)
+00019/0773 7 (256,448,3)
+00019/0777 7 (256,448,3)
+00019/0795 7 (256,448,3)
+00019/0806 7 (256,448,3)
+00019/0815 7 (256,448,3)
+00019/0840 7 (256,448,3)
+00019/0844 7 (256,448,3)
+00019/0848 7 (256,448,3)
+00019/0853 7 (256,448,3)
+00019/0863 7 (256,448,3)
+00019/0888 7 (256,448,3)
+00019/0894 7 (256,448,3)
+00019/0901 7 (256,448,3)
+00019/0995 7 (256,448,3)
+00021/0030 7 (256,448,3)
+00021/0035 7 (256,448,3)
+00021/0039 7 (256,448,3)
+00021/0041 7 (256,448,3)
+00021/0044 7 (256,448,3)
+00021/0045 7 (256,448,3)
+00021/0264 7 (256,448,3)
+00021/0330 7 (256,448,3)
+00021/0332 7 (256,448,3)
+00021/0333 7 (256,448,3)
+00021/0336 7 (256,448,3)
+00021/0337 7 (256,448,3)
+00021/0338 7 (256,448,3)
+00021/0343 7 (256,448,3)
+00021/0472 7 (256,448,3)
+00021/0667 7 (256,448,3)
+00021/0731 7 (256,448,3)
+00021/0779 7 (256,448,3)
+00021/0805 7 (256,448,3)
+00021/0814 7 (256,448,3)
+00021/0818 7 (256,448,3)
+00021/0874 7 (256,448,3)
+00022/0008 7 (256,448,3)
+00022/0010 7 (256,448,3)
+00022/0231 7 (256,448,3)
+00022/0323 7 (256,448,3)
+00022/0337 7 (256,448,3)
+00022/0359 7 (256,448,3)
+00022/0377 7 (256,448,3)
+00022/0378 7 (256,448,3)
+00022/0385 7 (256,448,3)
+00022/0393 7 (256,448,3)
+00022/0424 7 (256,448,3)
+00022/0582 7 (256,448,3)
+00022/0583 7 (256,448,3)
+00022/0605 7 (256,448,3)
+00022/0632 7 (256,448,3)
+00022/0633 7 (256,448,3)
+00022/0666 7 (256,448,3)
+00022/0671 7 (256,448,3)
+00022/0673 7 (256,448,3)
+00022/0702 7 (256,448,3)
+00022/0852 7 (256,448,3)
+00022/0853 7 (256,448,3)
+00022/0971 7 (256,448,3)
+00023/0037 7 (256,448,3)
+00023/0224 7 (256,448,3)
+00023/0308 7 (256,448,3)
+00023/0393 7 (256,448,3)
+00023/0633 7 (256,448,3)
+00023/0637 7 (256,448,3)
+00023/0638 7 (256,448,3)
+00023/0770 7 (256,448,3)
+00023/0786 7 (256,448,3)
+00023/0898 7 (256,448,3)
+00024/0247 7 (256,448,3)
+00024/0251 7 (256,448,3)
+00024/0267 7 (256,448,3)
+00024/0288 7 (256,448,3)
+00024/0530 7 (256,448,3)
+00024/0569 7 (256,448,3)
+00024/0587 7 (256,448,3)
+00024/0730 7 (256,448,3)
+00024/0736 7 (256,448,3)
+00024/0742 7 (256,448,3)
+00024/0832 7 (256,448,3)
+00024/0936 7 (256,448,3)
+00025/0044 7 (256,448,3)
+00025/0047 7 (256,448,3)
+00025/0540 7 (256,448,3)
+00025/0552 7 (256,448,3)
+00025/0554 7 (256,448,3)
+00025/0559 7 (256,448,3)
+00025/0572 7 (256,448,3)
+00025/0576 7 (256,448,3)
+00025/0699 7 (256,448,3)
+00025/0709 7 (256,448,3)
+00025/0743 7 (256,448,3)
+00025/0767 7 (256,448,3)
+00025/0780 7 (256,448,3)
+00025/0782 7 (256,448,3)
+00025/0784 7 (256,448,3)
+00025/0791 7 (256,448,3)
+00025/0889 7 (256,448,3)
+00025/0890 7 (256,448,3)
+00025/0894 7 (256,448,3)
+00025/0896 7 (256,448,3)
+00025/0898 7 (256,448,3)
+00025/0905 7 (256,448,3)
+00025/0999 7 (256,448,3)
+00026/0003 7 (256,448,3)
+00026/0005 7 (256,448,3)
+00026/0011 7 (256,448,3)
+00026/0017 7 (256,448,3)
+00026/0036 7 (256,448,3)
+00026/0129 7 (256,448,3)
+00026/0131 7 (256,448,3)
+00026/0161 7 (256,448,3)
+00026/0177 7 (256,448,3)
+00026/0178 7 (256,448,3)
+00026/0180 7 (256,448,3)
+00026/0298 7 (256,448,3)
+00026/0307 7 (256,448,3)
+00026/0308 7 (256,448,3)
+00026/0312 7 (256,448,3)
+00026/0352 7 (256,448,3)
+00026/0440 7 (256,448,3)
+00026/0706 7 (256,448,3)
+00026/0708 7 (256,448,3)
+00026/0715 7 (256,448,3)
+00026/0769 7 (256,448,3)
+00026/0777 7 (256,448,3)
+00026/0779 7 (256,448,3)
+00026/0789 7 (256,448,3)
+00026/0924 7 (256,448,3)
+00026/0928 7 (256,448,3)
+00026/0932 7 (256,448,3)
+00026/0935 7 (256,448,3)
+00027/0118 7 (256,448,3)
+00027/0121 7 (256,448,3)
+00027/0155 7 (256,448,3)
+00027/0168 7 (256,448,3)
+00027/0196 7 (256,448,3)
+00027/0289 7 (256,448,3)
+00027/0294 7 (256,448,3)
+00027/0803 7 (256,448,3)
+00028/0016 7 (256,448,3)
+00028/0045 7 (256,448,3)
+00028/0063 7 (256,448,3)
+00028/0601 7 (256,448,3)
+00028/0638 7 (256,448,3)
+00028/0733 7 (256,448,3)
+00028/0736 7 (256,448,3)
+00028/0741 7 (256,448,3)
+00028/0753 7 (256,448,3)
+00028/0770 7 (256,448,3)
+00028/0771 7 (256,448,3)
+00028/0777 7 (256,448,3)
+00028/0950 7 (256,448,3)
+00028/0951 7 (256,448,3)
+00029/0048 7 (256,448,3)
+00029/0060 7 (256,448,3)
+00029/0362 7 (256,448,3)
+00029/0399 7 (256,448,3)
+00029/0404 7 (256,448,3)
+00029/0412 7 (256,448,3)
+00029/0416 7 (256,448,3)
+00029/0418 7 (256,448,3)
+00029/0428 7 (256,448,3)
+00030/0131 7 (256,448,3)
+00030/0135 7 (256,448,3)
+00030/0150 7 (256,448,3)
+00030/0245 7 (256,448,3)
+00030/0339 7 (256,448,3)
+00030/0472 7 (256,448,3)
+00030/0482 7 (256,448,3)
+00030/0500 7 (256,448,3)
+00030/0501 7 (256,448,3)
+00030/0697 7 (256,448,3)
+00030/0707 7 (256,448,3)
+00030/0733 7 (256,448,3)
+00030/0743 7 (256,448,3)
+00030/0747 7 (256,448,3)
+00030/0754 7 (256,448,3)
+00030/0755 7 (256,448,3)
+00030/0759 7 (256,448,3)
+00030/0762 7 (256,448,3)
+00030/0764 7 (256,448,3)
+00030/0767 7 (256,448,3)
+00030/0794 7 (256,448,3)
+00030/0796 7 (256,448,3)
+00030/0799 7 (256,448,3)
+00030/0814 7 (256,448,3)
+00030/0823 7 (256,448,3)
+00030/0829 7 (256,448,3)
+00030/0833 7 (256,448,3)
+00030/0848 7 (256,448,3)
+00030/0853 7 (256,448,3)
+00030/0861 7 (256,448,3)
+00031/0182 7 (256,448,3)
+00031/0275 7 (256,448,3)
+00031/0279 7 (256,448,3)
+00031/0555 7 (256,448,3)
+00031/0648 7 (256,448,3)
+00031/0663 7 (256,448,3)
+00031/0680 7 (256,448,3)
+00031/0880 7 (256,448,3)
+00031/0922 7 (256,448,3)
+00031/0925 7 (256,448,3)
+00031/0928 7 (256,448,3)
+00032/0025 7 (256,448,3)
+00032/0377 7 (256,448,3)
+00032/0378 7 (256,448,3)
+00032/0382 7 (256,448,3)
+00032/0384 7 (256,448,3)
+00032/0386 7 (256,448,3)
+00032/0389 7 (256,448,3)
+00032/0391 7 (256,448,3)
+00032/0393 7 (256,448,3)
+00032/0492 7 (256,448,3)
+00032/0497 7 (256,448,3)
+00032/0505 7 (256,448,3)
+00032/0523 7 (256,448,3)
+00032/0542 7 (256,448,3)
+00032/0544 7 (256,448,3)
+00032/0712 7 (256,448,3)
+00032/0847 7 (256,448,3)
+00032/0850 7 (256,448,3)
+00032/0875 7 (256,448,3)
+00033/0062 7 (256,448,3)
+00033/0063 7 (256,448,3)
+00033/0098 7 (256,448,3)
+00033/0101 7 (256,448,3)
+00033/0105 7 (256,448,3)
+00033/0114 7 (256,448,3)
+00033/0432 7 (256,448,3)
+00033/0441 7 (256,448,3)
+00033/0606 7 (256,448,3)
+00033/0611 7 (256,448,3)
+00033/0634 7 (256,448,3)
+00033/0787 7 (256,448,3)
+00033/0792 7 (256,448,3)
+00033/0802 7 (256,448,3)
+00033/0825 7 (256,448,3)
+00033/0835 7 (256,448,3)
+00034/0249 7 (256,448,3)
+00034/0253 7 (256,448,3)
+00034/0254 7 (256,448,3)
+00034/0282 7 (256,448,3)
+00034/0318 7 (256,448,3)
+00034/0319 7 (256,448,3)
+00034/0323 7 (256,448,3)
+00034/0336 7 (256,448,3)
+00034/0348 7 (256,448,3)
+00034/0356 7 (256,448,3)
+00034/0379 7 (256,448,3)
+00034/0387 7 (256,448,3)
+00034/0575 7 (256,448,3)
+00034/0608 7 (256,448,3)
+00034/0663 7 (256,448,3)
+00034/0811 7 (256,448,3)
+00034/0812 7 (256,448,3)
+00034/0946 7 (256,448,3)
+00034/0948 7 (256,448,3)
+00034/0950 7 (256,448,3)
+00035/0204 7 (256,448,3)
+00035/0243 7 (256,448,3)
+00035/0308 7 (256,448,3)
+00035/0465 7 (256,448,3)
+00035/0478 7 (256,448,3)
+00035/0523 7 (256,448,3)
+00035/0540 7 (256,448,3)
+00035/0544 7 (256,448,3)
+00035/0556 7 (256,448,3)
+00035/0568 7 (256,448,3)
+00035/0570 7 (256,448,3)
+00035/0609 7 (256,448,3)
+00035/0643 7 (256,448,3)
+00035/0644 7 (256,448,3)
+00035/0645 7 (256,448,3)
+00035/0646 7 (256,448,3)
+00035/0650 7 (256,448,3)
+00035/0661 7 (256,448,3)
+00035/0724 7 (256,448,3)
+00035/0725 7 (256,448,3)
+00035/0850 7 (256,448,3)
+00035/0863 7 (256,448,3)
+00035/0870 7 (256,448,3)
+00035/0951 7 (256,448,3)
+00036/0038 7 (256,448,3)
+00036/0062 7 (256,448,3)
+00036/0423 7 (256,448,3)
+00036/0737 7 (256,448,3)
+00036/0750 7 (256,448,3)
+00036/0751 7 (256,448,3)
+00036/0754 7 (256,448,3)
+00036/0929 7 (256,448,3)
+00037/0085 7 (256,448,3)
+00037/0113 7 (256,448,3)
+00037/0130 7 (256,448,3)
+00037/0153 7 (256,448,3)
+00037/0169 7 (256,448,3)
+00037/0263 7 (256,448,3)
+00037/0272 7 (256,448,3)
+00037/0273 7 (256,448,3)
+00037/0275 7 (256,448,3)
+00037/0280 7 (256,448,3)
+00037/0399 7 (256,448,3)
+00037/0456 7 (256,448,3)
+00037/0853 7 (256,448,3)
+00037/0855 7 (256,448,3)
+00037/0856 7 (256,448,3)
+00037/0857 7 (256,448,3)
+00037/0925 7 (256,448,3)
+00037/0947 7 (256,448,3)
+00038/0148 7 (256,448,3)
+00038/0533 7 (256,448,3)
+00038/0534 7 (256,448,3)
+00038/0560 7 (256,448,3)
+00038/0562 7 (256,448,3)
+00038/0566 7 (256,448,3)
+00038/0578 7 (256,448,3)
+00038/0652 7 (256,448,3)
+00038/0674 7 (256,448,3)
+00038/0685 7 (256,448,3)
+00038/0686 7 (256,448,3)
+00038/0692 7 (256,448,3)
+00038/0736 7 (256,448,3)
+00039/0035 7 (256,448,3)
+00039/0105 7 (256,448,3)
+00039/0109 7 (256,448,3)
+00039/0121 7 (256,448,3)
+00039/0128 7 (256,448,3)
+00039/0129 7 (256,448,3)
+00039/0132 7 (256,448,3)
+00039/0137 7 (256,448,3)
+00039/0157 7 (256,448,3)
+00039/0496 7 (256,448,3)
+00039/0502 7 (256,448,3)
+00039/0526 7 (256,448,3)
+00039/0529 7 (256,448,3)
+00039/0682 7 (256,448,3)
+00039/0690 7 (256,448,3)
+00039/0693 7 (256,448,3)
+00039/0703 7 (256,448,3)
+00039/0725 7 (256,448,3)
+00039/0734 7 (256,448,3)
+00040/0518 7 (256,448,3)
+00040/0728 7 (256,448,3)
+00040/0774 7 (256,448,3)
+00040/0812 7 (256,448,3)
+00040/0818 7 (256,448,3)
+00040/0827 7 (256,448,3)
+00040/0914 7 (256,448,3)
+00040/0917 7 (256,448,3)
+00040/0918 7 (256,448,3)
+00040/0924 7 (256,448,3)
+00040/0925 7 (256,448,3)
+00041/0004 7 (256,448,3)
+00041/0006 7 (256,448,3)
+00041/0013 7 (256,448,3)
+00041/0059 7 (256,448,3)
+00041/0110 7 (256,448,3)
+00041/0291 7 (256,448,3)
+00041/0366 7 (256,448,3)
+00041/0388 7 (256,448,3)
+00041/0434 7 (256,448,3)
+00041/0436 7 (256,448,3)
+00041/0450 7 (256,448,3)
+00041/0457 7 (256,448,3)
+00041/0460 7 (256,448,3)
+00041/0468 7 (256,448,3)
+00041/0471 7 (256,448,3)
+00041/0474 7 (256,448,3)
+00041/0809 7 (256,448,3)
+00041/0844 7 (256,448,3)
+00041/0858 7 (256,448,3)
+00041/0874 7 (256,448,3)
+00041/0876 7 (256,448,3)
+00042/0020 7 (256,448,3)
+00042/0205 7 (256,448,3)
+00042/0206 7 (256,448,3)
+00042/0432 7 (256,448,3)
+00042/0563 7 (256,448,3)
+00042/0569 7 (256,448,3)
+00042/0575 7 (256,448,3)
+00042/0576 7 (256,448,3)
+00042/0888 7 (256,448,3)
+00042/0892 7 (256,448,3)
+00042/0943 7 (256,448,3)
+00042/0944 7 (256,448,3)
+00043/0126 7 (256,448,3)
+00043/0130 7 (256,448,3)
+00043/0136 7 (256,448,3)
+00043/0233 7 (256,448,3)
+00043/0235 7 (256,448,3)
+00043/0237 7 (256,448,3)
+00043/0277 7 (256,448,3)
+00043/0301 7 (256,448,3)
+00043/0302 7 (256,448,3)
+00043/0303 7 (256,448,3)
+00043/0308 7 (256,448,3)
+00043/0309 7 (256,448,3)
+00043/0314 7 (256,448,3)
+00043/0713 7 (256,448,3)
+00043/0715 7 (256,448,3)
+00043/0923 7 (256,448,3)
+00044/0095 7 (256,448,3)
+00044/0255 7 (256,448,3)
+00044/0864 7 (256,448,3)
+00044/0892 7 (256,448,3)
+00044/0898 7 (256,448,3)
+00044/0993 7 (256,448,3)
+00044/0995 7 (256,448,3)
+00044/0997 7 (256,448,3)
+00045/0001 7 (256,448,3)
+00045/0006 7 (256,448,3)
+00045/0269 7 (256,448,3)
+00045/0276 7 (256,448,3)
+00045/0280 7 (256,448,3)
+00045/0281 7 (256,448,3)
+00045/0282 7 (256,448,3)
+00045/0284 7 (256,448,3)
+00045/0550 7 (256,448,3)
+00045/0571 7 (256,448,3)
+00045/0629 7 (256,448,3)
+00045/0631 7 (256,448,3)
+00045/0659 7 (256,448,3)
+00045/0693 7 (256,448,3)
+00045/0807 7 (256,448,3)
+00045/0810 7 (256,448,3)
+00045/0826 7 (256,448,3)
+00045/0849 7 (256,448,3)
+00045/0946 7 (256,448,3)
+00045/0987 7 (256,448,3)
+00045/0990 7 (256,448,3)
+00046/0104 7 (256,448,3)
+00046/0477 7 (256,448,3)
+00046/0490 7 (256,448,3)
+00046/0491 7 (256,448,3)
+00046/0509 7 (256,448,3)
+00046/0513 7 (256,448,3)
+00046/0603 7 (256,448,3)
+00046/0723 7 (256,448,3)
+00046/0744 7 (256,448,3)
+00046/0746 7 (256,448,3)
+00046/0750 7 (256,448,3)
+00046/0852 7 (256,448,3)
+00046/0927 7 (256,448,3)
+00046/0928 7 (256,448,3)
+00046/0929 7 (256,448,3)
+00046/0931 7 (256,448,3)
+00046/0936 7 (256,448,3)
+00046/0939 7 (256,448,3)
+00046/0947 7 (256,448,3)
+00046/0948 7 (256,448,3)
+00046/0950 7 (256,448,3)
+00046/0955 7 (256,448,3)
+00046/0961 7 (256,448,3)
+00047/0023 7 (256,448,3)
+00047/0029 7 (256,448,3)
+00047/0035 7 (256,448,3)
+00047/0058 7 (256,448,3)
+00047/0061 7 (256,448,3)
+00047/0065 7 (256,448,3)
+00047/0068 7 (256,448,3)
+00047/0072 7 (256,448,3)
+00047/0074 7 (256,448,3)
+00047/0148 7 (256,448,3)
+00047/0594 7 (256,448,3)
+00047/0782 7 (256,448,3)
+00047/0787 7 (256,448,3)
+00047/0860 7 (256,448,3)
+00047/0889 7 (256,448,3)
+00047/0893 7 (256,448,3)
+00047/0894 7 (256,448,3)
+00047/0902 7 (256,448,3)
+00047/0975 7 (256,448,3)
+00047/0995 7 (256,448,3)
+00048/0033 7 (256,448,3)
+00048/0113 7 (256,448,3)
+00048/0115 7 (256,448,3)
+00048/0120 7 (256,448,3)
+00048/0129 7 (256,448,3)
+00048/0136 7 (256,448,3)
+00048/0327 7 (256,448,3)
+00048/0329 7 (256,448,3)
+00048/0341 7 (256,448,3)
+00048/0343 7 (256,448,3)
+00048/0345 7 (256,448,3)
+00048/0346 7 (256,448,3)
+00048/0355 7 (256,448,3)
+00048/0359 7 (256,448,3)
+00048/0363 7 (256,448,3)
+00048/0378 7 (256,448,3)
+00048/0386 7 (256,448,3)
+00048/0387 7 (256,448,3)
+00048/0388 7 (256,448,3)
+00048/0428 7 (256,448,3)
+00048/0439 7 (256,448,3)
+00048/0507 7 (256,448,3)
+00048/0510 7 (256,448,3)
+00048/0512 7 (256,448,3)
+00048/0514 7 (256,448,3)
+00048/0539 7 (256,448,3)
+00048/0542 7 (256,448,3)
+00048/0544 7 (256,448,3)
+00048/0631 7 (256,448,3)
+00048/0632 7 (256,448,3)
+00048/0636 7 (256,448,3)
+00048/0640 7 (256,448,3)
+00048/0644 7 (256,448,3)
+00048/0653 7 (256,448,3)
+00048/0655 7 (256,448,3)
+00048/0658 7 (256,448,3)
+00048/0667 7 (256,448,3)
+00048/0688 7 (256,448,3)
+00048/0708 7 (256,448,3)
+00049/0005 7 (256,448,3)
+00049/0074 7 (256,448,3)
+00049/0077 7 (256,448,3)
+00049/0084 7 (256,448,3)
+00049/0516 7 (256,448,3)
+00049/0800 7 (256,448,3)
+00049/0900 7 (256,448,3)
+00050/0607 7 (256,448,3)
+00050/0661 7 (256,448,3)
+00050/0665 7 (256,448,3)
+00050/0685 7 (256,448,3)
+00050/0711 7 (256,448,3)
+00051/0068 7 (256,448,3)
+00051/0069 7 (256,448,3)
+00051/0076 7 (256,448,3)
+00051/0569 7 (256,448,3)
+00051/0801 7 (256,448,3)
+00051/0927 7 (256,448,3)
+00051/0945 7 (256,448,3)
+00051/0952 7 (256,448,3)
+00051/0976 7 (256,448,3)
+00051/0985 7 (256,448,3)
+00052/0012 7 (256,448,3)
+00052/0015 7 (256,448,3)
+00052/0052 7 (256,448,3)
+00052/0056 7 (256,448,3)
+00052/0060 7 (256,448,3)
+00052/0157 7 (256,448,3)
+00052/0265 7 (256,448,3)
+00052/0788 7 (256,448,3)
+00052/0790 7 (256,448,3)
+00052/0793 7 (256,448,3)
+00052/0816 7 (256,448,3)
+00052/0824 7 (256,448,3)
+00052/0918 7 (256,448,3)
+00052/0933 7 (256,448,3)
+00052/0947 7 (256,448,3)
+00053/0232 7 (256,448,3)
+00053/0277 7 (256,448,3)
+00053/0362 7 (256,448,3)
+00053/0577 7 (256,448,3)
+00053/0609 7 (256,448,3)
+00053/0612 7 (256,448,3)
+00053/0628 7 (256,448,3)
+00053/0629 7 (256,448,3)
+00053/0633 7 (256,448,3)
+00053/0659 7 (256,448,3)
+00053/0667 7 (256,448,3)
+00053/0671 7 (256,448,3)
+00053/0797 7 (256,448,3)
+00053/0804 7 (256,448,3)
+00053/0807 7 (256,448,3)
+00053/0952 7 (256,448,3)
+00053/0970 7 (256,448,3)
+00053/0981 7 (256,448,3)
+00053/0999 7 (256,448,3)
+00054/0003 7 (256,448,3)
+00054/0013 7 (256,448,3)
+00054/0020 7 (256,448,3)
+00054/0022 7 (256,448,3)
+00054/0023 7 (256,448,3)
+00054/0044 7 (256,448,3)
+00054/0051 7 (256,448,3)
+00054/0063 7 (256,448,3)
+00054/0065 7 (256,448,3)
+00054/0145 7 (256,448,3)
+00054/0153 7 (256,448,3)
+00054/0203 7 (256,448,3)
+00054/0325 7 (256,448,3)
+00054/0445 7 (256,448,3)
+00054/0448 7 (256,448,3)
+00054/0456 7 (256,448,3)
+00054/0457 7 (256,448,3)
+00054/0519 7 (256,448,3)
+00054/0524 7 (256,448,3)
+00054/0530 7 (256,448,3)
+00054/0532 7 (256,448,3)
+00054/0535 7 (256,448,3)
+00054/0574 7 (256,448,3)
+00054/0760 7 (256,448,3)
+00054/0767 7 (256,448,3)
+00054/0837 7 (256,448,3)
+00055/0011 7 (256,448,3)
+00055/0109 7 (256,448,3)
+00055/0111 7 (256,448,3)
+00055/0117 7 (256,448,3)
+00055/0119 7 (256,448,3)
+00055/0182 7 (256,448,3)
+00055/0192 7 (256,448,3)
+00055/0193 7 (256,448,3)
+00055/0200 7 (256,448,3)
+00055/0204 7 (256,448,3)
+00055/0207 7 (256,448,3)
+00055/0212 7 (256,448,3)
+00055/0213 7 (256,448,3)
+00055/0348 7 (256,448,3)
+00055/0423 7 (256,448,3)
+00055/0427 7 (256,448,3)
+00055/0456 7 (256,448,3)
+00055/0489 7 (256,448,3)
+00055/0689 7 (256,448,3)
+00055/0753 7 (256,448,3)
+00055/0802 7 (256,448,3)
+00055/0844 7 (256,448,3)
+00055/0850 7 (256,448,3)
+00055/0982 7 (256,448,3)
+00055/0993 7 (256,448,3)
+00056/0113 7 (256,448,3)
+00056/0148 7 (256,448,3)
+00056/0151 7 (256,448,3)
+00056/0316 7 (256,448,3)
+00056/0379 7 (256,448,3)
+00056/0380 7 (256,448,3)
+00056/0385 7 (256,448,3)
+00056/0505 7 (256,448,3)
+00056/0579 7 (256,448,3)
+00057/0254 7 (256,448,3)
+00057/0264 7 (256,448,3)
+00057/0272 7 (256,448,3)
+00057/0403 7 (256,448,3)
+00057/0501 7 (256,448,3)
+00057/0503 7 (256,448,3)
+00057/0884 7 (256,448,3)
+00058/0026 7 (256,448,3)
+00058/0029 7 (256,448,3)
+00058/0104 7 (256,448,3)
+00058/0124 7 (256,448,3)
+00058/0162 7 (256,448,3)
+00058/0288 7 (256,448,3)
+00058/0289 7 (256,448,3)
+00058/0323 7 (256,448,3)
+00058/0328 7 (256,448,3)
+00058/0329 7 (256,448,3)
+00058/0337 7 (256,448,3)
+00058/0367 7 (256,448,3)
+00058/0383 7 (256,448,3)
+00058/0395 7 (256,448,3)
+00060/0178 7 (256,448,3)
+00060/0182 7 (256,448,3)
+00061/0001 7 (256,448,3)
+00061/0003 7 (256,448,3)
+00061/0006 7 (256,448,3)
+00061/0443 7 (256,448,3)
+00061/0586 7 (256,448,3)
+00061/0587 7 (256,448,3)
+00061/0774 7 (256,448,3)
+00061/0789 7 (256,448,3)
+00061/0815 7 (256,448,3)
+00061/0817 7 (256,448,3)
+00061/0826 7 (256,448,3)
+00061/0829 7 (256,448,3)
+00061/0830 7 (256,448,3)
+00061/0832 7 (256,448,3)
+00061/0833 7 (256,448,3)
+00061/0836 7 (256,448,3)
+00061/0837 7 (256,448,3)
+00061/0839 7 (256,448,3)
+00061/0843 7 (256,448,3)
+00061/0849 7 (256,448,3)
+00061/0859 7 (256,448,3)
+00061/0861 7 (256,448,3)
+00061/0868 7 (256,448,3)
+00061/0877 7 (256,448,3)
+00061/0889 7 (256,448,3)
+00061/0905 7 (256,448,3)
+00062/0115 7 (256,448,3)
+00062/0118 7 (256,448,3)
+00062/0125 7 (256,448,3)
+00062/0134 7 (256,448,3)
+00062/0142 7 (256,448,3)
+00062/0400 7 (256,448,3)
+00062/0457 7 (256,448,3)
+00062/0459 7 (256,448,3)
+00062/0560 7 (256,448,3)
+00062/0650 7 (256,448,3)
+00062/0655 7 (256,448,3)
+00062/0715 7 (256,448,3)
+00062/0847 7 (256,448,3)
+00062/0905 7 (256,448,3)
+00062/0981 7 (256,448,3)
+00063/0177 7 (256,448,3)
+00063/0230 7 (256,448,3)
+00063/0253 7 (256,448,3)
+00063/0257 7 (256,448,3)
+00063/0326 7 (256,448,3)
+00063/0530 7 (256,448,3)
+00063/0677 7 (256,448,3)
+00063/0759 7 (256,448,3)
+00063/0761 7 (256,448,3)
+00063/0777 7 (256,448,3)
+00063/0842 7 (256,448,3)
+00063/0900 7 (256,448,3)
+00064/0014 7 (256,448,3)
+00064/0028 7 (256,448,3)
+00064/0029 7 (256,448,3)
+00064/0030 7 (256,448,3)
+00064/0037 7 (256,448,3)
+00064/0044 7 (256,448,3)
+00064/0280 7 (256,448,3)
+00064/0285 7 (256,448,3)
+00064/0286 7 (256,448,3)
+00064/0291 7 (256,448,3)
+00064/0300 7 (256,448,3)
+00064/0303 7 (256,448,3)
+00064/0308 7 (256,448,3)
+00064/0314 7 (256,448,3)
+00064/0316 7 (256,448,3)
+00064/0317 7 (256,448,3)
+00064/0323 7 (256,448,3)
+00064/0435 7 (256,448,3)
+00064/0733 7 (256,448,3)
+00064/0848 7 (256,448,3)
+00064/0868 7 (256,448,3)
+00064/0888 7 (256,448,3)
+00064/0898 7 (256,448,3)
+00065/0116 7 (256,448,3)
+00065/0121 7 (256,448,3)
+00065/0122 7 (256,448,3)
+00065/0124 7 (256,448,3)
+00065/0125 7 (256,448,3)
+00065/0126 7 (256,448,3)
+00065/0136 7 (256,448,3)
+00065/0146 7 (256,448,3)
+00065/0147 7 (256,448,3)
+00065/0163 7 (256,448,3)
+00065/0170 7 (256,448,3)
+00065/0175 7 (256,448,3)
+00065/0176 7 (256,448,3)
+00065/0180 7 (256,448,3)
+00065/0184 7 (256,448,3)
+00065/0186 7 (256,448,3)
+00065/0332 7 (256,448,3)
+00065/0343 7 (256,448,3)
+00065/0365 7 (256,448,3)
+00065/0393 7 (256,448,3)
+00065/0394 7 (256,448,3)
+00065/0442 7 (256,448,3)
+00065/0459 7 (256,448,3)
+00065/0462 7 (256,448,3)
+00065/0476 7 (256,448,3)
+00065/0483 7 (256,448,3)
+00065/0590 7 (256,448,3)
+00065/0593 7 (256,448,3)
+00065/0595 7 (256,448,3)
+00065/0774 7 (256,448,3)
+00065/0947 7 (256,448,3)
+00065/0985 7 (256,448,3)
+00065/0986 7 (256,448,3)
+00066/0015 7 (256,448,3)
+00066/0043 7 (256,448,3)
+00066/0131 7 (256,448,3)
+00066/0157 7 (256,448,3)
+00066/0169 7 (256,448,3)
+00066/0374 7 (256,448,3)
+00066/0382 7 (256,448,3)
+00066/0481 7 (256,448,3)
+00066/0482 7 (256,448,3)
+00066/0491 7 (256,448,3)
+00066/0493 7 (256,448,3)
+00066/0494 7 (256,448,3)
+00066/0496 7 (256,448,3)
+00066/0680 7 (256,448,3)
+00066/0700 7 (256,448,3)
+00066/0887 7 (256,448,3)
+00066/0910 7 (256,448,3)
+00066/0918 7 (256,448,3)
+00067/0024 7 (256,448,3)
+00067/0059 7 (256,448,3)
+00067/0408 7 (256,448,3)
+00067/0414 7 (256,448,3)
+00067/0417 7 (256,448,3)
+00067/0419 7 (256,448,3)
+00067/0423 7 (256,448,3)
+00067/0441 7 (256,448,3)
+00067/0467 7 (256,448,3)
+00067/0471 7 (256,448,3)
+00067/0487 7 (256,448,3)
+00067/0494 7 (256,448,3)
+00067/0497 7 (256,448,3)
+00067/0513 7 (256,448,3)
+00067/0521 7 (256,448,3)
+00068/0111 7 (256,448,3)
+00068/0123 7 (256,448,3)
+00068/0126 7 (256,448,3)
+00068/0129 7 (256,448,3)
+00068/0270 7 (256,448,3)
+00068/0330 7 (256,448,3)
+00068/0407 7 (256,448,3)
+00068/0428 7 (256,448,3)
+00068/0544 7 (256,448,3)
+00068/0635 7 (256,448,3)
+00068/0637 7 (256,448,3)
+00068/0736 7 (256,448,3)
+00068/0738 7 (256,448,3)
+00068/0747 7 (256,448,3)
+00068/0748 7 (256,448,3)
+00068/0749 7 (256,448,3)
+00068/0762 7 (256,448,3)
+00068/0815 7 (256,448,3)
+00068/0981 7 (256,448,3)
+00068/0982 7 (256,448,3)
+00069/0187 7 (256,448,3)
+00069/0191 7 (256,448,3)
+00070/0001 7 (256,448,3)
+00070/0003 7 (256,448,3)
+00070/0340 7 (256,448,3)
+00070/0341 7 (256,448,3)
+00070/0342 7 (256,448,3)
+00070/0347 7 (256,448,3)
+00070/0372 7 (256,448,3)
+00070/0383 7 (256,448,3)
+00070/0389 7 (256,448,3)
+00070/0728 7 (256,448,3)
+00070/0813 7 (256,448,3)
+00070/0814 7 (256,448,3)
+00070/0823 7 (256,448,3)
+00070/0840 7 (256,448,3)
+00070/0843 7 (256,448,3)
+00070/0861 7 (256,448,3)
+00071/0111 7 (256,448,3)
+00071/0138 7 (256,448,3)
+00071/0143 7 (256,448,3)
+00071/0150 7 (256,448,3)
+00071/0508 7 (256,448,3)
+00071/0514 7 (256,448,3)
+00071/0550 7 (256,448,3)
+00071/0556 7 (256,448,3)
+00071/0600 7 (256,448,3)
+00071/0665 7 (256,448,3)
+00071/0670 7 (256,448,3)
+00071/0672 7 (256,448,3)
+00071/0673 7 (256,448,3)
+00071/0705 7 (256,448,3)
+00071/0706 7 (256,448,3)
+00071/0707 7 (256,448,3)
+00071/0774 7 (256,448,3)
+00071/0799 7 (256,448,3)
+00071/0814 7 (256,448,3)
+00071/0816 7 (256,448,3)
+00071/0819 7 (256,448,3)
+00071/0823 7 (256,448,3)
+00071/0828 7 (256,448,3)
+00071/0830 7 (256,448,3)
+00071/0839 7 (256,448,3)
+00071/0841 7 (256,448,3)
+00072/0192 7 (256,448,3)
+00072/0194 7 (256,448,3)
+00072/0197 7 (256,448,3)
+00072/0199 7 (256,448,3)
+00072/0285 7 (256,448,3)
+00072/0586 7 (256,448,3)
+00072/0795 7 (256,448,3)
+00072/0811 7 (256,448,3)
+00072/0812 7 (256,448,3)
+00072/0824 7 (256,448,3)
+00072/0831 7 (256,448,3)
+00072/0835 7 (256,448,3)
+00072/0837 7 (256,448,3)
+00072/0841 7 (256,448,3)
+00072/0962 7 (256,448,3)
+00073/0296 7 (256,448,3)
+00073/0299 7 (256,448,3)
+00073/0300 7 (256,448,3)
+00073/0301 7 (256,448,3)
+00073/0427 7 (256,448,3)
+00073/0428 7 (256,448,3)
+00073/0494 7 (256,448,3)
+00073/0615 7 (256,448,3)
+00073/0620 7 (256,448,3)
+00073/0624 7 (256,448,3)
+00073/0979 7 (256,448,3)
+00074/0226 7 (256,448,3)
+00074/0250 7 (256,448,3)
+00074/0284 7 (256,448,3)
+00074/0503 7 (256,448,3)
+00074/0614 7 (256,448,3)
+00074/0629 7 (256,448,3)
+00074/0762 7 (256,448,3)
+00074/0765 7 (256,448,3)
+00074/0900 7 (256,448,3)
+00074/0908 7 (256,448,3)
+00075/0352 7 (256,448,3)
+00075/0360 7 (256,448,3)
+00075/0361 7 (256,448,3)
+00075/0365 7 (256,448,3)
+00075/0383 7 (256,448,3)
+00075/0384 7 (256,448,3)
+00075/0386 7 (256,448,3)
+00075/0407 7 (256,448,3)
+00075/0410 7 (256,448,3)
+00075/0412 7 (256,448,3)
+00075/0413 7 (256,448,3)
+00075/0459 7 (256,448,3)
+00075/0504 7 (256,448,3)
+00075/0515 7 (256,448,3)
+00075/0518 7 (256,448,3)
+00075/0567 7 (256,448,3)
+00075/0681 7 (256,448,3)
+00075/0693 7 (256,448,3)
+00075/0728 7 (256,448,3)
+00075/0731 7 (256,448,3)
+00075/0804 7 (256,448,3)
+00075/0974 7 (256,448,3)
+00075/0975 7 (256,448,3)
+00075/0983 7 (256,448,3)
+00075/0997 7 (256,448,3)
+00076/0006 7 (256,448,3)
+00076/0007 7 (256,448,3)
+00076/0011 7 (256,448,3)
+00076/0013 7 (256,448,3)
+00076/0014 7 (256,448,3)
+00076/0027 7 (256,448,3)
+00076/0029 7 (256,448,3)
+00076/0037 7 (256,448,3)
+00076/0041 7 (256,448,3)
+00076/0055 7 (256,448,3)
+00076/0071 7 (256,448,3)
+00076/0172 7 (256,448,3)
+00076/0275 7 (256,448,3)
+00076/0286 7 (256,448,3)
+00076/0467 7 (256,448,3)
+00076/0481 7 (256,448,3)
+00076/0527 7 (256,448,3)
+00076/0895 7 (256,448,3)
+00076/0896 7 (256,448,3)
+00076/0906 7 (256,448,3)
+00076/0924 7 (256,448,3)
+00076/0964 7 (256,448,3)
+00076/0984 7 (256,448,3)
+00077/0317 7 (256,448,3)
+00077/0322 7 (256,448,3)
+00077/0333 7 (256,448,3)
+00077/0334 7 (256,448,3)
+00077/0480 7 (256,448,3)
+00077/0488 7 (256,448,3)
+00077/0490 7 (256,448,3)
+00077/0582 7 (256,448,3)
+00077/0586 7 (256,448,3)
+00077/0969 7 (256,448,3)
+00078/0007 7 (256,448,3)
+00078/0011 7 (256,448,3)
+00078/0153 7 (256,448,3)
+00078/0289 7 (256,448,3)
+00078/0312 7 (256,448,3)
+00078/0492 7 (256,448,3)
+00078/0580 7 (256,448,3)
+00078/0595 7 (256,448,3)
+00078/0814 7 (256,448,3)
+00078/0950 7 (256,448,3)
+00078/0955 7 (256,448,3)
+00079/0060 7 (256,448,3)
+00079/0067 7 (256,448,3)
+00080/0216 7 (256,448,3)
+00080/0308 7 (256,448,3)
+00080/0504 7 (256,448,3)
+00080/0552 7 (256,448,3)
+00080/0576 7 (256,448,3)
+00080/0583 7 (256,448,3)
+00080/0837 7 (256,448,3)
+00080/0839 7 (256,448,3)
+00080/0871 7 (256,448,3)
+00080/0877 7 (256,448,3)
+00080/0880 7 (256,448,3)
+00080/0969 7 (256,448,3)
+00080/0973 7 (256,448,3)
+00080/0980 7 (256,448,3)
+00081/0202 7 (256,448,3)
+00081/0203 7 (256,448,3)
+00081/0210 7 (256,448,3)
+00081/0268 7 (256,448,3)
+00081/0281 7 (256,448,3)
+00081/0283 7 (256,448,3)
+00081/0317 7 (256,448,3)
+00081/0327 7 (256,448,3)
+00082/0018 7 (256,448,3)
+00082/0025 7 (256,448,3)
+00082/0089 7 (256,448,3)
+00082/0140 7 (256,448,3)
+00082/0442 7 (256,448,3)
+00082/0465 7 (256,448,3)
+00082/0473 7 (256,448,3)
+00082/0481 7 (256,448,3)
+00082/0492 7 (256,448,3)
+00082/0495 7 (256,448,3)
+00082/0497 7 (256,448,3)
+00082/0502 7 (256,448,3)
+00082/0504 7 (256,448,3)
+00082/0506 7 (256,448,3)
+00082/0507 7 (256,448,3)
+00082/0510 7 (256,448,3)
+00082/0519 7 (256,448,3)
+00082/0523 7 (256,448,3)
+00082/0588 7 (256,448,3)
+00082/0597 7 (256,448,3)
+00082/0632 7 (256,448,3)
+00082/0751 7 (256,448,3)
+00082/0767 7 (256,448,3)
+00082/0771 7 (256,448,3)
+00082/0790 7 (256,448,3)
+00082/0804 7 (256,448,3)
+00082/0823 7 (256,448,3)
+00083/0052 7 (256,448,3)
+00083/0056 7 (256,448,3)
+00083/0113 7 (256,448,3)
+00083/0114 7 (256,448,3)
+00083/0122 7 (256,448,3)
+00083/0137 7 (256,448,3)
+00083/0270 7 (256,448,3)
+00083/0295 7 (256,448,3)
+00083/0303 7 (256,448,3)
+00083/0308 7 (256,448,3)
+00083/0586 7 (256,448,3)
+00083/0592 7 (256,448,3)
+00083/0640 7 (256,448,3)
+00083/0648 7 (256,448,3)
+00083/0654 7 (256,448,3)
+00083/0662 7 (256,448,3)
+00083/0666 7 (256,448,3)
+00083/0668 7 (256,448,3)
+00083/0669 7 (256,448,3)
+00083/0675 7 (256,448,3)
+00083/0679 7 (256,448,3)
+00083/0681 7 (256,448,3)
+00083/0682 7 (256,448,3)
+00083/0694 7 (256,448,3)
+00083/0695 7 (256,448,3)
+00083/0697 7 (256,448,3)
+00083/0704 7 (256,448,3)
+00083/0713 7 (256,448,3)
+00083/0721 7 (256,448,3)
+00083/0855 7 (256,448,3)
+00084/0109 7 (256,448,3)
+00084/0113 7 (256,448,3)
+00084/0306 7 (256,448,3)
+00084/0442 7 (256,448,3)
+00084/0669 7 (256,448,3)
+00084/0679 7 (256,448,3)
+00084/0685 7 (256,448,3)
+00084/0691 7 (256,448,3)
+00084/0768 7 (256,448,3)
+00084/0817 7 (256,448,3)
+00085/0027 7 (256,448,3)
+00085/0035 7 (256,448,3)
+00085/0038 7 (256,448,3)
+00085/0223 7 (256,448,3)
+00085/0233 7 (256,448,3)
+00085/0281 7 (256,448,3)
+00085/0287 7 (256,448,3)
+00085/0313 7 (256,448,3)
+00085/0521 7 (256,448,3)
+00085/0848 7 (256,448,3)
+00085/0855 7 (256,448,3)
+00085/0865 7 (256,448,3)
+00085/0952 7 (256,448,3)
+00085/0964 7 (256,448,3)
+00085/0973 7 (256,448,3)
+00085/0986 7 (256,448,3)
+00085/0993 7 (256,448,3)
+00086/0070 7 (256,448,3)
+00086/0075 7 (256,448,3)
+00086/0094 7 (256,448,3)
+00086/0103 7 (256,448,3)
+00086/0112 7 (256,448,3)
+00086/0288 7 (256,448,3)
+00086/0576 7 (256,448,3)
+00086/0580 7 (256,448,3)
+00086/0584 7 (256,448,3)
+00086/0599 7 (256,448,3)
+00086/0600 7 (256,448,3)
+00086/0602 7 (256,448,3)
+00086/0612 7 (256,448,3)
+00086/0629 7 (256,448,3)
+00086/0655 7 (256,448,3)
+00086/0679 7 (256,448,3)
+00086/0694 7 (256,448,3)
+00086/0695 7 (256,448,3)
+00086/0701 7 (256,448,3)
+00086/0760 7 (256,448,3)
+00086/0786 7 (256,448,3)
+00086/0845 7 (256,448,3)
+00086/0868 7 (256,448,3)
+00086/0889 7 (256,448,3)
+00086/0891 7 (256,448,3)
+00086/0927 7 (256,448,3)
+00086/0938 7 (256,448,3)
+00086/0946 7 (256,448,3)
+00086/0963 7 (256,448,3)
+00086/0969 7 (256,448,3)
+00087/0023 7 (256,448,3)
+00087/0029 7 (256,448,3)
+00087/0144 7 (256,448,3)
+00087/0148 7 (256,448,3)
+00087/0159 7 (256,448,3)
+00087/0174 7 (256,448,3)
+00087/0283 7 (256,448,3)
+00087/0284 7 (256,448,3)
+00087/0294 7 (256,448,3)
+00087/0296 7 (256,448,3)
+00087/0498 7 (256,448,3)
+00087/0502 7 (256,448,3)
+00087/0532 7 (256,448,3)
+00087/0557 7 (256,448,3)
+00087/0559 7 (256,448,3)
+00087/0574 7 (256,448,3)
+00087/0577 7 (256,448,3)
+00088/0006 7 (256,448,3)
+00088/0268 7 (256,448,3)
+00088/0320 7 (256,448,3)
+00088/0412 7 (256,448,3)
+00088/0431 7 (256,448,3)
+00088/0432 7 (256,448,3)
+00088/0465 7 (256,448,3)
+00088/0507 7 (256,448,3)
+00088/0565 7 (256,448,3)
+00088/0629 7 (256,448,3)
+00088/0831 7 (256,448,3)
+00088/0836 7 (256,448,3)
+00088/0972 7 (256,448,3)
+00088/0974 7 (256,448,3)
+00088/0980 7 (256,448,3)
+00089/0067 7 (256,448,3)
+00089/0244 7 (256,448,3)
+00089/0404 7 (256,448,3)
+00089/0416 7 (256,448,3)
+00089/0419 7 (256,448,3)
+00089/0428 7 (256,448,3)
+00089/0712 7 (256,448,3)
+00089/0713 7 (256,448,3)
+00089/0723 7 (256,448,3)
+00089/0727 7 (256,448,3)
+00089/0770 7 (256,448,3)
+00089/0809 7 (256,448,3)
+00089/0811 7 (256,448,3)
+00089/0888 7 (256,448,3)
+00089/0898 7 (256,448,3)
+00089/0903 7 (256,448,3)
+00089/0907 7 (256,448,3)
+00089/0911 7 (256,448,3)
+00089/0915 7 (256,448,3)
+00089/0926 7 (256,448,3)
+00089/0955 7 (256,448,3)
+00090/0027 7 (256,448,3)
+00090/0028 7 (256,448,3)
+00090/0032 7 (256,448,3)
+00090/0038 7 (256,448,3)
+00090/0076 7 (256,448,3)
+00090/0081 7 (256,448,3)
+00090/0086 7 (256,448,3)
+00090/0119 7 (256,448,3)
+00090/0258 7 (256,448,3)
+00090/0261 7 (256,448,3)
+00090/0447 7 (256,448,3)
+00090/0498 7 (256,448,3)
+00090/0514 7 (256,448,3)
+00090/0523 7 (256,448,3)
+00090/0530 7 (256,448,3)
+00090/0540 7 (256,448,3)
+00090/0548 7 (256,448,3)
+00090/0565 7 (256,448,3)
+00090/0578 7 (256,448,3)
+00090/0580 7 (256,448,3)
+00090/0581 7 (256,448,3)
+00090/0780 7 (256,448,3)
+00090/0940 7 (256,448,3)
+00090/0984 7 (256,448,3)
+00091/0023 7 (256,448,3)
+00091/0051 7 (256,448,3)
+00091/0317 7 (256,448,3)
+00091/0320 7 (256,448,3)
+00091/0582 7 (256,448,3)
+00091/0585 7 (256,448,3)
+00091/0588 7 (256,448,3)
+00091/0601 7 (256,448,3)
+00091/0602 7 (256,448,3)
+00091/0603 7 (256,448,3)
+00091/0634 7 (256,448,3)
+00091/0693 7 (256,448,3)
+00091/0741 7 (256,448,3)
+00091/0966 7 (256,448,3)
+00091/0973 7 (256,448,3)
+00091/0985 7 (256,448,3)
+00092/0007 7 (256,448,3)
+00092/0132 7 (256,448,3)
+00092/0270 7 (256,448,3)
+00092/0296 7 (256,448,3)
+00092/0611 7 (256,448,3)
+00092/0625 7 (256,448,3)
+00092/0627 7 (256,448,3)
+00092/0651 7 (256,448,3)
+00092/0652 7 (256,448,3)
+00092/0910 7 (256,448,3)
+00093/0075 7 (256,448,3)
+00093/0078 7 (256,448,3)
+00093/0100 7 (256,448,3)
+00093/0132 7 (256,448,3)
+00093/0133 7 (256,448,3)
+00093/0176 7 (256,448,3)
+00093/0177 7 (256,448,3)
+00093/0178 7 (256,448,3)
+00093/0181 7 (256,448,3)
+00093/0183 7 (256,448,3)
+00093/0184 7 (256,448,3)
+00093/0286 7 (256,448,3)
+00093/0304 7 (256,448,3)
+00093/0305 7 (256,448,3)
+00093/0319 7 (256,448,3)
+00093/0324 7 (256,448,3)
+00093/0325 7 (256,448,3)
+00093/0327 7 (256,448,3)
+00093/0331 7 (256,448,3)
+00093/0444 7 (256,448,3)
+00093/0450 7 (256,448,3)
+00093/0593 7 (256,448,3)
+00094/0032 7 (256,448,3)
+00094/0057 7 (256,448,3)
+00094/0139 7 (256,448,3)
+00094/0206 7 (256,448,3)
+00094/0211 7 (256,448,3)
+00094/0215 7 (256,448,3)
+00094/0218 7 (256,448,3)
+00094/0257 7 (256,448,3)
+00094/0329 7 (256,448,3)
+00094/0331 7 (256,448,3)
+00094/0332 7 (256,448,3)
+00094/0369 7 (256,448,3)
+00094/0370 7 (256,448,3)
+00094/0383 7 (256,448,3)
+00094/0385 7 (256,448,3)
+00094/0387 7 (256,448,3)
+00094/0399 7 (256,448,3)
+00094/0605 7 (256,448,3)
+00094/0648 7 (256,448,3)
+00094/0649 7 (256,448,3)
+00094/0759 7 (256,448,3)
+00094/0800 7 (256,448,3)
+00094/0894 7 (256,448,3)
+00094/0896 7 (256,448,3)
+00095/0089 7 (256,448,3)
+00095/0108 7 (256,448,3)
+00095/0109 7 (256,448,3)
+00095/0114 7 (256,448,3)
+00095/0128 7 (256,448,3)
+00095/0133 7 (256,448,3)
+00095/0150 7 (256,448,3)
+00095/0153 7 (256,448,3)
+00095/0154 7 (256,448,3)
+00095/0196 7 (256,448,3)
+00095/0209 7 (256,448,3)
+00095/0228 7 (256,448,3)
+00095/0230 7 (256,448,3)
+00095/0231 7 (256,448,3)
+00095/0242 7 (256,448,3)
+00095/0243 7 (256,448,3)
+00095/0253 7 (256,448,3)
+00095/0280 7 (256,448,3)
+00095/0281 7 (256,448,3)
+00095/0283 7 (256,448,3)
+00095/0314 7 (256,448,3)
+00095/0868 7 (256,448,3)
+00095/0894 7 (256,448,3)
+00096/0062 7 (256,448,3)
+00096/0347 7 (256,448,3)
+00096/0348 7 (256,448,3)
+00096/0359 7 (256,448,3)
+00096/0363 7 (256,448,3)
+00096/0373 7 (256,448,3)
+00096/0378 7 (256,448,3)
+00096/0387 7 (256,448,3)
+00096/0395 7 (256,448,3)
+00096/0396 7 (256,448,3)
+00096/0404 7 (256,448,3)
+00096/0653 7 (256,448,3)
+00096/0668 7 (256,448,3)
+00096/0679 7 (256,448,3)
+00096/0729 7 (256,448,3)
+00096/0736 7 (256,448,3)
+00096/0823 7 (256,448,3)

basicsr/data/paired_image_dataset.py

@@ -0,0 +1,106 @@
+from torch.utils import data as data
+from torchvision.transforms.functional import normalize
+
+from basicsr.data.data_util import paired_paths_from_folder, paired_paths_from_lmdb, paired_paths_from_meta_info_file
+from basicsr.data.transforms import augment, paired_random_crop
+from basicsr.utils import FileClient, bgr2ycbcr, imfrombytes, img2tensor
+from basicsr.utils.registry import DATASET_REGISTRY
+
+
+@DATASET_REGISTRY.register()
+class PairedImageDataset(data.Dataset):
+    """Paired image dataset for image restoration.
+
+    Read LQ (Low Quality, e.g. LR (Low Resolution), blurry, noisy, etc) and GT image pairs.
+
+    There are three modes:
+
+    1. **lmdb**: Use lmdb files. If opt['io_backend'] == lmdb.
+    2. **meta_info_file**: Use meta information file to generate paths. \
+        If opt['io_backend'] != lmdb and opt['meta_info_file'] is not None.
+    3. **folder**: Scan folders to generate paths. The rest.
+
+    Args:
+        opt (dict): Config for train datasets. It contains the following keys:
+        dataroot_gt (str): Data root path for gt.
+        dataroot_lq (str): Data root path for lq.
+        meta_info_file (str): Path for meta information file.
+        io_backend (dict): IO backend type and other kwarg.
+        filename_tmpl (str): Template for each filename. Note that the template excludes the file extension.
+            Default: '{}'.
+        gt_size (int): Cropped patched size for gt patches.
+        use_hflip (bool): Use horizontal flips.
+        use_rot (bool): Use rotation (use vertical flip and transposing h and w for implementation).
+        scale (bool): Scale, which will be added automatically.
+        phase (str): 'train' or 'val'.
+    """
+
+    def __init__(self, opt):
+        super(PairedImageDataset, self).__init__()
+        self.opt = opt
+        # file client (io backend)
+        self.file_client = None
+        self.io_backend_opt = opt['io_backend']
+        self.mean = opt['mean'] if 'mean' in opt else None
+        self.std = opt['std'] if 'std' in opt else None
+
+        self.gt_folder, self.lq_folder = opt['dataroot_gt'], opt['dataroot_lq']
+        if 'filename_tmpl' in opt:
+            self.filename_tmpl = opt['filename_tmpl']
+        else:
+            self.filename_tmpl = '{}'
+
+        if self.io_backend_opt['type'] == 'lmdb':
+            self.io_backend_opt['db_paths'] = [self.lq_folder, self.gt_folder]
+            self.io_backend_opt['client_keys'] = ['lq', 'gt']
+            self.paths = paired_paths_from_lmdb([self.lq_folder, self.gt_folder], ['lq', 'gt'])
+        elif 'meta_info_file' in self.opt and self.opt['meta_info_file'] is not None:
+            self.paths = paired_paths_from_meta_info_file([self.lq_folder, self.gt_folder], ['lq', 'gt'],
+                                                          self.opt['meta_info_file'], self.filename_tmpl)
+        else:
+            self.paths = paired_paths_from_folder([self.lq_folder, self.gt_folder], ['lq', 'gt'], self.filename_tmpl)
+
+    def __getitem__(self, index):
+        if self.file_client is None:
+            self.file_client = FileClient(self.io_backend_opt.pop('type'), **self.io_backend_opt)
+
+        scale = self.opt['scale']
+
+        # Load gt and lq images. Dimension order: HWC; channel order: BGR;
+        # image range: [0, 1], float32.
+        gt_path = self.paths[index]['gt_path']
+        img_bytes = self.file_client.get(gt_path, 'gt')
+        img_gt = imfrombytes(img_bytes, float32=True)
+        lq_path = self.paths[index]['lq_path']
+        img_bytes = self.file_client.get(lq_path, 'lq')
+        img_lq = imfrombytes(img_bytes, float32=True)
+
+        # augmentation for training
+        if self.opt['phase'] == 'train':
+            gt_size = self.opt['gt_size']
+            # random crop
+            img_gt, img_lq = paired_random_crop(img_gt, img_lq, gt_size, scale, gt_path)
+            # flip, rotation
+            img_gt, img_lq = augment([img_gt, img_lq], self.opt['use_hflip'], self.opt['use_rot'])
+
+        # color space transform
+        if 'color' in self.opt and self.opt['color'] == 'y':
+            img_gt = bgr2ycbcr(img_gt, y_only=True)[..., None]
+            img_lq = bgr2ycbcr(img_lq, y_only=True)[..., None]
+
+        # crop the unmatched GT images during validation or testing, especially for SR benchmark datasets
+        # TODO: It is better to update the datasets, rather than force to crop
+        if self.opt['phase'] != 'train':
+            img_gt = img_gt[0:img_lq.shape[0] * scale, 0:img_lq.shape[1] * scale, :]
+
+        # BGR to RGB, HWC to CHW, numpy to tensor
+        img_gt, img_lq = img2tensor([img_gt, img_lq], bgr2rgb=True, float32=True)
+        # normalize
+        if self.mean is not None or self.std is not None:
+            normalize(img_lq, self.mean, self.std, inplace=True)
+            normalize(img_gt, self.mean, self.std, inplace=True)
+
+        return {'lq': img_lq, 'gt': img_gt, 'lq_path': lq_path, 'gt_path': gt_path}
+
+    def __len__(self):
+        return len(self.paths)

basicsr/data/prefetch_dataloader.py

@@ -0,0 +1,122 @@
+import queue as Queue
+import threading
+import torch
+from torch.utils.data import DataLoader
+
+
+class PrefetchGenerator(threading.Thread):
+    """A general prefetch generator.
+
+    Reference: https://stackoverflow.com/questions/7323664/python-generator-pre-fetch
+
+    Args:
+        generator: Python generator.
+        num_prefetch_queue (int): Number of prefetch queue.
+    """
+
+    def __init__(self, generator, num_prefetch_queue):
+        threading.Thread.__init__(self)
+        self.queue = Queue.Queue(num_prefetch_queue)
+        self.generator = generator
+        self.daemon = True
+        self.start()
+
+    def run(self):
+        for item in self.generator:
+            self.queue.put(item)
+        self.queue.put(None)
+
+    def __next__(self):
+        next_item = self.queue.get()
+        if next_item is None:
+            raise StopIteration
+        return next_item
+
+    def __iter__(self):
+        return self
+
+
+class PrefetchDataLoader(DataLoader):
+    """Prefetch version of dataloader.
+
+    Reference: https://github.com/IgorSusmelj/pytorch-styleguide/issues/5#
+
+    TODO:
+    Need to test on single gpu and ddp (multi-gpu). There is a known issue in
+    ddp.
+
+    Args:
+        num_prefetch_queue (int): Number of prefetch queue.
+        kwargs (dict): Other arguments for dataloader.
+    """
+
+    def __init__(self, num_prefetch_queue, **kwargs):
+        self.num_prefetch_queue = num_prefetch_queue
+        super(PrefetchDataLoader, self).__init__(**kwargs)
+
+    def __iter__(self):
+        return PrefetchGenerator(super().__iter__(), self.num_prefetch_queue)
+
+
+class CPUPrefetcher():
+    """CPU prefetcher.
+
+    Args:
+        loader: Dataloader.
+    """
+
+    def __init__(self, loader):
+        self.ori_loader = loader
+        self.loader = iter(loader)
+
+    def next(self):
+        try:
+            return next(self.loader)
+        except StopIteration:
+            return None
+
+    def reset(self):
+        self.loader = iter(self.ori_loader)
+
+
+class CUDAPrefetcher():
+    """CUDA prefetcher.
+
+    Reference: https://github.com/NVIDIA/apex/issues/304#
+
+    It may consume more GPU memory.
+
+    Args:
+        loader: Dataloader.
+        opt (dict): Options.
+    """
+
+    def __init__(self, loader, opt):
+        self.ori_loader = loader
+        self.loader = iter(loader)
+        self.opt = opt
+        self.stream = torch.cuda.Stream()
+        self.device = torch.device('cuda' if opt['num_gpu'] != 0 else 'cpu')
+        self.preload()
+
+    def preload(self):
+        try:
+            self.batch = next(self.loader)  # self.batch is a dict
+        except StopIteration:
+            self.batch = None
+            return None
+        # put tensors to gpu
+        with torch.cuda.stream(self.stream):
+            for k, v in self.batch.items():
+                if torch.is_tensor(v):
+                    self.batch[k] = self.batch[k].to(device=self.device, non_blocking=True)
+
+    def next(self):
+        torch.cuda.current_stream().wait_stream(self.stream)
+        batch = self.batch
+        self.preload()
+        return batch
+
+    def reset(self):
+        self.loader = iter(self.ori_loader)
+        self.preload()

basicsr/data/realesrgan_dataset.py

@@ -0,0 +1,181 @@
+import cv2
+import math
+import numpy as np
+import os
+import os.path as osp
+import random
+import time
+import torch
+from pathlib import Path
+from torch.utils import data as data
+
+from basicsr.data.degradations import circular_lowpass_kernel, random_mixed_kernels
+from basicsr.data.transforms import augment
+from basicsr.utils import FileClient, get_root_logger, imfrombytes, img2tensor
+from basicsr.utils.registry import DATASET_REGISTRY
+
+@DATASET_REGISTRY.register(suffix='basicsr')
+class RealESRGANDataset(data.Dataset):
+    """Dataset used for Real-ESRGAN model:
+    Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data.
+
+    It loads gt (Ground-Truth) images, and augments them.
+    It also generates blur kernels and sinc kernels for generating low-quality images.
+    Note that the low-quality images are processed in tensors on GPUS for faster processing.
+
+    Args:
+        opt (dict): Config for train datasets. It contains the following keys:
+            dataroot_gt (str): Data root path for gt.
+            meta_info (str): Path for meta information file.
+            io_backend (dict): IO backend type and other kwarg.
+            use_hflip (bool): Use horizontal flips.
+            use_rot (bool): Use rotation (use vertical flip and transposing h and w for implementation).
+            Please see more options in the codes.
+    """
+
+    def __init__(self, opt):
+        super(RealESRGANDataset, self).__init__()
+        self.opt = opt
+        self.file_client = None
+        self.io_backend_opt = opt['io_backend']
+
+        # file client (lmdb io backend)
+        self.paths = sorted([str(x) for x in Path(opt['df2k_path']).glob('*.png')])
+        self.paths.extend(sorted([str(x) for x in Path(opt['wed_path']).glob('*.bmp')]))
+
+        # blur settings for the first degradation
+        self.blur_kernel_size = opt['blur_kernel_size']
+        self.kernel_list = opt['kernel_list']
+        self.kernel_prob = opt['kernel_prob']  # a list for each kernel probability
+        self.blur_sigma = opt['blur_sigma']
+        self.betag_range = opt['betag_range']  # betag used in generalized Gaussian blur kernels
+        self.betap_range = opt['betap_range']  # betap used in plateau blur kernels
+        self.sinc_prob = opt['sinc_prob']  # the probability for sinc filters
+
+        # blur settings for the second degradation
+        self.blur_kernel_size2 = opt['blur_kernel_size2']
+        self.kernel_list2 = opt['kernel_list2']
+        self.kernel_prob2 = opt['kernel_prob2']
+        self.blur_sigma2 = opt['blur_sigma2']
+        self.betag_range2 = opt['betag_range2']
+        self.betap_range2 = opt['betap_range2']
+        self.sinc_prob2 = opt['sinc_prob2']
+
+        # a final sinc filter
+        self.final_sinc_prob = opt['final_sinc_prob']
+
+        self.kernel_range = [2 * v + 1 for v in range(3, 11)]  # kernel size ranges from 7 to 21
+        # TODO: kernel range is now hard-coded, should be in the configure file
+        self.pulse_tensor = torch.zeros(21, 21).float()  # convolving with pulse tensor brings no blurry effect
+        self.pulse_tensor[10, 10] = 1
+
+    def __getitem__(self, index):
+        if self.file_client is None:
+            self.file_client = FileClient(self.io_backend_opt.pop('type'), **self.io_backend_opt)
+
+        # -------------------------------- Load gt images -------------------------------- #
+        # Shape: (h, w, c); channel order: BGR; image range: [0, 1], float32.
+        gt_path = self.paths[index]
+        # avoid errors caused by high latency in reading files
+        retry = 3
+        while retry > 0:
+            try:
+                img_bytes = self.file_client.get(gt_path, 'gt')
+            except (IOError, OSError) as e:
+                # logger = get_root_logger()
+                # logger.warn(f'File client error: {e}, remaining retry times: {retry - 1}')
+                # change another file to read
+                index = random.randint(0, self.__len__())
+                gt_path = self.paths[index]
+                time.sleep(1)  # sleep 1s for occasional server congestion
+            else:
+                break
+            finally:
+                retry -= 1
+        img_gt = imfrombytes(img_bytes, float32=True)
+
+        # -------------------- Do augmentation for training: flip, rotation -------------------- #
+        img_gt = augment(img_gt, self.opt['use_hflip'], self.opt['use_rot'])
+
+        # crop or pad to 400
+        # TODO: 400 is hard-coded. You may change it accordingly
+        h, w = img_gt.shape[0:2]
+        crop_pad_size = 400
+        # pad
+        if h < crop_pad_size or w < crop_pad_size:
+            pad_h = max(0, crop_pad_size - h)
+            pad_w = max(0, crop_pad_size - w)
+            img_gt = cv2.copyMakeBorder(img_gt, 0, pad_h, 0, pad_w, cv2.BORDER_REFLECT_101)
+        # crop
+        if img_gt.shape[0] > crop_pad_size or img_gt.shape[1] > crop_pad_size:
+            h, w = img_gt.shape[0:2]
+            # randomly choose top and left coordinates
+            top = random.randint(0, h - crop_pad_size)
+            left = random.randint(0, w - crop_pad_size)
+            img_gt = img_gt[top:top + crop_pad_size, left:left + crop_pad_size, ...]
+
+        # ------------------------ Generate kernels (used in the first degradation) ------------------------ #
+        kernel_size = random.choice(self.kernel_range)
+        if np.random.uniform() < self.opt['sinc_prob']:
+            # this sinc filter setting is for kernels ranging from [7, 21]
+            if kernel_size < 13:
+                omega_c = np.random.uniform(np.pi / 3, np.pi)
+            else:
+                omega_c = np.random.uniform(np.pi / 5, np.pi)
+            kernel = circular_lowpass_kernel(omega_c, kernel_size, pad_to=False)
+        else:
+            kernel = random_mixed_kernels(
+                self.kernel_list,
+                self.kernel_prob,
+                kernel_size,
+                self.blur_sigma,
+                self.blur_sigma, [-math.pi, math.pi],
+                self.betag_range,
+                self.betap_range,
+                noise_range=None)
+        # pad kernel
+        pad_size = (21 - kernel_size) // 2
+        kernel = np.pad(kernel, ((pad_size, pad_size), (pad_size, pad_size)))
+
+        # ------------------------ Generate kernels (used in the second degradation) ------------------------ #
+        kernel_size = random.choice(self.kernel_range)
+        if np.random.uniform() < self.opt['sinc_prob2']:
+            if kernel_size < 13:
+                omega_c = np.random.uniform(np.pi / 3, np.pi)
+            else:
+                omega_c = np.random.uniform(np.pi / 5, np.pi)
+            kernel2 = circular_lowpass_kernel(omega_c, kernel_size, pad_to=False)
+        else:
+            kernel2 = random_mixed_kernels(
+                self.kernel_list2,
+                self.kernel_prob2,
+                kernel_size,
+                self.blur_sigma2,
+                self.blur_sigma2, [-math.pi, math.pi],
+                self.betag_range2,
+                self.betap_range2,
+                noise_range=None)
+
+        # pad kernel
+        pad_size = (21 - kernel_size) // 2
+        kernel2 = np.pad(kernel2, ((pad_size, pad_size), (pad_size, pad_size)))
+
+        # ------------------------------------- the final sinc kernel ------------------------------------- #
+        if np.random.uniform() < self.opt['final_sinc_prob']:
+            kernel_size = random.choice(self.kernel_range)
+            omega_c = np.random.uniform(np.pi / 3, np.pi)
+            sinc_kernel = circular_lowpass_kernel(omega_c, kernel_size, pad_to=21)
+            sinc_kernel = torch.FloatTensor(sinc_kernel)
+        else:
+            sinc_kernel = self.pulse_tensor
+
+        # BGR to RGB, HWC to CHW, numpy to tensor
+        img_gt = img2tensor([img_gt], bgr2rgb=True, float32=True)[0]
+        kernel = torch.FloatTensor(kernel)
+        kernel2 = torch.FloatTensor(kernel2)
+
+        return_d = {'gt': img_gt, 'kernel1': kernel, 'kernel2': kernel2, 'sinc_kernel': sinc_kernel, 'gt_path': gt_path}
+        return return_d
+
+    def __len__(self):
+        return len(self.paths)

basicsr/data/realesrgan_paired_dataset.py

@@ -0,0 +1,106 @@
+import os
+from torch.utils import data as data
+from torchvision.transforms.functional import normalize
+
+from basicsr.data.data_util import paired_paths_from_folder, paired_paths_from_lmdb
+from basicsr.data.transforms import augment, paired_random_crop
+from basicsr.utils import FileClient, imfrombytes, img2tensor
+from basicsr.utils.registry import DATASET_REGISTRY
+
+
+@DATASET_REGISTRY.register(suffix='basicsr')
+class RealESRGANPairedDataset(data.Dataset):
+    """Paired image dataset for image restoration.
+
+    Read LQ (Low Quality, e.g. LR (Low Resolution), blurry, noisy, etc) and GT image pairs.
+
+    There are three modes:
+
+    1. **lmdb**: Use lmdb files. If opt['io_backend'] == lmdb.
+    2. **meta_info_file**: Use meta information file to generate paths. \
+        If opt['io_backend'] != lmdb and opt['meta_info_file'] is not None.
+    3. **folder**: Scan folders to generate paths. The rest.
+
+    Args:
+        opt (dict): Config for train datasets. It contains the following keys:
+        dataroot_gt (str): Data root path for gt.
+        dataroot_lq (str): Data root path for lq.
+        meta_info (str): Path for meta information file.
+        io_backend (dict): IO backend type and other kwarg.
+        filename_tmpl (str): Template for each filename. Note that the template excludes the file extension.
+            Default: '{}'.
+        gt_size (int): Cropped patched size for gt patches.
+        use_hflip (bool): Use horizontal flips.
+        use_rot (bool): Use rotation (use vertical flip and transposing h and w for implementation).
+        scale (bool): Scale, which will be added automatically.
+        phase (str): 'train' or 'val'.
+    """
+
+    def __init__(self, opt):
+        super(RealESRGANPairedDataset, self).__init__()
+        self.opt = opt
+        self.file_client = None
+        self.io_backend_opt = opt['io_backend']
+        # mean and std for normalizing the input images
+        self.mean = opt['mean'] if 'mean' in opt else None
+        self.std = opt['std'] if 'std' in opt else None
+
+        self.gt_folder, self.lq_folder = opt['dataroot_gt'], opt['dataroot_lq']
+        self.filename_tmpl = opt['filename_tmpl'] if 'filename_tmpl' in opt else '{}'
+
+        # file client (lmdb io backend)
+        if self.io_backend_opt['type'] == 'lmdb':
+            self.io_backend_opt['db_paths'] = [self.lq_folder, self.gt_folder]
+            self.io_backend_opt['client_keys'] = ['lq', 'gt']
+            self.paths = paired_paths_from_lmdb([self.lq_folder, self.gt_folder], ['lq', 'gt'])
+        elif 'meta_info' in self.opt and self.opt['meta_info'] is not None:
+            # disk backend with meta_info
+            # Each line in the meta_info describes the relative path to an image
+            with open(self.opt['meta_info']) as fin:
+                paths = [line.strip() for line in fin]
+            self.paths = []
+            for path in paths:
+                gt_path, lq_path = path.split(', ')
+                gt_path = os.path.join(self.gt_folder, gt_path)
+                lq_path = os.path.join(self.lq_folder, lq_path)
+                self.paths.append(dict([('gt_path', gt_path), ('lq_path', lq_path)]))
+        else:
+            # disk backend
+            # it will scan the whole folder to get meta info
+            # it will be time-consuming for folders with too many files. It is recommended using an extra meta txt file
+            self.paths = paired_paths_from_folder([self.lq_folder, self.gt_folder], ['lq', 'gt'], self.filename_tmpl)
+
+    def __getitem__(self, index):
+        if self.file_client is None:
+            self.file_client = FileClient(self.io_backend_opt.pop('type'), **self.io_backend_opt)
+
+        scale = self.opt['scale']
+
+        # Load gt and lq images. Dimension order: HWC; channel order: BGR;
+        # image range: [0, 1], float32.
+        gt_path = self.paths[index]['gt_path']
+        img_bytes = self.file_client.get(gt_path, 'gt')
+        img_gt = imfrombytes(img_bytes, float32=True)
+        lq_path = self.paths[index]['lq_path']
+        img_bytes = self.file_client.get(lq_path, 'lq')
+        img_lq = imfrombytes(img_bytes, float32=True)
+
+        # augmentation for training
+        if self.opt['phase'] == 'train':
+            gt_size = self.opt['gt_size']
+            # random crop
+            img_gt, img_lq = paired_random_crop(img_gt, img_lq, gt_size, scale, gt_path)
+            # flip, rotation
+            img_gt, img_lq = augment([img_gt, img_lq], self.opt['use_hflip'], self.opt['use_rot'])
+
+        # BGR to RGB, HWC to CHW, numpy to tensor
+        img_gt, img_lq = img2tensor([img_gt, img_lq], bgr2rgb=True, float32=True)
+        # normalize
+        if self.mean is not None or self.std is not None:
+            normalize(img_lq, self.mean, self.std, inplace=True)
+            normalize(img_gt, self.mean, self.std, inplace=True)
+
+        return {'lq': img_lq, 'gt': img_gt, 'lq_path': lq_path, 'gt_path': gt_path}
+
+    def __len__(self):
+        return len(self.paths)

basicsr/data/reds_dataset.py

@@ -0,0 +1,352 @@
+import numpy as np
+import random
+import torch
+from pathlib import Path
+from torch.utils import data as data
+
+from basicsr.data.transforms import augment, paired_random_crop
+from basicsr.utils import FileClient, get_root_logger, imfrombytes, img2tensor
+from basicsr.utils.flow_util import dequantize_flow
+from basicsr.utils.registry import DATASET_REGISTRY
+
+
+@DATASET_REGISTRY.register()
+class REDSDataset(data.Dataset):
+    """REDS dataset for training.
+
+    The keys are generated from a meta info txt file.
+    basicsr/data/meta_info/meta_info_REDS_GT.txt
+
+    Each line contains:
+    1. subfolder (clip) name; 2. frame number; 3. image shape, separated by
+    a white space.
+    Examples:
+    000 100 (720,1280,3)
+    001 100 (720,1280,3)
+    ...
+
+    Key examples: "000/00000000"
+    GT (gt): Ground-Truth;
+    LQ (lq): Low-Quality, e.g., low-resolution/blurry/noisy/compressed frames.
+
+    Args:
+        opt (dict): Config for train dataset. It contains the following keys:
+        dataroot_gt (str): Data root path for gt.
+        dataroot_lq (str): Data root path for lq.
+        dataroot_flow (str, optional): Data root path for flow.
+        meta_info_file (str): Path for meta information file.
+        val_partition (str): Validation partition types. 'REDS4' or 'official'.
+        io_backend (dict): IO backend type and other kwarg.
+        num_frame (int): Window size for input frames.
+        gt_size (int): Cropped patched size for gt patches.
+        interval_list (list): Interval list for temporal augmentation.
+        random_reverse (bool): Random reverse input frames.
+        use_hflip (bool): Use horizontal flips.
+        use_rot (bool): Use rotation (use vertical flip and transposing h and w for implementation).
+        scale (bool): Scale, which will be added automatically.
+    """
+
+    def __init__(self, opt):
+        super(REDSDataset, self).__init__()
+        self.opt = opt
+        self.gt_root, self.lq_root = Path(opt['dataroot_gt']), Path(opt['dataroot_lq'])
+        self.flow_root = Path(opt['dataroot_flow']) if opt['dataroot_flow'] is not None else None
+        assert opt['num_frame'] % 2 == 1, (f'num_frame should be odd number, but got {opt["num_frame"]}')
+        self.num_frame = opt['num_frame']
+        self.num_half_frames = opt['num_frame'] // 2
+
+        self.keys = []
+        with open(opt['meta_info_file'], 'r') as fin:
+            for line in fin:
+                folder, frame_num, _ = line.split(' ')
+                self.keys.extend([f'{folder}/{i:08d}' for i in range(int(frame_num))])
+
+        # remove the video clips used in validation
+        if opt['val_partition'] == 'REDS4':
+            val_partition = ['000', '011', '015', '020']
+        elif opt['val_partition'] == 'official':
+            val_partition = [f'{v:03d}' for v in range(240, 270)]
+        else:
+            raise ValueError(f'Wrong validation partition {opt["val_partition"]}.'
+                             f"Supported ones are ['official', 'REDS4'].")
+        self.keys = [v for v in self.keys if v.split('/')[0] not in val_partition]
+
+        # file client (io backend)
+        self.file_client = None
+        self.io_backend_opt = opt['io_backend']
+        self.is_lmdb = False
+        if self.io_backend_opt['type'] == 'lmdb':
+            self.is_lmdb = True
+            if self.flow_root is not None:
+                self.io_backend_opt['db_paths'] = [self.lq_root, self.gt_root, self.flow_root]
+                self.io_backend_opt['client_keys'] = ['lq', 'gt', 'flow']
+            else:
+                self.io_backend_opt['db_paths'] = [self.lq_root, self.gt_root]
+                self.io_backend_opt['client_keys'] = ['lq', 'gt']
+
+        # temporal augmentation configs
+        self.interval_list = opt['interval_list']
+        self.random_reverse = opt['random_reverse']
+        interval_str = ','.join(str(x) for x in opt['interval_list'])
+        logger = get_root_logger()
+        logger.info(f'Temporal augmentation interval list: [{interval_str}]; '
+                    f'random reverse is {self.random_reverse}.')
+
+    def __getitem__(self, index):
+        if self.file_client is None:
+            self.file_client = FileClient(self.io_backend_opt.pop('type'), **self.io_backend_opt)
+
+        scale = self.opt['scale']
+        gt_size = self.opt['gt_size']
+        key = self.keys[index]
+        clip_name, frame_name = key.split('/')  # key example: 000/00000000
+        center_frame_idx = int(frame_name)
+
+        # determine the neighboring frames
+        interval = random.choice(self.interval_list)
+
+        # ensure not exceeding the borders
+        start_frame_idx = center_frame_idx - self.num_half_frames * interval
+        end_frame_idx = center_frame_idx + self.num_half_frames * interval
+        # each clip has 100 frames starting from 0 to 99
+        while (start_frame_idx < 0) or (end_frame_idx > 99):
+            center_frame_idx = random.randint(0, 99)
+            start_frame_idx = (center_frame_idx - self.num_half_frames * interval)
+            end_frame_idx = center_frame_idx + self.num_half_frames * interval
+        frame_name = f'{center_frame_idx:08d}'
+        neighbor_list = list(range(start_frame_idx, end_frame_idx + 1, interval))
+        # random reverse
+        if self.random_reverse and random.random() < 0.5:
+            neighbor_list.reverse()
+
+        assert len(neighbor_list) == self.num_frame, (f'Wrong length of neighbor list: {len(neighbor_list)}')
+
+        # get the GT frame (as the center frame)
+        if self.is_lmdb:
+            img_gt_path = f'{clip_name}/{frame_name}'
+        else:
+            img_gt_path = self.gt_root / clip_name / f'{frame_name}.png'
+        img_bytes = self.file_client.get(img_gt_path, 'gt')
+        img_gt = imfrombytes(img_bytes, float32=True)
+
+        # get the neighboring LQ frames
+        img_lqs = []
+        for neighbor in neighbor_list:
+            if self.is_lmdb:
+                img_lq_path = f'{clip_name}/{neighbor:08d}'
+            else:
+                img_lq_path = self.lq_root / clip_name / f'{neighbor:08d}.png'
+            img_bytes = self.file_client.get(img_lq_path, 'lq')
+            img_lq = imfrombytes(img_bytes, float32=True)
+            img_lqs.append(img_lq)
+
+        # get flows
+        if self.flow_root is not None:
+            img_flows = []
+            # read previous flows
+            for i in range(self.num_half_frames, 0, -1):
+                if self.is_lmdb:
+                    flow_path = f'{clip_name}/{frame_name}_p{i}'
+                else:
+                    flow_path = (self.flow_root / clip_name / f'{frame_name}_p{i}.png')
+                img_bytes = self.file_client.get(flow_path, 'flow')
+                cat_flow = imfrombytes(img_bytes, flag='grayscale', float32=False)  # uint8, [0, 255]
+                dx, dy = np.split(cat_flow, 2, axis=0)
+                flow = dequantize_flow(dx, dy, max_val=20, denorm=False)  # we use max_val 20 here.
+                img_flows.append(flow)
+            # read next flows
+            for i in range(1, self.num_half_frames + 1):
+                if self.is_lmdb:
+                    flow_path = f'{clip_name}/{frame_name}_n{i}'
+                else:
+                    flow_path = (self.flow_root / clip_name / f'{frame_name}_n{i}.png')
+                img_bytes = self.file_client.get(flow_path, 'flow')
+                cat_flow = imfrombytes(img_bytes, flag='grayscale', float32=False)  # uint8, [0, 255]
+                dx, dy = np.split(cat_flow, 2, axis=0)
+                flow = dequantize_flow(dx, dy, max_val=20, denorm=False)  # we use max_val 20 here.
+                img_flows.append(flow)
+
+            # for random crop, here, img_flows and img_lqs have the same
+            # spatial size
+            img_lqs.extend(img_flows)
+
+        # randomly crop
+        img_gt, img_lqs = paired_random_crop(img_gt, img_lqs, gt_size, scale, img_gt_path)
+        if self.flow_root is not None:
+            img_lqs, img_flows = img_lqs[:self.num_frame], img_lqs[self.num_frame:]
+
+        # augmentation - flip, rotate
+        img_lqs.append(img_gt)
+        if self.flow_root is not None:
+            img_results, img_flows = augment(img_lqs, self.opt['use_hflip'], self.opt['use_rot'], img_flows)
+        else:
+            img_results = augment(img_lqs, self.opt['use_hflip'], self.opt['use_rot'])
+
+        img_results = img2tensor(img_results)
+        img_lqs = torch.stack(img_results[0:-1], dim=0)
+        img_gt = img_results[-1]
+
+        if self.flow_root is not None:
+            img_flows = img2tensor(img_flows)
+            # add the zero center flow
+            img_flows.insert(self.num_half_frames, torch.zeros_like(img_flows[0]))
+            img_flows = torch.stack(img_flows, dim=0)
+
+        # img_lqs: (t, c, h, w)
+        # img_flows: (t, 2, h, w)
+        # img_gt: (c, h, w)
+        # key: str
+        if self.flow_root is not None:
+            return {'lq': img_lqs, 'flow': img_flows, 'gt': img_gt, 'key': key}
+        else:
+            return {'lq': img_lqs, 'gt': img_gt, 'key': key}
+
+    def __len__(self):
+        return len(self.keys)
+
+
+@DATASET_REGISTRY.register()
+class REDSRecurrentDataset(data.Dataset):
+    """REDS dataset for training recurrent networks.
+
+    The keys are generated from a meta info txt file.
+    basicsr/data/meta_info/meta_info_REDS_GT.txt
+
+    Each line contains:
+    1. subfolder (clip) name; 2. frame number; 3. image shape, separated by
+    a white space.
+    Examples:
+    000 100 (720,1280,3)
+    001 100 (720,1280,3)
+    ...
+
+    Key examples: "000/00000000"
+    GT (gt): Ground-Truth;
+    LQ (lq): Low-Quality, e.g., low-resolution/blurry/noisy/compressed frames.
+
+    Args:
+        opt (dict): Config for train dataset. It contains the following keys:
+        dataroot_gt (str): Data root path for gt.
+        dataroot_lq (str): Data root path for lq.
+        dataroot_flow (str, optional): Data root path for flow.
+        meta_info_file (str): Path for meta information file.
+        val_partition (str): Validation partition types. 'REDS4' or 'official'.
+        io_backend (dict): IO backend type and other kwarg.
+        num_frame (int): Window size for input frames.
+        gt_size (int): Cropped patched size for gt patches.
+        interval_list (list): Interval list for temporal augmentation.
+        random_reverse (bool): Random reverse input frames.
+        use_hflip (bool): Use horizontal flips.
+        use_rot (bool): Use rotation (use vertical flip and transposing h and w for implementation).
+        scale (bool): Scale, which will be added automatically.
+    """
+
+    def __init__(self, opt):
+        super(REDSRecurrentDataset, self).__init__()
+        self.opt = opt
+        self.gt_root, self.lq_root = Path(opt['dataroot_gt']), Path(opt['dataroot_lq'])
+        self.num_frame = opt['num_frame']
+
+        self.keys = []
+        with open(opt['meta_info_file'], 'r') as fin:
+            for line in fin:
+                folder, frame_num, _ = line.split(' ')
+                self.keys.extend([f'{folder}/{i:08d}' for i in range(int(frame_num))])
+
+        # remove the video clips used in validation
+        if opt['val_partition'] == 'REDS4':
+            val_partition = ['000', '011', '015', '020']
+        elif opt['val_partition'] == 'official':
+            val_partition = [f'{v:03d}' for v in range(240, 270)]
+        else:
+            raise ValueError(f'Wrong validation partition {opt["val_partition"]}.'
+                             f"Supported ones are ['official', 'REDS4'].")
+        if opt['test_mode']:
+            self.keys = [v for v in self.keys if v.split('/')[0] in val_partition]
+        else:
+            self.keys = [v for v in self.keys if v.split('/')[0] not in val_partition]
+
+        # file client (io backend)
+        self.file_client = None
+        self.io_backend_opt = opt['io_backend']
+        self.is_lmdb = False
+        if self.io_backend_opt['type'] == 'lmdb':
+            self.is_lmdb = True
+            if hasattr(self, 'flow_root') and self.flow_root is not None:
+                self.io_backend_opt['db_paths'] = [self.lq_root, self.gt_root, self.flow_root]
+                self.io_backend_opt['client_keys'] = ['lq', 'gt', 'flow']
+            else:
+                self.io_backend_opt['db_paths'] = [self.lq_root, self.gt_root]
+                self.io_backend_opt['client_keys'] = ['lq', 'gt']
+
+        # temporal augmentation configs
+        self.interval_list = opt.get('interval_list', [1])
+        self.random_reverse = opt.get('random_reverse', False)
+        interval_str = ','.join(str(x) for x in self.interval_list)
+        logger = get_root_logger()
+        logger.info(f'Temporal augmentation interval list: [{interval_str}]; '
+                    f'random reverse is {self.random_reverse}.')
+
+    def __getitem__(self, index):
+        if self.file_client is None:
+            self.file_client = FileClient(self.io_backend_opt.pop('type'), **self.io_backend_opt)
+
+        scale = self.opt['scale']
+        gt_size = self.opt['gt_size']
+        key = self.keys[index]
+        clip_name, frame_name = key.split('/')  # key example: 000/00000000
+
+        # determine the neighboring frames
+        interval = random.choice(self.interval_list)
+
+        # ensure not exceeding the borders
+        start_frame_idx = int(frame_name)
+        if start_frame_idx > 100 - self.num_frame * interval:
+            start_frame_idx = random.randint(0, 100 - self.num_frame * interval)
+        end_frame_idx = start_frame_idx + self.num_frame * interval
+
+        neighbor_list = list(range(start_frame_idx, end_frame_idx, interval))
+
+        # random reverse
+        if self.random_reverse and random.random() < 0.5:
+            neighbor_list.reverse()
+
+        # get the neighboring LQ and GT frames
+        img_lqs = []
+        img_gts = []
+        for neighbor in neighbor_list:
+            if self.is_lmdb:
+                img_lq_path = f'{clip_name}/{neighbor:08d}'
+                img_gt_path = f'{clip_name}/{neighbor:08d}'
+            else:
+                img_lq_path = self.lq_root / clip_name / f'{neighbor:08d}.png'
+                img_gt_path = self.gt_root / clip_name / f'{neighbor:08d}.png'
+
+            # get LQ
+            img_bytes = self.file_client.get(img_lq_path, 'lq')
+            img_lq = imfrombytes(img_bytes, float32=True)
+            img_lqs.append(img_lq)
+
+            # get GT
+            img_bytes = self.file_client.get(img_gt_path, 'gt')
+            img_gt = imfrombytes(img_bytes, float32=True)
+            img_gts.append(img_gt)
+
+        # randomly crop
+        img_gts, img_lqs = paired_random_crop(img_gts, img_lqs, gt_size, scale, img_gt_path)
+
+        # augmentation - flip, rotate
+        img_lqs.extend(img_gts)
+        img_results = augment(img_lqs, self.opt['use_hflip'], self.opt['use_rot'])
+
+        img_results = img2tensor(img_results)
+        img_gts = torch.stack(img_results[len(img_lqs) // 2:], dim=0)
+        img_lqs = torch.stack(img_results[:len(img_lqs) // 2], dim=0)
+
+        # img_lqs: (t, c, h, w)
+        # img_gts: (t, c, h, w)
+        # key: str
+        return {'lq': img_lqs, 'gt': img_gts, 'key': key}
+
+    def __len__(self):
+        return len(self.keys)