does it work with TensorRT for dev and schnell?
#24
by
froilo
- opened
does it work with TensorRT?
Any minimal example available?
It seem onnx some operator not support torch.bfloat16 data type
Could this somehow help? https://github.com/microsoft/onnxscript/pull/1492
See also: https://github.com/microsoft/onnxruntime/issues/13001
and: https://github.com/microsoft/onnxscript/issues/1462
I found ComfyUI TensorRT node (https://github.com/comfyanonymous/ComfyUI_TensorRT/tree/master) support the bf16 Flux, but ComfyUI only supports t5xxl_fp16 (https://comfyanonymous.github.io/ComfyUI_examples/flux/).
Maybe the TensorRT node only convert the transformer, while the text encoders and vae need to be converted by other methods.